Description:

We are seeking a skilled and experienced Data Engineer to join our Threat Research team.
The primary responsibility of this role will be to design, develop, and maintain data pipelines for threat intelligence ingestion, validation, and export automation flows.
The role involves designing, developing, and maintaining data pipelines for ingesting threat intelligence data from various sources into our data ecosystem.
You will implement data validation processes to ensure data accuracy, completeness, and consistency.
Collaboration with threat analysts is essential to understand data requirements and design appropriate solutions.
You will develop automation scripts and workflows for data export processes to external systems or partners.
The position requires optimizing and enhancing existing data pipelines for improved performance and scalability.
Monitoring data pipelines and troubleshooting issues as they arise is crucial to ensure continuous data availability and integrity.
You will document technical specifications, data flows, and procedures for data pipeline maintenance and support.
Staying updated on emerging technologies and best practices in data engineering is important, and you will incorporate them into our data ecosystem.
Providing technical guidance and support to other team members on data engineering best practices and methodologies is also part of the role.

Requirements:

Proven experience as a Data Engineer or similar role, with a focus on data ingest, validation, and export automation is required.
Strong proficiency in Python is necessary.
Experience with data pipeline orchestration tools such as Apache Airflow, Apache NiFi, or similar is essential.
Familiarity with cloud platforms such as Snowflake, AWS, Azure, or Google Cloud Platform is required.
Experience with data validation techniques and tools for ensuring data quality is necessary.
Experience building and deploying images using containerization technologies such as Docker and Kubernetes is required.
Excellent problem-solving skills and attention to detail are essential.
Strong communication and collaboration skills, with the ability to work effectively in a team environment, are required.

The job offers an opportunity to work in a dynamic and innovative environment within the Threat Research team.
You will have the chance to enhance your skills and knowledge in data engineering and threat intelligence.
The position provides opportunities for collaboration with experienced threat analysts and other team members.
You will be part of a team that values continuous learning and staying updated on emerging technologies and best practices.
The role allows for the development of automation processes that can significantly impact the efficiency of data handling.