Remote Senior Data Engineer - ElasticSearch

Description:

• Lead the design, development, and optimization of complex data systems, including data flows, lakes, warehouses, and ETLs, across various cloud platforms.
• Architect and manage Elasticsearch clusters, focusing on performance optimization, security, and scalability.
• Create and optimize indices for performance and storage efficiency, leveraging index templates and mappings.
• Build and optimize Elasticsearch queries using the Query DSL, including complex aggregations and full-text search.
• Implement data ingestion pipelines to load data from various sources, including relational databases (RDBMS), into Elasticsearch.
• Oversee performance monitoring, cluster management, security protocols, backup and restore procedures, and version management of Elasticsearch clusters.
• Develop and maintain data pipelines using industry-standard tools such as Apache Spark, Apache Hive, Apache Airflow, and Stitch.
• Implement data pipeline observability strategies to ensure reliable data flow and system transparency.
• Apply modern software development practices, including Agile methodologies, Test-Driven Development (TDD), and Continuous Integration/Continuous Deployment (CI/CD), to enhance data engineering workflows.

Requirements:

• A minimum of 5 years of experience as a technical lead, building and extending complex data systems.
• Advanced proficiency in Python, with at least 3 years of experience.
• Strong SQL writing and tuning skills.
• Expert-level knowledge of Elasticsearch, including configuration, administration, and optimization techniques.
• Demonstrated experience with building and maintaining data pipelines using tools such as Spark, Hive, Airflow, and Stitch.
• Proven experience with data pipeline observability strategies and tools.
• Strong understanding of software development practices, including Agile, TDD, and CI/CD.

Benefits:

• Competitive salary and benefits package;
• Flexible remote work arrangements;
• Opportunities for professional development and growth;
• Periodic review of the salary;
• Paid events attendance;