Remote Senior Data Engineer (Spark)

Description:

• Builds and runs large scale batch data pipelines, ensuring efficiency and cost-effectiveness as data volumes increase.
• Utilizes Apache Spark for managing large scale data processing jobs, including performance tuning for reliable execution.
• Supports data modeling for analytics and organizes data in a lakehouse for downstream usability.
• Handles automated scheduling, monitoring, and data quality checks, collaborating with platform and product teams on end-to-end data flows.

Requirements:

• Strong hands-on experience with Apache Spark, including performance tuning (not just through managed notebooks).
• Experience in data modeling for analytics and organizing data in a lakehouse.
• Proficient in automated scheduling, monitoring, and data quality checks.
• Ability to work with platform and product teams on end-to-end data flows.
• Familiarity with Apache Iceberg or other open table formats.
• Experience with Trino or similar query engines.
• Experience with on-premises or self-managed clusters.

Benefits:

• Opportunity to work with cutting-edge technologies in data engineering.
• Collaborative work environment with cross-functional teams.