Remote Senior Software Engineer, Data Platform

Description:

• The Senior Software Engineer, Data Platform will be part of the Platform Infrastructure Engineering and Reliability group at Stack.
• This role involves designing, implementing, and maintaining the Stack AV on-premises data platform.
• The Data Platform team supports large scale OLAP/OLTP and feature engineering workloads for multiple Product Development groups across the company.
• The team is responsible for building a highly-available and scalable data infrastructure to power critical services and pipelines.
• Members are expected to focus on requirements gathering and the design process to ensure the platform meets business needs and internal customer requirements.

Requirements:

• Candidates must have a data platform engineering focus, with experience supporting large scale interactive and batch workloads.
• Experience with on-premises infrastructure, Kubernetes compute, and S3 compatible object stores (such as Vast Data and Minio) managed via Terraform or ArgoCD is required.
• An expert level understanding of Data Engineering concepts and open source technologies is necessary, along with the ability to distill concepts and provide training to customers across the company.
• Experience running a modern data stack on-premises using open source technologies is essential.
• Familiarity with streaming platforms (Kafka, Pulsar), processing engines (Trino, Apache Spark), table formats (ICEBERG, Delta Lake, Hudi, Xtable), and Postgres is required.
• Proficiency in Python and SQL is necessary, with experience writing and maintaining highly available data applications using Trino and Apache Spark.
• A strong focus on requirements gathering and overall design process is expected, along with experience architecting and optimizing large-scale systems for machine learning, simulation, and data processing.
• Experience running and supporting workflow orchestration systems such as Argo Workflows, Airflow, or Flyte.io is required.
• Candidates should have a strong focus on providing production-grade service to internal engineering teams through observability, instrumentation, and well-defined SLOs.
• Experience working in a diverse and distributed team, with a focus on communication and customer empathy, is essential.

Benefits:

• Stack is committed to being an equal opportunity workplace, promoting diversity across gender, race, age, sexual orientation, religion, disability, and identity.
• The company fosters a culture of inclusion, entrepreneurship, and innovation.
• Employees will have the opportunity to work in a dynamic environment focused on AI advancements and diverse technical domains.
• The position may involve working with cutting-edge technologies and contributing to significant projects that transform modern industries.