Solvd is an AI-first advisory and digital engineering firm focused on transforming business through AI.
The company has over 12 years of experience and operates in multiple countries including the USA, Brazil, Mexico, Ukraine, Poland, Argentina, and Georgia.
The role of Senior Data Platform Engineer involves designing and optimizing data clusters for batch and streaming workloads to ensure performance and reliability.
The engineer will build and expand ecosystems including Spark, Hadoop, Kubernetes, Trino, Delta Lake, and Druid to meet business needs and facilitate new integrations, data ingestion, and transformations.
The position requires introducing and scaling new data platform solutions, iterating on OLAP platforms, and exploring next-generation data formats.
Collaboration with cross-functional teams, including infrastructure engineers, is essential to align platform capabilities with organizational goals.
Requirements:
Proven experience in scaling and tuning large deployments of Spark-on-Kubernetes and Spark-on-Hadoop is required.
Knowledge of open-source S3 alternatives, such as Ceph and MinIO, is necessary.
An in-depth understanding of Hadoop and the HDFS protocol is essential.
Candidates must be skilled in designing and optimizing shuffle-heavy systems, utilizing YARN or Kubernetes with remote shuffle services.
Hands-on experience with at least one lakehouse file format, such as Delta Lake, Apache Iceberg, or Apache Hudi, is required.
Familiarity with OLAP technologies, including ClickHouse, Apache Druid, Apache Pinot, or Apache Doris, is necessary.
Strong communication skills are required to collaborate with diverse stakeholders and effectively convey complex technical concepts.
A proven track record of troubleshooting and resolving issues in large-scale, production environments is essential.
Benefits:
The position offers the opportunity to work in a dynamic and innovative environment at the forefront of AI and digital engineering.
Employees will have the chance to collaborate with cross-functional teams and contribute to impactful projects.
The role provides opportunities for professional growth and development in advanced data platform technologies.
Solvd promotes a culture of innovation, allowing engineers to introduce and scale new solutions.
The company supports a flexible work environment, accommodating various work styles and preferences.
