Remote Lead Data Engineer Databricks

Description:

• Allata is a global consulting and technology services firm with offices in the US, India, and Argentina.
• The company helps organizations accelerate growth, drive innovation, and solve complex challenges by combining strategy, design, and advanced technology.
• Allata's expertise includes defining business vision, optimizing processes, and creating engaging digital experiences.
• The firm architects and modernizes secure, scalable solutions using cloud platforms and top engineering practices.
• Allata empowers clients to unlock data value through analytics and visualization and leverages artificial intelligence to automate processes and enhance decision-making.
• The Lead Data Engineer Databricks will contribute to transformative enterprise data platform projects focused on developing data pipelines and logic engines for managing ingest, staging, and multi-tier data product modeling.
• Responsibilities include data enrichment using various OEM-specific data warehouse and data lake house platform implementations for consumption via analytics clients.
• The role requires full life cycle design, build, deployment, and optimization of data products for multiple large enterprise industry vertical-specific implementations.

Requirements:

• Candidates must collaborate in defining the overall architecture of the solution, with knowledge of modern Enterprise Data Warehouse and Data Lakehouse architectures that implement Medallion or Lambda architectures.
• The position requires designing, developing, testing, and deploying processing modules to implement data-driven rules using SQL, Stored Procedures, and Pyspark.
• Applicants should understand and own data product engineering deliverables relative to a CI-CD pipeline and standard DevOps practices and principles.
• Experience in building and optimizing data pipelines on platforms like Databricks, SQL Server, or Azure Data Fabric is essential.
• Candidates must have current knowledge of modern data tools like Databricks, FiveTran, Data Fabric, and core experience with data architecture, data integrations, data warehousing, and ETL/ELT processes.
• Applied experience with developing and deploying custom whl and/or in-session notebook scripts for custom execution across parallel executor and worker nodes is required.
• Strong knowledge of cloud and hybrid relational database systems, such as MS SQL Server, PostgreSQL, Oracle, Azure SQL, AWS RDS, Aurora, or a comparable engine is necessary.
• Strong experience with batch and streaming data processing techniques and file compactization strategies is also required.

Benefits:

• Allata values differences and is committed to creating an inclusive environment for all employees.
• The company is an equal opportunity employer and makes employment decisions without regard to race, color, creed, religion, age, ancestry, national origin, veteran status, sex, sexual orientation, gender, gender identity, gender expression, marital status, disability, or any other legally protected category.
• This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation, and training.