Remote Data Engineer (Databricks+ ADF)

Description:

• Allata is a global consulting and technology services firm with offices in the US, India, and Argentina.
• The company helps organizations accelerate growth, drive innovation, and solve complex challenges by combining strategy, design, and advanced technology.
• Allata's expertise includes defining business vision, optimizing processes, and creating engaging digital experiences.
• The firm architects and modernizes secure, scalable solutions using cloud platforms and top engineering practices.
• Allata empowers clients to unlock data value through analytics and visualization and leverages artificial intelligence to automate processes and enhance decision-making.
• The Data Engineer will contribute to transformative enterprise data platform projects focused on developing data pipelines and logic engines for managing ingest, staging, and multi-tier data product modeling.
• Responsibilities include data enrichment using various OEM-specific data warehouse and data lake house platform implementations for consumption via analytics clients.
• The role requires full life cycle design, build, deployment, and optimization of data products for multiple large enterprise industry vertical-specific implementations.

Requirements:

• Candidates must have current knowledge of modern data tools like Databricks, FiveTran, Data Fabric, and core experience with data architecture, data integrations, data warehousing, and ETL/ELT processes.
• Applied experience with developing and deploying custom whl and/or in-session notebook scripts for custom execution across parallel executor and worker nodes is required.
• Proficiency in SQL, Stored Procedures, and Pyspark based on the area of data platform specialization is necessary.
• Strong knowledge of cloud and hybrid relational database systems, such as MS SQL Server, PostgreSQL, Oracle, Azure SQL, AWS RDS, Aurora, or a comparable engine is essential.
• Candidates should have strong experience with batch and streaming data processing techniques and file compactization strategies.
• Automation experience with CI/CD pipelines to support deployment and integration workflows, including trunk-based development using automation services such as Azure DevOps, Jenkins, or Octopus is a plus.
• Advanced proficiency in Pyspark for advanced data processing tasks and spark workflow optimization and orchestration using tools such as Asset Bundles or DAG orchestration is also advantageous.
• Strong teamwork, communication skills, and intellectual curiosity to work collaboratively with cross-functional teams are important.
• A commitment to delivering high-quality, accurate, and reliable data product solutions is required.
• Candidates should be willing to embrace new tools, technologies, and methodologies and be innovative thinkers with a proactive approach to overcoming challenges.

Benefits:

• Allata values differences and is an equal opportunity employer, celebrating diversity and creating an inclusive environment for all employees.
• The company makes employment decisions without regard to race, color, creed, religion, age, ancestry, national origin, veteran status, sex, sexual orientation, gender, gender identity, gender expression, marital status, disability, or any other legally protected category.
• This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation, and training.