Remote Databricks, Senior Data Engineer

Description:

• Zencore is a fast-growing company founded by former Google Cloud leaders, architects, and engineers.
• The company is seeking candidates with significant experience in Google Cloud to join their team.
• The role involves working with innovative companies, leveraging Databricks as a key driver of transformation.
• As a member of the Data Engineering team, you will act as a trusted advisor to customers, assisting them on their journey with Databricks.
• Responsibilities include covering a wide range of technology topics such as ETL/ELT, Data Warehousing, Distributed Data Processing, Streaming Data systems, Data Modeling, Data Security & Governance, and BI/AI.

Requirements:

• Candidates must have 5+ years of experience in a data engineering capacity.
• A minimum of 2+ years of experience working with Databricks is required.
• Preferred candidates will hold the Databricks’ Data Engineer Associate or Data Engineer Professional certification.
• Familiarity with the Databricks tech stack related to data engineering, including notebooks, clusters, jobs, workflows, serverless SQL, unity catalog, repos, dashboards, and DLT is necessary.
• Experience in developing production Spark applications is essential.
• Strong skills in Python, PySpark, and Spark SQL are required.
• A strong understanding of Delta Lake technology and best practices for applying its capabilities during the data pipeline development lifecycle is needed.
• Experience with Spark Structured Streaming and Databricks Delta Live Tables (DLT) is required.
• Advanced working knowledge of SQL and experience with relational databases, including query authoring (SQL) and familiarity with various databases is necessary.
• Understanding of incremental update patterns and technologies available on Databricks, such as CDC, SCD#2, autoloader, change data feed (CDF), DLT, and Delta Lake ACID is required.

Benefits:

• Employees can trust the professionalism and extensive experience of their colleagues to get the job done.
• The company values unbiased opinions and encourages sharing hard truths as subject matter experts for customers.
• Zencore acts as a missing link, connecting customers to Google and facilitating quick solutions.
• The company aims to remove uncertainty and risk by clearly outlining the cloud migration process for customers.
• Zencore accelerates timelines and focuses on overcoming challenges quickly, exposing "unknown unknowns" for customers.
• The company fosters an empathetic environment, building trust with customers by relating to their complexities.