Senior Data Engineer

- Lead and develop end-to-end data engineering projects, aligning data architecture with critical business goals. - Apply data governance best practices to ensure high-quality, secure, and compliant data assets. - Utilize Delta Tables for reliable data storage, transactional integrity (ACID), and schema versioning. - Implement real-time data streaming pipelines using Change Data Capture (CDC), Apache Kafka, and PySpark. - Build, schedule, and manage Apache Airflow DAGs for robust workflow orchestration and monitoring. - Train and mentor team members, fostering a collaborative culture of continuous technical and professional growth. - Advocated for Git and GitHub best practices, leading version control, branch management, and CI/CD integrations. - Implemented rigorous data quality validations across pipelines using the SodaCL library. - Developed scalable backend APIs using Python and FastAPI to provide rapid, secured access to curated datasets. - Deployed and orchestrated containerized services and pipelines using Docker and Kubernetes.