Senior Data Engineer

- Achieved 95% improvement in account visibility and reduced manual analysis effort by 70% by leading Azure Databricks and PySpark pipelines for automated NHI discovery across enterprise systems. - Productionized data integration solutions using Azure Data Factory, ADLS, and Delta Lake, ensuring alignment with enterprise architecture and business objectives. - Defined delta lake optimization techniques using liquid clustering to improve query performance and maintain effective partition pruning. - Engineered enterprise-scale data platform infrastructure for IAM datasets using Azure DevOps and YAML-based CI/CD workflows, enabling reliable deployments, controlled releases, and standardized infrastructure provisioning across environments. - Designed governed data access and lineage through Unity Catalog, improving data consistency, auditability and strengthening security protocols for NPA datasets. - Architected and enabled secure ADLS-to-Power BI integration services to deliver enterprise dashboards supporting application owners in identifying and mitigating NHI-related risks across 1,000+ applications. - Contributed to roadmap discussions for scalable architecture of pipeline, supporting future onboarding of additional security and infrastructure domains. - Acted as primary liaison with the bank’s Global Data Platform (GDP) team to design and implement scalable data infrastructure aligned with governance, compliance, and enterprise platform standards.

- Designed and maintained financial reporting data pipelines at Visa, a leading payment gateway with nearly 60% market share globally using PySpark, ADLS and Airflow. - Processed terabytes of governed financial datasets to improve reporting efficiency by 40% and enable faster payment trend analysis for BI stakeholders. - Collaborated with business analysts and finance stakeholders to design curated reporting datasets enabling deeper visibility into authorized payment volumes and underpenetrated market segments.

- Engineered a PySpark-based climate risk analytics platform processing multi-domain banking datasets across 11 climate risk categories, enabling scalable regulatory risk assessment and downstream reporting. - Optimized distributed Spark workloads across 25+ Hive datasets through partitioning, performance tuning, and query optimization, significantly reducing cost, runtime and improving cluster efficiency. - Designed low-code configuration tables for datasets, joins, and enrichment logic, accelerating new sector onboarding and simplifying codebase. - Established a configurable data quality and reconciliation framework with automated validation rules, preventing downstream failures and improving production reliability. - Adopted test-driven development (TDD) practices using Pytest to enhance code reliability and minimize production defects. - Improved process observability and governance by implementing metadata-driven monitoring and job history tracking, enabling faster root cause analysis and issue resolution.

- Built and operated 8+ production-grade Databricks pipelines processing 3.5 TB of enterprise retail data workloads on markets, brands and customer details. - Optimized the SQL queries for production incidents to reduce the execution time by 19 hours on production by performing root cause analysis. - Optimized Databricks cluster configurations, auto-scaling policies, and Spark resource utilization to improve workload performance, reduce execution bottlenecks, and enhance cost efficiency for large-scale retail data. - Automated Databricks and ARM deployments via Azure DevOps, YAML, and CI/CD, improving release consistency for enterprise ETL workloads in cloud infrastructure. - Designed Medallion lakehouse architecture in Azure Databricks, delivering curated and refined data marts for downstream analytics and reporting. - Led mentoring, code and engineering reviews for junior data engineers, establishing Databricks development standards, CI/CD best practices, and scalable production delivery patterns

- Built conversational flow analytics to identify user drop-off points and optimize chatbot interaction journeys through PySpark with the data size of 70 GB. - Originated an innovative NLP-integrated conversational AI platform with 80% accuracy as DialogFlow using Python, spaCy, MongoDB, and Redis. - Developed an enterprise integration platform with 50 pre-configured data source connectors, containerized with Docker and exposing data services through REST APIs for consistent deployment and scalability. - Enhanced NLP model performance by retraining dependency parser and POS tagging processes with enriched datasets, improving language processing efficiency by 30%.