Remote Senior Lead Data Engineer (GCP)

Description:

• The Senior GCP Data Engineer is responsible for preparing big data infrastructure for analytical or operational uses.
• This role involves developing and maintaining scalable data pipelines and building new API integrations to support increasing data volume and complexity.
• The engineer will collaborate with data scientists and business teams to enhance data models that feed business intelligence tools, promoting data accessibility and data-driven decision-making across the organization.
• Responsibilities include collaborating with cross-functional teams to define, prioritize, and execute data engineering initiatives aligned with business objectives.
• The engineer will design and implement scalable, reliable, and secure data solutions following industry best practices and compliance requirements.
• They will drive the adoption of cloud-native technologies and architectural patterns to optimize performance, cost, and reliability of data pipelines and analytics solutions.
• The role includes mentoring and leading a team of Data Engineers and demonstrating a commitment to learning new technologies and techniques.
• Strong problem-solving skills are essential, with an emphasis on building data-driven or AI-enhanced products.
• Coordination with ML/AI and engineering teams to understand data requirements is also a key responsibility.

Requirements:

• Candidates must have 8+ years of strong experience in ETL and ELT data from various sources in Data Warehouses.
• A minimum of 8+ years of experience in Python, Pandas, Numpy, and SciPy is required.
• Candidates should have at least 5+ years of experience in GCP and BigQuery, PySpark, and Pub/Sub.
• Experience in creating data architectures for 5+ years is necessary.
• Certification as a Google Cloud Professional Data Engineer is required.
• Advanced proficiency in Google Cloud services such as Dataflow, Dataproc, Dataprep, Data Studio, and Cloud Composer is essential.
• Proficiency in writing complex Spark (PySpark) User Defined Functions (UDFs), Spark SQL, and HiveQL is required.
• A good understanding of Elastic search is necessary.
• Candidates should have experience in assessing and ensuring data quality, data testing, and addressing data quality issues.
• An excellent understanding of Spark architecture and underlying frameworks, including storage management, is required.
• A solid background in database design and development, database administration, and software engineering across full life cycles is necessary.
• Experience with NoSQL data stores like MongoDB, DocumentDB, and DynamoDB is required.
• Knowledge of data governance principles and practices, including data lineage, metadata management, and access control mechanisms, is essential.
• Experience in implementing and optimizing data security controls, encryption, and compliance measures in GCP environments is necessary.
• Candidates must have the ability to troubleshoot complex issues, perform root cause analysis, and implement effective solutions in a timely manner.
• Proficiency in data visualization tools such as Tableau, Looker, or Data Studio to create insightful dashboards and reports for business users is required.
• Strong communication and interpersonal skills are essential for effective collaboration with technical and non-technical stakeholders.
• Experience with agile methodologies and project management tools like Jira or Asana for sprint planning, backlog grooming, and task tracking is necessary.

Benefits:

• Experience.com offers a dynamic work environment that values every experience.
• The position allows for remote work flexibility, with a base location in Chennai.
• Employees are part of a company that is backed by reputable investors and has a strong focus on impactful business outcomes.
• The role provides opportunities for professional growth and development through mentorship and leadership responsibilities.
• Employees can expect to work with cutting-edge technologies and contribute to innovative data solutions.
• The company promotes a culture of collaboration and data-driven decision-making, enhancing overall job satisfaction.