Remote Sr Big Data Engineer - Oozie and Pig (GCP)

Description:

• We are seeking a Senior Big Data Engineer with deep expertise in distributed systems, batch data processing, and large-scale data pipelines.
• The ideal candidate has strong hands-on experience with Oozie, Pig, the Apache Hadoop ecosystem, and programming proficiency in Java (preferred) or Python.
• This role requires a deep understanding of data structures and algorithms, along with a proven track record of writing production-grade code and building robust data workflows.
• This is a fully remote position and requires an independent, self-driven engineer who thrives in complex technical environments and communicates effectively across teams.
• Key responsibilities include designing and developing scalable batch processing systems using technologies like Hadoop, Oozie, Pig, Hive, MapReduce, and HBase, with hands-on coding in Java or Python.
• The engineer will write clean, efficient, and production-ready code with a strong focus on data structures and algorithmic problem-solving applied to real-world data engineering tasks.
• The role involves developing, managing, and optimizing complex data workflows within the Apache Hadoop ecosystem, focusing on Oozie orchestration and job scheduling.
• The engineer will leverage Google Cloud Platform (GCP) tools such as Dataproc, GCS, and Composer to build scalable and cloud-native big data solutions.
• Implementing DevOps and automation best practices, including CI/CD pipelines, infrastructure as code (IaC), and performance tuning across distributed systems is also required.
• Collaboration with cross-functional teams to ensure data pipeline reliability, code quality, and operational excellence in a remote-first environment is essential.

Requirements:

• A Bachelor's degree in Computer Science, software engineering, or a related field of study is required.
• Experience with managed cloud services and understanding of cloud-based batch processing systems are critical.
• Proficiency in Oozie, Airflow, Map Reduce, and Java is necessary.
• Strong programming skills with Java (specifically Spark), Python, Pig, and SQL are required.
• Expertise in public cloud services, particularly in GCP, is essential.
• Proficiency in the Apache Hadoop ecosystem with Oozie, Pig, Hive, and Map Reduce is required.
• Familiarity with BigTable and Redis is preferred.
• Experience in Infrastructure and Applied DevOps principles in daily work is necessary, utilizing tools for continuous integration and continuous deployment (CI/CD), and Infrastructure as Code (IaC) like Terraform to automate and improve development and release processes.
• Proven experience in engineering batch processing systems at scale is required.
• Must have 5+ years of experience in customer-facing software/technology or consulting.
• Must have 5+ years of experience with “on-premises to cloud” migrations or IT transformations.
• Must have 5+ years of experience building and operating solutions built on GCP.
• Proficiency in Oozie and Pig is a must.
• Proficiency in Java or Python is required.

Benefits:

• The anticipated starting pay range for Colorado is $116,100 - $170,280.
• The anticipated starting pay range for Hawaii and New York (not including NYC) is $123,600 - $181,280.
• The anticipated starting pay range for California, New York City, and Washington is $135,300 - $198,440.
• The role may include variable compensation in the form of bonus, commissions, or other discretionary payments based on company and/or individual performance.
• Actual compensation is influenced by a wide array of factors including skill set, level of experience, licenses and certifications, and specific work location.
• Rackspace Technology offers a commitment to equal employment opportunity without regard to various legally protected characteristics.
• The company is recognized as a best place to work by Fortune, Forbes, and Glassdoor, attracting and developing world-class talent.