Description:

PayPay is seeking a Data Engineer for the Data Insights department to support the rapid expansion of its product teams and the need for a robust Data Engineering Platform.
The mission of the Data Insights department is to drive product improvements through a scientific understanding of user and merchant behavior.
The Data Engineer will be responsible for designing, developing, and maintaining scalable data ingestion pipelines using AWS Glue, Step Functions, Lambda, and Terraform.
Responsibilities include optimizing and managing large-scale data pipelines for high performance, reliability, and efficiency.
The role involves implementing data processing workflows using Hudi, Delta Lake, Spark, and Scala.
The Data Engineer will maintain and enhance Lakeformation and Glue Data Catalog for effective data management and discovery.
The position requires designing, building, and maintaining infrastructure to support the improvement and deployment of machine learning models.
Collaboration with cross-functional teams is essential to ensure seamless data flow and integration across the organization.
Best practices for observability, data governance, security, and compliance must be implemented.

Requirements:

Candidates must have 5+ years of experience as a Data Engineer or in a similar role.
Hands-on experience with Apache Hudi, Delta Lake, Spark, and Scala is required.
Experience in designing, building, and operating a DataLake or Data Warehouse is necessary.
Knowledge of Data Orchestration tools such as Airflow, Dagster, or Prefect is essential.
Strong expertise in AWS services, including Glue, Step Functions, Lambda, and EMR, is required.
Familiarity with change data capture tools like Canal, Debezium, and Maxwell is needed.
Experience with data warehousing tools like AWS Athena, BigQuery, or Databricks is necessary.
Proficiency in Python and SQL (any variant) is required, with preferable experience in Scala and/or Java.
Experience with data cataloging and metadata management using AWS Glue Data Catalog, Lakeformation, or Unity Catalog is essential.
Proficiency in Terraform for infrastructure as code (IaC) is required.
An overall understanding of machine learning technologies and deep learning concepts is necessary.
Strong problem-solving skills and the ability to troubleshoot complex data issues are essential.
Excellent communication and collaboration skills are required.
The ability to work in a fast-paced, dynamic environment and manage multiple tasks simultaneously is necessary.

The position offers a full-time employment status with the option to work remotely from anywhere in Japan.
Employees enjoy super flex time with no core hours, typically working from 10:00 am to 6:45 pm.
Paid leave includes annual leave (up to 14 days in the first year) and personal leave (5 days each year).
An annual salary is paid in 12 installments, based on skills, experience, and abilities, with annual reviews and a special incentive based on company performance.
Additional benefits include a late overtime allowance, a Work from Anywhere allowance (JPY 100,000), and the option for digital salary payment through "PayPay Paycheck."
Employees are provided with social insurance (health insurance, employee pension, employment insurance, and compensation insurance), a 401K plan, translation/interpretation support, and VISA sponsorship with relocation support.