Remote Data Engineer

Description:

• The position is for a Data Engineer on a 6-month contract inside IR35, located in the UK.
• The role is fully remote, allowing for flexible working arrangements.
• Candidates must have active SC Clearance for vetting purposes.
• The job requires a strong understanding of data concepts, including data types, data structures, and schemas (both JSON and Spark).
• Experience with complex JSON manipulation and working with data pipelines using custom Python/PySpark frameworks is essential.
• The candidate should have a solid understanding of the four core data categories: Reference, Master, Transactional, and Freeform, with a focus on managing Reference Data.
• Knowledge of data security principles, including data ownership, access controls, and GDPR compliance, is crucial, especially in handling sensitive datasets.
• Strong problem-solving and analytical skills are required, with an emphasis on intuitive demonstration of these abilities.

Requirements:

• Candidates must be proficient in JSON and YAML.
• Strong programming skills in Python are required, with Pydantic experience considered a bonus.
• Proficiency in SQL and PySpark is necessary.
• Experience with Delta Lake and Bash (both CLI usage and scripting) is required.
• Familiarity with Git and Markdown is essential.
• Knowledge of Scala is a bonus but not compulsory.
• Experience with Azure SQL Server as a HIVE Metastore is considered a bonus.
• Candidates should have experience with Azure Databricks, Apache Spark, and Delta Tables.
• Proficiency in data processing with Python and integration/data ingestion using PowerBI is required.
• Familiarity with JIRA is also necessary.

Benefits:

• The position offers a fully remote working environment, providing flexibility for work-life balance.
• The contract duration is for 6 months, allowing for a defined period of engagement.
• The role is inside IR35, which may provide certain tax benefits depending on the candidate's situation.
• Active SC Clearance is a requirement, which may enhance the candidate's professional credentials and opportunities in the future.