Description:

The Data Engineer will join the Personalization’s Hulk squad at Spotify, focusing on producing Human Understandable Language Knowledge to enhance content understanding.
The team utilizes Large Language Models to comprehend podcasts and audiobooks, creating reliable and scalable systems to share knowledge with Spotify's internal teams, users, and creators.
Responsibilities include working with large-scale data pipelines using frameworks like Scio, Hendrix ML, Ray, PyTorch, and Google Cloud Platform offerings.
The role involves supporting streaming and batch inference of core LLM models and developing new data pipelines to meet emerging needs while maintaining existing systems.
The Data Engineer will collaborate with ML engineers to create training and evaluation datasets for Generative LLM models and own the development of data solutions from architecture to delivery.
The position requires collaboration with backend and data engineers, ML engineers, data scientists, researchers, and other stakeholders.
The engineer will develop monitoring tools to track AI product quality and work in an agile team to experiment, iterate, and deliver on new product objectives.
Knowledge sharing and promoting best practices through mentorship and accountability are also key aspects of the role.

Requirements:

Candidates must be passionate about data and proficient in data modeling, access, and storage optimization.
Experience in building high-scale production data pipelines with frameworks such as Scio is required, with additional knowledge of Ray being a plus.
Familiarity with orchestration frameworks like Flyte is preferred.
A strong understanding of agile software processes, data-driven development, reliability, and responsible experimentation is essential.
The ideal candidate values collaboration and partnership within teams.
Experience or curiosity about ML data processing, inference pipelines, LLM fine-tuning, and using LLMs for data analysis is necessary.
Bonus points for experience with near real-time systems and prompt engineering for LLMs, including tools like DSPy.

The position offers a competitive salary range of $160,091 to $228,702, plus equity.
Benefits include health insurance, a six-month paid parental leave, and a 401(k) retirement plan.
Employees receive a monthly meal allowance, 23 paid days off, 13 paid flexible holidays, and paid sick leave.
Spotify promotes inclusivity and provides reasonable accommodations during the recruitment process to support all candidates.