Description:

Deepgram is seeking a Voice AI Evaluation Lead to benchmark and evaluate the performance of their voice AI models.
This role involves building robust benchmarking pipelines and producing actionable model cards.
The position requires collaboration with research, product, QA, marketing, and data labeling teams to shape model evaluation and improvement.
Responsibilities include maintaining scalable benchmarking pipelines for STT, TTS, and voice agent use cases, and running evaluations on real-world datasets.
The candidate will partner with various teams to develop new evaluation methodologies and integrate them into the development cycle.
The role also involves designing evaluation metrics that reflect product experience and performance goals.
The candidate will author comprehensive model cards and internal reports detailing model strengths and weaknesses.
Collaboration with Data Labeling Ops is necessary to prepare evaluation datasets, and with QA Engineers to integrate model tests into workflows.
The position supports marketing and product teams with data-backed comparisons to competitors and tracks market developments.

Requirements:

Candidates must have experience designing, executing, and iterating on evaluation pipelines for ML models.
Proficiency in Python and data analysis libraries is required.
The ability to develop automated evaluation systems is essential.
Comfort with large-scale datasets and crafting meaningful performance metrics and visualizations is necessary.
Experience using LLMs or internal tools for analysis, QA, or pipeline prototyping is preferred.
Strong communication skills are required, especially in translating raw data into structured insights.
Proven success in cross-functional collaboration with research, engineering, QA, and product teams is essential.

Deepgram offers the opportunity to work on cutting-edge technology in the AI industry.
The company is backed by prominent investors and has raised over $85 million in funding.
Deepgram promotes a collaborative and inclusive work environment, valuing diverse voices and perspectives.
The company is committed to providing accommodations for applicants who need them.