As a Prompt Engineer focused on Data Science and Quality Analysis, you will design, test, and evaluate prompts for AI systems that interact with real-world restaurant data.
You will work cross-functionally to develop AI solutions that drive operational efficiency, improve data interpretation, and support smarter decision-making for restaurant operators.
Your work will directly shape how AI models perform in high-stakes, dynamic environments like order processing, reporting, support automation, and performance analysis.
Essential job functions include developing, testing, and refining prompts for tasks such as text generation, question answering, data classification, and structured data extraction to optimize Voice AI performance.
You will design evaluation frameworks and analyze prompt outputs using quantitative metrics, human-in-the-loop evaluation, and user feedback to identify improvement opportunities.
Conduct experiments to test prompt variations, measure their business and operational impact, and iterate to enhance accuracy, consistency, and safety.
Build principled regression test suites using tools like LangFuse and Galileo to ensure prompts remain compliant and high-performing as models and use cases evolve.
Collaborate closely with data science, product, legal, engineering, and operations teams to align prompt designs with business goals, operational workflows, and compliance requirements.
Develop prompts across multiple LLMs (GPT, LLaMA, Gemini, and Checkmate’s fine-tuned models), understanding model differences to optimize outputs effectively.
Lead a team of analysts focused on prompt evaluation and data quality analysis, guiding prioritization, experimentation, and reporting.
Stay up to date on emerging prompting techniques, LLM behaviors, evaluation frameworks, and AI safety practices to keep Checkmate’s AI solutions best-in-class.
Requirements:
Strong analytical and data science skills, with hands-on experience in Python (pandas, NumPy, scikit-learn).
Experience designing and conducting experiments and evaluations in applied AI or NLP contexts.
Proficiency in SQL and working with relational databases (e.g. MySQL, PostgreSQL, Oracle, MS SQL).
Good understanding of data processing, quality measurement, and testing fundamentals.
Experience leading analyst or operations teams, with strong prioritization, mentorship, and collaboration skills.
Strong problem-solving mindset with a drive to explore, optimize, and automate workflows.
Excellent communication skills for presenting insights to technical and non-technical stakeholders.
Bachelor’s degree in Data Science, Computer Science, Statistics, Engineering, or a related field.
Flexible to work US hours until at least 6 pm ET, with a strong remote setup.
Benefits:
Health Care Plan (Medical, Dental & Vision) is provided.
A Retirement Plan (401k) is available for employees.
Life Insurance options include Basic, Voluntary & AD&D.
Flexible Paid Time Off is offered to support work-life balance.
Family Leave (Maternity, Paternity) is available for new parents.
Short Term & Long Term Disability coverage is provided.
Opportunities for Training & Development are available to enhance skills.
Work From Home options are supported to promote flexibility.
A Stock Option Plan is included as part of the compensation package.
