Observability & Evaluation Engineer

Job Title: Observability & Evaluation Engineer

Rate: 65-70/ Hr on C2C

Type -- Contract

Location: Charlotte, NC (3 days onsite)

Experience: 10+ Years

Job Description

We are looking for an experienced Observability & Eval Engineer to build and maintain monitoring, tracing, and evaluation solutions for AI/LLM applications and agent-based systems. The candidate will ensure AI applications perform reliably in production by implementing telemetry, dashboards, alerts, automated evaluations, and service-level objectives.

Key Responsibilities

• Implement monitoring, tracing, and telemetry for AI/LLM applications and AI agents.
• Develop evaluation frameworks to measure AI response quality, accuracy, and performance.
• Build dashboards, alerts, and metrics to monitor application health.
• Analyze prompt and model performance, latency, errors, and reliability.
• Define Service Level Objectives (SLOs) and support production readiness.
• Develop automated tests and evaluation suites using Python.
• Troubleshoot production issues and maintain operational runbooks.
• Collaborate with engineering and AI teams to improve system reliability.

Required Skills

• 10+ years of overall IT/software engineering experience.
• Strong Python programming and test automation skills.
• Hands-on experience with LLM/AI agent evaluation.
• Knowledge of observability, tracing, and telemetry.
• Experience building dashboards, alerts, and monitoring metrics.
• Understanding of SLOs, production operations, and incident troubleshooting.
• Familiarity with tools such as OpenTelemetry, Grafana, Prometheus, Datadog, LangSmith, or Langfuse.

Preferred Qualifications

• Experience supporting GenAI or agentic AI applications in production.
• Knowledge of prompt evaluation, model performance analysis, and regression testing.
• Experience with cloud platforms and CI/CD pipelines.

Ideal Candidate

A senior engineer with strong Python skills and practical experience in both AI/LLM evaluation and production observability, capable of ensuring AI systems are reliable, measurable, and production-ready.

Disclaimer: Diverse Lynx LLC is an Equal Opportunity Employer. All applicants and employees are evaluated without discrimination, based solely on their qualifications, ability, competence and performance. This email and its attachments may contain confidential or proprietary information and is intended only for the recipient(s). If you received this message in error, please disregard it and notify the sender. If you no longer wish to receive our communications, you may unsubscribe at any time.

Security Notice: Our official website is www.diverselynx.com We do not operate or authorize any other websites representing Diverse Lynx LLC.