AI Engineer Agentic Systems & RLAIF

As an AI Engineer you will own the engineering behind that path: detecting capability and context failures before they become customer-visible, converting root-cause analysis into one-factor variants, running them through hard gates and calibrated AI judges, and promoting only what is approved, versioned, monitored, and reversible. Today, small errors compound across interconnected agents, missing capabilities and stale business context silently produce fluent-but-wrong answers, and prompt, policy, and model changes lack one governed path to independent proof.

Role Name: AI Engineer – Agentic Systems & RLAIF
Work Site: Dallas, TX (Onsite)

**AI Engineer – Agentic Systems & RLAIF**
## **About the Role**
We are building an RLAIF (Reinforcement Learning from AI Feedback) Plane: a governed learning system that turns production evidence from Client's customer-facing agents into better agent decisions—safely, measurably, and reversibly. Today, small errors compound across interconnected agents, missing capabilities and stale business context silently produce fluent-but-wrong answers, and prompt, policy, and model changes lack one governed path to independent proof.
As an AI Engineer you will own the engineering behind that path: detecting capability and context failures before they become customer-visible, converting root-cause analysis into one-factor variants, running them through hard gates and calibrated AI judges, and promoting only what is approved, versioned, monitored, and reversible.
## **What You Will Do**
• **Build the agent runtime.** Design, build, and operate production agents and multi-agent workflows using LangGraph, LangChain, and deep-agent patterns—governed, monitored, and versioned from day one.
• **Instrument for observability.** Implement tracing, telemetry, and failure taxonomies across agents, tools, memory, and knowledge sources so every failure has a reproducible trail back to its root cause.
• **Engineer the evaluation harness.** Own golden sets, offline and online evals, and AI-judge pipelines with order-swap bias control and human calibration loops; report against business outcomes, not just model metrics.
• **Drive harness optimization.** Use observed failures to improve prompts, tools, skills, and rules through one-change-at-a-time experiments with regression-safe release: hard gates, approval, canary, and rollback.
• **Separate the optimization paths.** Keep harness, contextual-policy, and model-level changes distinct and independently evaluated; build the mechanisms that detect capability and knowledge-coverage gaps and propose governed updates.
• **Ship on GCP.** Deploy and scale the RLAIF Plane on Google Cloud (Vertex AI, GKE/Cloud Run, BigQuery, Pub/Sub) with CI/CD, IaC, and cost/latency controls.
• **Collaborate and lead.** Partner with architects, data scientists, SRE, and Client business stakeholders; document decisions, mentor engineers, and raise the bar on agent engineering practice.
## **Key Responsibilities**
- Ingesting and analyzing traces
- Reconstructing the sequence of events, tool calls, and business flow from traces
- Creating prompt/context/routing variants
- Defining and implementing evaluation dimensions
- Building and calibrating AI judges
- Designing scoring and reward-signal frameworks
- Generating autonomous optimization recommendations
- Identifying root causes such as:
- Hallucinations
- Context drift
- Poor prompts
- Misrouting
- Policy/compliance issue
- Safety: Harmlessness, different biases, toxicity
- Missing or underperforming tools
- Taxonomy/ontology gaps
- Re-configuration opportunities (agent) for identification and recommendation and project outcomes.
## **What You Bring (Required)**
• Demonstrable experience building and running AI agents in production—shipped systems serving real users, with ownership of reliability, latency, cost, and incident response.
• Hands-on experience with AI observability and evals: tracing frameworks (e.g., LangSmith, Arize Phoenix, OpenTelemetry), LLM-as-judge design, golden-set curation, and regression testing.
• Deep, practical experience with LangGraph and LangChain, including deep-agent patterns (planning, sub-agents, tool use, memory, human-in-the-loop, and state persistence).
• Strong Python engineering skills; comfortable with async services, APIs, containers, and testing discipline in a CI/CD environment.
• Working understanding of RLAIF/RLHF concepts, prompt and policy optimization, and how to isolate a single causal change in a complex system.
• 5+ years of software engineering experience, including 2+ years focused on LLM or agentic applications.
## **Preferred Qualifications**
• Cloud experience on Google Cloud (Vertex AI, GKE, Cloud Run, BigQuery, Pub/Sub, IAM); GCP certification is a plus.
• Experience with retrieval, enterprise knowledge bases, and memory systems with provenance, trust scoring, and decay.
• Familiarity with governed release practices in regulated or large-enterprise settings (approval workflows, canary, rollback, audit trails).
• Telecom or large-scale customer-service domain exposure; experience with MLOps tooling and feature flags.