Machine Learning Evaluation Engineer

About Us
We are Pepoltek, an IT company building a portfolio of SaaS products used across education, HR, and enterprise operations, including BIDYATek, a school management platform for institutions across Bangladesh, our Intelligent HR Solution, a cloud-based HRMS used by growing companies, and an Enterprise ERP suite for supply chain and financial operations. Alongside our own products, we work with software development partners across the globe to extend our engineering capacity on their behalf. 

We are hiring a Machine Learning Evaluation Engineer for one of these partners, a research-driven AI startup in New York building the memory and identity layer for agentic systems, to own evaluation of their core product end to end.

The Role
You'll own evaluation of a complex multi-agent system end to end, the instrument that tells a research-driven team whether their identity representations are actually getting better. The product works, and it has real black-box behavior, so the job is defining what better means for representations that change over time, without ground truth, and building the machinery that measures it. If you love constructing measurements from nothing, reading traces to find what's actually broken, and shipping the fix yourself, this role is built for you.

What You'll Do

• Design the evals, defining the scores and what better means for an entity representation that changes over time, and keeping that definition current as the product and methods move
• Build the pipelines and harnesses, including data ingestion, labels, versions, reruns, and judges, so the team can ask a new question this week and get an answer this week
• Run the evals and harvest insights, reading traces and results to find what's actually broken and proposing fixes that produce higher-fidelity representations
• Build simulation agents at scale, raising the ceiling on how many entities can be modeled and measured at once
• Own the loop end to end, from the question through the pipeline, the rerun, and the writeup, without handing work off or waiting on someone else's sprint

What You'll Bring

• 3+ years designing and owning LLM evaluation systems
• Experience designing, building, and running evals end to end for LLMs
• Production-quality Python, including tests, maintenance, and release cadence
• Hands-on experience with PyTorch, Hugging Face, and Weights and Biases

Nice to Have

• Work on memory, identity, personalization, agent evaluation, or agentic optimization
• A shipped research record, as first or co-author at NeurIPS, ICML, ICLR, or equivalent, or self-published open-source work of comparable quality
• Data pipeline tooling experience, such as Hydra, Kafka, or SQL
• A degree in computer science, mathematics, statistics, or a related quantitative field

Growth
You will own evaluation for the product end to end, shaping what better means for the company's core measurement problem, with direct influence over the research direction and methods the team relies on.

Compensation and Perks

• $220,000 to $300,000 base salary
• Competitive equity

Work Authorization
Candidates must have current work authorization in the United States without need for employer sponsorship.

Job Details

• Employment type: Full time
• Location: New York, NY
• Workplace: On-site, 5 days in-office
• Vacancies: 1

How to Apply
Apply with your updated Resume and a Cover letter or Motivation letter. We only move forward with candidates who clearly show us what they bring to the table.