Job Description: AI SDET - AI Testing / Test Automation
Location: Sunnyvale, CA
Job Title: AI Software Development Engineer in Test (AI SDET)
Rate: $55 -$65/hr
Job Summary
We are seeking an experienced AI Software Development Engineer in Test (AI SDET) to design, develop, and implement comprehensive testing strategies for AI-powered applications, machine learning models, and Generative AI systems. The ideal candidate will have strong experience in software test automation, API testing, Python programming, and AI/ML testing methodologies.
The candidate will be responsible for validating the functionality, performance, reliability, accuracy, and security of AI-driven solutions, including Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) pipelines, AI agents, and intelligent automation workflows. This role requires close collaboration with software developers, data scientists, machine learning engineers, and product teams to ensure high-quality AI products.
Key Responsibilities
- Design, develop, and maintain scalable automated test frameworks for AI/ML applications using Python and modern automation tools.
- Create and execute test plans, test cases, and test strategies for AI-powered applications, APIs, and backend services.
- Validate Generative AI and LLM-based applications for accuracy, relevance, consistency, hallucinations, bias, toxicity, and response quality.
- Develop automated evaluation frameworks to measure AI model performance, precision, recall, latency, and reliability.
- Test RAG-based applications, including document ingestion, chunking, embedding generation, vector database retrieval, and response generation.
- Perform API testing, integration testing, end-to-end testing, regression testing, and system testing.
- Build automation scripts using Python, Pytest, Selenium, Playwright, or equivalent testing frameworks.
- Validate AI model outputs using predefined datasets, evaluation metrics, assertions, and quality thresholds.
- Test AI agents and tool-calling workflows for correct decision-making, tool selection, execution, and error handling.
- Conduct performance, load, stress, and scalability testing for AI services and inference APIs.
- Integrate automated tests into CI/CD pipelines using Jenkins, GitHub Actions, GitLab CI, or similar tools.
- Test AI application security, including prompt injection, jailbreak attempts, sensitive data exposure, and unsafe model outputs.
- Analyze test failures, identify root causes, report defects, and collaborate with engineering teams to resolve issues.
- Develop test datasets, edge-case scenarios, negative test cases, and adversarial test scenarios for AI systems.
- Monitor test results, maintain test documentation, and continuously improve automation coverage and testing efficiency.
Required Technical Skills
- Strong programming experience in Python and object-oriented programming.
- Hands-on experience with test automation frameworks such as Pytest, Selenium, Playwright, or Robot Framework.
- Experience with REST APIs, JSON, HTTP protocols, Postman, and API automation.
- Knowledge of AI/ML concepts, including model inference, training and validation data, evaluation metrics, and model behavior.
- Experience testing Generative AI applications, LLM-powered solutions, or AI-based APIs.
- Familiarity with AI evaluation techniques, including exact-match scoring, semantic similarity, groundedness, relevance, and hallucination detection.
- Understanding of RAG architecture, embeddings, vector databases, and retrieval quality.
- Experience with SQL, databases, and test data management.
- Knowledge of Git, Linux/Unix, debugging, and software development life cycle methodologies.
- Experience with CI/CD tools and automated build and deployment pipelines.
- Strong analytical, problem-solving, communication, and collaboration skills.
Preferred Qualifications
- Experience with AI testing and evaluation tools such as Ragas, DeepEval, promptfoo, or LangSmith.
- Familiarity with LLM APIs and frameworks such as OpenAI APIs, Hugging Face, LanEligible to workhain, or LlamaIndex.
- Experience with machine learning libraries such as scikit-learn, PyTorch, or TensorFlow.
- Familiarity with cloud platforms such as AWS, Microsoft Azure, or Google Cloud.
- Experience with Docker, Kubernetes, distributed systems, and microservices.
- Knowledge of AI safety, responsible AI, model bias evaluation, privacy, and security testing.
- Experience building reusable test utilities, synthetic test data generators, and automated quality dashboards.
- Familiarity with observability tools, model monitoring, and production AI evaluation.
Education and Experience
- Bachelor''s or Master''s degree in Computer Science, Software Engineering, Information Technology, or a related field, or equivalent practical experience.
- Typically 3-8 years of experience in software quality engineering, SDET, test automation, or software development, with relevant experience in AI testing.
- Demonstrated ability to build and maintain automated testing solutions for complex software systems.
- Experience working in Agile development environments and cross-functional engineering teams.
Key Success Metrics
- Increased automated test coverage for AI-powered applications.
- Improved detection of AI-specific defects, hallucinations, and reliability issues.
- Reduced regression testing time and faster release cycles.
- Improved model evaluation consistency and reproducibility.
- Reliable performance, scalability, security, and quality of AI applications in production.
Ideal Candidate Profile
The ideal candidate combines traditional SDET expertise with emerging AI testing capabilities. They should be comfortable writing Python automation code, testing APIs and distributed systems, designing AI evaluation datasets, and assessing the quality and reliability of LLM-powered applications. The candidate should be proactive in identifying edge cases and capable of translating complex AI behaviors into measurable, repeatable test scenarios.
Key skills to highlight on your resume
- Programming: Python, SQL, scripting
- Test automation: Pytest, Selenium, Playwright, API automation
- AI testing: LLM evaluation, hallucination testing, prompt testing, AI quality metrics
- AI frameworks: RAG, LanEligible to workhain, LlamaIndex, Ragas, DeepEval
- API and backend: REST APIs, Postman, microservices
- DevOps: Git, Jenkins, CI/CD, Docker
- Testing types: Functional, integration, regression, performance, security, and end-to-end testing
Disclaimer: Diverse Lynx LLC is an Equal Opportunity Employer. All applicants and employees are evaluated without discrimination, based solely on their qualifications, ability, competence and performance. This email and its attachments may contain confidential or proprietary information and is intended only for the recipient(s). If you received this message in error, please disregard it and notify the sender. If you no longer wish to receive our communications, you may unsubscribe at any time.
Security Notice: Our official website is www.diverselynx.com We do not operate or authorize any other websites representing Diverse Lynx LLC.