Data Scientist

• Design, build, and evaluate Generative AI-powered analytics agents.
• Develop controlled experiments to evaluate AI agent performance.
• Form hypotheses and design statistically sound research methodologies.
• Measure model accuracy, repeatability, reliability, and failure modes.
• Analyze experimental results and communicate findings with quantitative evidence.
• Evaluate different LLM configurations including prompting strategies, model parameters, RAG, and agent architectures.
• Define analytical metrics and establish evaluation baselines.
• Create reproducible research documentation, structured experiment logs, and technical reports.
• Collaborate with Data Scientists, AI Engineers, Product teams, and business stakeholders.
• Present research findings and recommendations to technical and executive audiences.

• Bachelor's or Master's degree in Computer Science, Data Science, Statistics, Machine Learning, Artificial Intelligence, or a related discipline.
• 5+ years of experience in Data Science using:

• Python
• SQL
• Snowflake

• 3+ years of experience designing and conducting controlled experiments, including:

• Hypothesis testing
• Experimental design
• Baseline creation
• Bias reduction
• Statistical evaluation

• 3+ years evaluating ML models or AI systems using:

• Error analysis
• Variability measurement
• Reproducibility testing
• Performance benchmarking

• 1+ year of hands-on experience with Generative AI, including:

• LLMs
• Prompt Engineering
• AI Agents
• Retrieval-Augmented Generation (RAG)
• API-based model interaction
• Model parameter tuning (temperature, top-p, etc.)

• Strong understanding of:

• Statistical analysis
• Data quality
• Analytical metrics
• Experimental research

• Experience maintaining reproducible research documentation.
• Excellent communication and presentation skills.

• Experience evaluating Large Language Models (LLMs) for correctness, consistency, and repeatability.
• Experience with Tableau or other BI/Visualization platforms.
• Experience in Operations, Support, or IT Analytics.
• Experience presenting research findings to leadership or executive stakeholders.
• Familiarity with LangChain, LangGraph, CrewAI, AutoGen, or similar AI agent frameworks.

• Pay any fee to be considered for, submitted to, or selected for any opportunity. 

• Purchase any product, service, or gift cards from us or for us as part of an application, interview, or selection process. 

• Provide sensitive financial information such as credit card numbers or banking information. Successfully placed or hired candidates would only be asked for banking details after accepting an offer from us during our official onboarding processes as part of payroll setup. 

legal@pinnacle1.com