We are sharing a specialised full-time opportunity for experienced counsel-level attorneys with substantial legal practice experience and the ability to evaluate, structure, and improve advanced AI systems working on complex professional legal tasks.
This role works closely with AI research and programme teams to improve how advanced models reason about substantive legal work. Selected attorneys will review legal tasks and model outputs, develop instruction specifications and reference solutions, design challenging benchmarks, and translate experienced legal judgement into clear, reproducible evaluation standards.
Key Responsibilities
Legal Quality Review & Analysis
- Review complex legal knowledge-work tasks and AI-generated outputs for substantive accuracy and professional quality
- Identify weak reasoning, overlooked issues, unsupported conclusions, and analyses that would not withstand professional legal scrutiny
- Evaluate legal work for completeness, precision, practical relevance, and appropriate application of legal principles
- Assess model performance across sophisticated legal scenarios
- Provide precise written feedback explaining errors, omissions, and required improvements
Instruction Design & Reference Solutions
- Develop detailed instruction specifications for advanced legal AI tasks
- Produce high-quality reference solutions that establish clear standards for correct legal reasoning
- Design tasks reflecting authentic workflows encountered in professional legal practice
- Translate practitioner judgement into explicit, teachable, and reproducible evaluation criteria
- Ensure reference materials demonstrate appropriate substantive depth and analytical rigour
Legal Benchmark Development
- Design challenging legal benchmarks and evaluation sets
- Create tasks testing issue spotting, substantive knowledge, analytical reasoning, and professional judgement
- Develop criteria that distinguish rigorous legal analysis from plausible but incomplete responses
- Refine evaluation materials based on model performance and research requirements
- Help develop legal-specific AI capabilities, evaluation methodologies, and supporting tools
Substantive Legal Expertise
- Apply specialist knowledge from one or more practice areas such as corporate and transactional law, litigation and dispute resolution, regulatory and compliance, intellectual property, employment and labour law, or tax
- Evaluate complex matters using professional methodologies and applicable legal frameworks
- Identify nuanced legal issues that may be overlooked by generalist reviewers
- Apply judgement developed through direct ownership of significant legal matters
- Maintain standards consistent with experienced counsel-level practice
AI Research Collaboration & Calibration
- Collaborate with AI researchers, programme teams, and specialists across adjacent professional domains
- Participate in calibration processes to maintain consistent evaluation standards
- Apply hands-on experience with large language models to professional legal workflows
- Distinguish genuinely well-reasoned responses from outputs that are persuasive but substantively incorrect
- Communicate complex legal concepts clearly to technical and non-legal stakeholders
Ideal Profile
- Juris Doctor (JD) from an accredited law school
- Approximately 8–15 years of substantive post-qualification legal practice
- Professional experience within an established law firm, corporate legal department, regulatory body, court, or comparable legal institution
- Current or previous seniority at Counsel, Senior Associate, Senior In-House Counsel, Assistant General Counsel, or a comparable level
- Genuine specialisation in at least one substantive legal practice area
- Relevant expertise may include corporate and transactional law, litigation and dispute resolution, regulatory and compliance, intellectual property, employment and labour, or tax
- Active admission to at least one U.S. state bar and good standing
- Direct ownership of significant matters, transactions, disputes, regulatory work, or advisory responsibilities
- Hands-on professional experience using large language models or AI tools
- Strong ability to distinguish rigorous legal reasoning from plausible but technically incorrect analysis
- Excellent written communication and ability to provide precise, structured feedback
- Must live in the San Francisco Bay Area or be willing to relocate there before the engagement begins
- Must be able to work onsite with the assigned team multiple days per week when required
Engagement Details
- Full-time W-2 employment position
- Initial engagement of approximately 6 months
- 40 hours per week
- Hybrid working arrangement based in the San Francisco Bay Area
- Regular onsite participation is required when requested
- Candidates outside the Bay Area must be willing to relocate at their own expense before starting
- Relocation assistance is not provided
- Compensation: $75–$110/hour
- Client-issued accounts and equipment will be provided as required
- Work will be conducted within assigned internal tools and workflows
- Responsibilities and project scope may evolve based on research and programme requirements
- Work must be completed without using confidential or proprietary information belonging to any current or former employer, client, institution, or other third party
About the Platform
This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.
By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy.