We are sharing a specialised part-time consulting opportunity for detail-oriented evaluators with experience interpreting JSON-formatted outputs, applying structured grading rubrics, and assessing human data or AI evaluation tasks consistently.
This urgent project supports a short-term evaluation sprint focused on grading AI-generated outputs against detailed task criteria. Selected professionals will review written instructions, interpret JSON-based responses, and assess each output independently using predefined rubrics. Strong accuracy, consistency, and careful attention to small details are essential.
Key Responsibilities
JSON Output Review
- Read, parse, and interpret AI-generated outputs presented in JSON format
- Understand nested fields, structured values, and relationships between output components
- Identify missing, inconsistent, malformed, or incorrectly represented information
- Assess whether each response satisfies the underlying task instructions
Rubric-Based Evaluation
- Review detailed task descriptions and corresponding grading criteria
- Evaluate each JSON output against predefined rubric requirements
- Apply scoring standards consistently across high-volume task batches
- Distinguish minor issues from material failures based on the provided criteria
- Document grading decisions clearly where explanations are required
Quality & Consistency
- Maintain strong attention to detail across repetitive evaluation workflows
- Identify edge cases, ambiguous outputs, and subtle inconsistencies
- Follow project instructions precisely without introducing unsupported assumptions
- Escalate unclear rubric language or conflicting task requirements appropriately
- Incorporate calibration feedback into subsequent evaluations
Independent Assessment
- Complete all assigned assessments without assistance from AI tools
- Manage tasks independently within short turnaround windows
- Review completed work before submission for accuracy and completeness
- Maintain reliable quality throughout the duration of the sprint
Ideal Profile
Strong candidates may have:
- Experience reading and interpreting JSON files or JSON-formatted data
- Familiarity with structured grading rubrics and evaluation criteria
- Prior experience as a grader, reviewer, rater, annotator, or quality evaluator
- Strong written reasoning and attention to detail
- Ability to apply complex instructions consistently across multiple tasks
- Comfort working within fast-paced, high-volume evaluation workflows
- Availability to contribute approximately 20–40 hours per week
- Ability to complete assigned tasks within approximately 12–24 hours
- Current residence in the United States
Educational Background
- A degree in computer science, data analysis, information systems, research, operations, linguistics, or a related discipline may be helpful
- Formal technical qualifications are secondary to practical JSON fluency and strong evaluation experience
- Training in quality assurance, data annotation, structured review, or AI evaluation may strengthen an application
- Equivalent professional or project-based experience may also be considered
Nice to Have
- Extensive experience designing or applying detailed rubrics
- Previous work on human data or AI model evaluation projects
- Familiarity with JSON syntax, nested objects, arrays, and structured data validation
- Background in quality assurance, content review, operations, or data labelling
- Experience completing calibration exercises and maintaining reviewer consistency
- Prior experience working on similar structured evaluation programmes
- Ability to identify subtle grading errors and ambiguous task conditions
- Experience handling urgent, short-duration project sprints
Why This Opportunity
- Apply JSON and rubric-evaluation skills to advanced AI research
- Contribute to a focused, high-priority evaluation sprint
- Work with clearly defined task instructions and grading criteria
- Build experience in structured model-output assessment
- Access potential extension opportunities based on project needs and performance
Contract Details
- Independent contractor role
- Fully remote within the United States
- Immediate availability is required
- Initial sprint expected to run through approximately 27–28 July, with possible extension
- Expected commitment of approximately 20–40 hours per week
- Tasks should generally be completed within 12–24 hours
- Competitive rates between $45–$170 per hour depending on experience, task complexity, and project scope
- Desktop or laptop computer required
- Chromebook devices are not supported
- Assessments must be completed without assistance from AI tools
- Weekly payments via Stripe or Wise
- Projects may be extended, shortened, or adjusted depending on operational requirements and performance
- Work will not involve access to confidential or proprietary information from any employer, client, or institution
About the Platform
This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.
By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy.