About Us

Deccan AI Experts is a pioneering AI company founded by IIT Bombay and IIM Ahmedabad alumni, with a strong founding team from IITs, NITs, and BITS. We specialize in high-quality human-curated data, AI-first operations, and advanced AI evaluation systems.

About the Role

We are seeking a Data Quality Reviewer (Freelancer) to support advanced AI evaluation initiatives focused on data quality assurance, annotation review, dataset validation, AI benchmarking, and quality calibration.

In this role, you will review annotated datasets, evaluate AI-generated outputs, validate labeling accuracy, and ensure high-quality training and evaluation data for large language models (LLMs), computer vision, speech AI, and multimodal AI systems. Your expertise will help improve the reliability, consistency, and performance of next-generation AI models.

This position is ideal for professionals with experience in data quality assurance, annotation review, AI evaluation, quality auditing, content review, data operations, or analytical quality control.

Responsibilities

• Audit datasets to identify annotation inconsistencies, missing labels, duplicate records, formatting issues, taxonomy violations, and policy deviations.
• Compare annotations against project guidelines and gold-standard references.
• Review AI-generated responses for factual accuracy, logical reasoning, safety compliance, hallucinations, and overall response quality where applicable.
• Document quality trends, recurring error patterns, and reviewer performance metrics.
• Provide structured feedback and coaching recommendations to annotators and reviewers.
• Participate in calibration sessions to improve inter-rater reliability and annotation consistency.
• Collaborate with project leads to refine annotation guidelines and quality assurance processes.

Requirements

• Bachelor's degree in any discipline is preferred.
• 2+ years of experience in data quality assurance, annotation review, AI evaluation, quality auditing, content review, operations, or similar analytical roles.
• Reviewing annotations against detailed project guidelines.
• Identifying annotation errors, inconsistencies, and quality issues.
• Performing structured quality audits across large datasets.
• Applying scoring rubrics and documenting review decisions.
• Providing actionable feedback to improve annotation quality.
• Familiarity with annotation tools, spreadsheets, dashboards, and quality tracking systems.
• Strong analytical thinking and attention to detail.
• Excellent written English communication skills.
• Ability to make objective, evidence-based quality decisions.
• Ability to work independently in a remote, fast-paced environment.

Preferred Qualifications

• Experience working on RLHF (Reinforcement Learning from Human Feedback), LLM evaluation, response ranking, preference evaluation, prompt evaluation, or multimodal AI projects.
• Experience using annotation platforms such as Label Studio, Labelbox, Scale AI, SuperAnnotate, CVAT, Doccano, Prodigy, or similar tools.
• Familiarity with quality metrics such as precision, recall, agreement rates, inter-rater reliability, and calibration methodologies.
• Experience in content moderation, AI safety evaluation, policy enforcement, or benchmark dataset creation.
• Background in Computer Science, Linguistics, Data Science, Research, Psychology, Journalism, Education, or Operations is an advantage.
• Experience reviewing multilingual datasets is a plus.

Why Join Us

• Competitive hourly pay: ₹2,000/hour
• Fully remote with flexible working hours.
• Opportunity to contribute to cutting-edge AI evaluation and data quality initiatives.
• Exposure to advanced AI systems, benchmark development, and human feedback pipelines.
• Flexible project-based opportunities with global teams.
• Work on next-generation AI systems powering enterprise and consumer applications worldwide.