Build, analyze, and repair training data and evaluation benchmarks
Models are as good as their their training data and evaluation benchmarks. Calibrion Data Lab is where teacher models, probability distributions, deep learning, and human experts come together to build or repair training datasets and evaluation benchmarks.
The loop is the product — click the animation to pause it.
Debug the data before using it
In a dataset of millions of examples, even a small percentage of defective examples can teach the model wrong behaviours. Recent research: Goodfire’s Anatomy of Post-Training showed that industry-standard post-training datasets implicitly taught models to erode their own safety guardrails, fabricate authoritative-looking links, and flatter users. These behaviors were traced to specific data clusters that are difficult to identify without the right tools.
Dataset defects are orders of magnitude cheaper to catch in the data than to discover in a trained model. Calibrion Data Lab exposes and repairs many of these problems early on, before they find their way into model training.
How Does Data Lab Work
The lab combines statistical tests, probability distributions, body of LLM-as-judges, and ML-based evaluators to examine a dataset from various angles.
Detecting & fixing bias
Detect various types of biases using ML and statistical models. Repair (when possible) or filter those biases to ensure an unbiased data.
Distributional analysis
Measure entropy, imbalance, coverage, and divergence from the target distribution to identify and fix overrepresented regions and missing data.
Anomaly detection
Statistical, probabilistic, and ML models surface unusual examples and clusters and flag or remove them from data to increase training accuracy.
Trained data evaluators
Task-specific models score safety, coherence, cadence, and other properties at example and corpus level.
Agent juries
Independent agents review the same data. Inter-agent agreement supports stable findings; disagreement is used as uncertainty signal and can trigger expert review.
Integrity and leakage
Detect duplicates, corruption, schema violations, train–test leakage, contamination, and PII before deeper analysis begins.
Synthesis
Describe the data you need and expert agents find and recommend public datasets that fit, calibrate an existing dataset to your specification, or build a new one from scratch or from a handful of seed examples.
From fixing issues to creating datasets
Diagnosis
Statistical tests, ML-based evaluators, and agent juries identify and localize bias, toxicity, anomalies, and other dataset defects.
Filter
Defective data the lab flags is removed.
Repair
Agents correct recoverable errors in labels, formatting, structure, and content.
Synthesis from description
Teacher models and agents create a dataset from a task description, specification, or seed examples.
Expand
Agents grow an existing dataset with new examples that close measured coverage gaps.
Calibrate
Distribution, balance, coverage, and difficulty are reshaped to fit a target domain, task, or use case.
Main dataset operations
Diagnose & Repair — diagnose dataset issues such as bias and repair them, or filter it into a smaller but better dataset.
Create & Grow — build a new dataset from scratch using a task description, specification, or seed examples, or expand an existing dataset.
Calibrate & Adapt — reshape an existing or public dataset for a target domain, task, or use case by adjusting its distributions, balance, coverage, and difficulty, while retaining what fits and filling what is missing.
Dataset Provenance & Auditability
We provide an extensive report documenting provenance, lineage, diagnostics, evaluator scores, and expert-review decisions. Filtering, repair, and synthesis operations are logged as transformations, creating an auditable record for reproducibility, governance, and reporting.
Bring your dataset to the lab.
Data Lab is in early access. Tell us what you need: dataset type, task, scale. Within one business day you get a spec, an indicative price, and sample datapoints from a practitioner.