Looking back
In early 2025, we wrote about the coming data crisis for LLMs. Human-generated web data was projected to be running out, and the early attempts to fill the gap with LLM-generated data were producing disappointing results.
Eighteen months later, a lot has changed. One of those predictions aged well, the other one did not.
Data shortage remains a real challenge
The data shortage turned out to be a real challenge. Epoch AI's projection that models would exhaust the stock of human-generated public text between 2026 and 2032 remains the canonical estimate, and frontier labs have been licensing books and archives, racing to secure human expertise, and building entire supply chains around data in addition to compute. Moreover, naive synthetic data still fails**.** The model-collapse result which means recursively training on unfiltered machine-generated data degrades models, still holds.
Synthetic data made a comeback
By any measure, 2025–26 was the era when synthetic data made a comeback. It still does not look like a replacement for human data, but it started looking much more useful when it is targeted, filtered, and verified.
Microsoft's Phi-4 (released in December 2024) was already an early signal that a small model can achieve unusually strong reasoning performance with heavily synthetic training data. But the stronger evidence came after our original post. In 2025, Microsoft followed with Phi-4-reasoning, trained from carefully selected prompts and reasoning demonstrations generated by o3-mini. NVIDIA released Llama-Nemotron, a family of efficient reasoning models, together with a post-training dataset whose responses were synthetically generated and filtered. It also released OpenMathReasoning, a math-reasoning dataset with millions of generated solutions, and later the Nemotron Nano 2 training data, which included large synthetic pre-training and SFT-style components.
The shift is not limited to text. NVIDIA's Cosmos platform made synthetic data a major part of physical AI, with world foundation models for generating physics-aware video and simulated environments for robotics and autonomous vehicles. The newer Cosmos 3 work pushes the same idea further: synthetic data is increasingly central for training, testing, and evaluating AI systems.
What about the early model-collapse warnings then? Were they wrong? Not quite. Some parallel research showed that accumulating and mixing real human data with synthetic data can stabilize training and prevent collapse. Model collapse happens when an AI trains on low-quality, recursively generated, or unverified outputs. The newer successful recipes avoid this by generating targeted data, filtering aggressively, and verifying the outputs before they enter the training mix.
Synthetic data that works
At least three practices separate the synthetic data that works from the synthetic data that leads to model collapse:
Expand, don't replace. Model collapse seems to happen when synthetic data fully replaces real data. But if you keep real data in the mix, the researchers at Stanford and MIT now show that accumulation alongside real data avoids collapse. The practical recipe that emerged: a small seed of high-quality human examples, a much larger volume of synthetic generations grounded in it, and a floor of real data that never goes away.
Filter ruthlessly. An unfiltered synthetic dataset can be worse than a smaller, verified one. The teams who get good results with data synthesis treat data generation as the easy half and spend their effort on verification: automated checks where outputs are checkable (does the code run? does the proof verify?), statistical analysis, model-based scoring, and human review where it matters most.
Keep experts in the loop, but move them up the stack. This is the deepest shift. Human expertise didn't become less important; it moved from writing data to judging it. Verification is faster than generation, which is exactly why a small group of experts can now handle a large synthetic pipeline.
What was shown to be a dead-end by some research in mid 2024 might have been only missing a powerful quality layer to allow only high-quality generations to make their way into the final results.
Data quality is the real bottleneck
This is where synthetic data starts to look much more like human data than people sometimes assume. Human-written data is not automatically good, and synthetic data is not automatically bad. In both cases, the useful question is the same: does this example improve the model, is it in the right distribution, and does it avoid teaching the wrong behavior?
A note on how we approach data quality at Calibrion
At Calibrion, we believe in the importance of data quality.
Our platform quality panel combines distributional analysis and statistical checks for coverage, drift, duplication, and imbalance with panels of LLMs-as-judges, plus uncertainty and entropy mapping. In practice, this sits close to established ideas such as dataset cartography, influence functions, data Shapley, confident learning, and data-selection methods for instruction tuning. Human-in-the-loop review sits at the escalation layer for ambiguous, high-impact, or domain-sensitive cases.
None of these layers is enough on its own. The point is to combine them into a quality system: automated filters for scale, model-based judges for semantic review, entropy and distributional methods for mathematical verification, and human review only where judgment or domain expertise is necessary.
We believe data quality is becoming a crucial layer in its own right. Catching dataset issues early helps models learn expected behaviors, reduces unknown or unsafe failure modes, and prevents expensive rework later in the training flow. As training and post-training become more widespread, retraining a bad model is not only costly and slow; it also carries considerable compute and environmental footprint.
In practice, it is more efficient to remove bad training examples before they enter the dataset than to compensate later with guardrails, classifiers, and evaluation harnesses around a model that has already learned from flawed data.
What this means if you're fine-tuning models
If you're an enterprise or AI team creating training or evaluation data, the current playbook is:
- Don't prompt a frontier model for tens of thousands of examples and call it a dataset. That will cause model collapse.
- Start from a real seed (e.g. your documents, your tickets, your experts' answers) and then let the data generator build from there.
- Plan and budget more for verification than generation. Automated checks, statistical vetting, and expert review are crucial.
Today, almost any team can create large-scale synthetic data in-house by leveraging frontier models. Steps 1 and 2 are increasingly easy. Step 3 is the hard part. Turning raw generations into trusted training data requires evaluation design, domain rubrics, sampling strategy, automated checks, and expert review loops. The bottleneck is no longer producing examples. It is knowing which examples are good enough to teach the next model.
A note on the market: synthetic data became a product category
Although market estimates should always be treated cautiously, the direction is clear. One recent forecast puts the synthetic data generation market at roughly 46% annual growth between 2025 and 2035. That is not just because teams want more data. It is because they need data they can generate under constraints: privacy, domain coverage, rare cases, safety testing, simulation, and evaluation.
NVIDIA is a useful signal here as well. Alongside its open datasets and Cosmos work, it was reported to have acquired Gretel, a company focused on synthetic data generation and privacy-preserving data workflows. Whether the use case is LLM post-training, enterprise fine-tuning, robotics, or autonomous systems, synthetic data is becoming something companies buy, build, govern, and evaluate as part of their AI stack.