Curriculum-Aware Synthetic Data Generation for Mathematical Reasoning

1. Introduction

The last eighteen months have seen synthetic problem generation become the workhorse of math reasoning model training. Pipelines such as those described by [Yu et al. 2024] and [Toshniwal et al. 2024] produce millions of problems by prompting a strong teacher model with seed problems and accepting outputs that pass a verifier. These pipelines, however, generally sample uniformly from whatever the teacher emits, producing distributions skewed toward problems the teacher finds easy.

We argue that the shape of the difficulty distribution matters at least as much as its size, and that aligning it to a curriculum yields measurable gains.

2. Background

Classical curriculum learning [Bengio et al. 2009] presents easier examples first, gradually increasing difficulty. In LLM fine-tuning, however, the dynamics are more nuanced: gradients on too-easy examples produce vanishing learning signal, while problems above a model's current capability often produce confidently-wrong rationales that the model then memorizes [Zelikman et al. 2022].

We define problem difficulty operationally. Let $T$ be a teacher model and let $p \in \mathcal{P}$ be a generated problem. Define

$$d(p) = 1 - \frac{1}{K} \sum_{k=1}^{K} \mathbb{1}[T_k(p) = y^*(p)]$$

where $T_k$ are $K=8$ stochastic samples from the teacher and $y^*$ is the verified ground-truth answer. Difficulty $d(p)$ is bounded in $[0, 1]$ and discretized into ten bins.

3. Method

Our generator proceeds in three stages.

Stage A (Generation). A teacher emits raw candidate problems by mutation of a seed bank of 38,000 olympiad and textbook problems. We accept candidates whose verifier-checked answer matches the proposed solution.

Stage B (Difficulty estimation). We sample $K$ teacher rollouts per accepted problem, compute $d(p)$, and assign the problem to a difficulty bin.

Stage C (Curriculum resampling). We resample with replacement to match a target schedule $\pi_t(d)$ over training step $t$. We use a sigmoidal schedule

$$\pi_t(d) \propto \exp!\left( -\frac{(d - \mu_t)^2}{2 \sigma^2} \right), \qquad \mu_t = \mu_0 + (\mu_1 - \mu_0) \cdot t/T_{\max}$$

with $\mu_0 = 0.2$, $\mu_1 = 0.7$, $\sigma = 0.18$, advancing the curriculum from easy to hard over training.

def sample_batch(problems, step, total_steps, sigma=0.18):
    mu = 0.2 + 0.5 * (step / total_steps)
    weights = np.exp(-((problems.difficulty - mu) ** 2) / (2 * sigma ** 2))
    weights /= weights.sum()
    return np.random.choice(problems, size=batch_size, p=weights)

4. Experimental Setup

We fine-tune a 7B base model on 1.2M problems for three epochs with AdamW, lr $5 \times 10^{-6}$, cosine decay. The compute budget is fixed at 14,400 H100-hours; the curriculum-aware variant uses additional teacher samples for difficulty estimation, accounted for in this budget.

We compare four conditions: (i) Uniform — sample uniformly from accepted problems; (ii) Easy-first — train only on $d < 0.4$ problems; (iii) Hard-first — train only on $d > 0.6$; (iv) Curriculum — our method.

5. Results

The curriculum gain on Olympiad-bench is 4.7 absolute points ($p < 0.001$, paired bootstrap, $n = 4{,}000$). Confidence intervals are tight on GSM8K because the benchmark is near-saturated.

We also observed that the Hard-first condition exhibited training instability: roughly 11% of training steps produced gradient norms exceeding $3 \sigma$ of the running mean, and final loss curves had higher variance across seeds (CoV 0.18 vs. 0.07 for curriculum).

6. Failure Mode: Distributional Collapse

We noticed that beyond $d = 0.85$ the teacher's accepted problems exhibit reduced lexical diversity (token-2gram entropy drops 22%). This is consistent with the teacher reusing a small repertoire of hard templates. Models trained on this slice memorize template structure rather than learning generalizable reasoning. Practitioners should cap the hardest bin or supplement it with human-curated problems.

7. Discussion and Limitations

Our difficulty signal depends on the teacher's competence. If the teacher cannot solve a class of problems at all, $d$ saturates at 1.0 and the curriculum loses resolution at the top. A multi-teacher ensemble may mitigate this.

We have not tested transfer to non-mathematical reasoning domains; the curriculum hyperparameters were tuned on math and may need re-tuning for code or proof generation.

8. Conclusion

Difficulty-aware resampling is a low-cost addition to existing synthetic data pipelines and produces consistent gains on harder benchmarks. The mechanism — matching gradient signal to current model capability — is general, and we expect related schedules to help in adjacent domains.

References

1. Bengio, Y. et al. (2009). Curriculum learning.
2. Yu, L. et al. (2024). MetaMath: Bootstrap your own mathematical questions.
3. Toshniwal, S. et al. (2024). OpenMathInstruct.
4. Zelikman, E. et al. (2022). STaR: Self-taught reasoner.
5. Hendrycks, D. et al. (2021). Measuring mathematical problem solving with the MATH dataset.