LLM-as-judge has become the de facto evaluator for open-ended generation, but the calibration of its confidence scores has received less scrutiny than its accuracy. We collect 38,400 judge decisions across nine LLM judges spanning 1.
This paper investigates the relationship between self improvement and llm agents through controlled experiments on 14 diverse datasets totaling 22,801 samples. We propose a novel methodology that achieves 30.
Overparameterized neural networks are widely believed to gracefully handle label noise because their excess capacity can absorb corrupted examples without degrading clean-sample performance. We directly test this assumption by training 2,400 models spanning four architectures (ResNet-18, VGG-16, DenseNet-121, ViT-Small) at five width multipliers (0.
Multi-agent LLM systems chain multiple model instances via natural language, but scaling properties are unknown. We study 2-16 agents across four patterns (sequential, broadcast, hierarchical, peer-to-peer).
Distributed LLM training suffers from straggler nodes that impose synchronization barriers. We analyze 2,400 training runs on clusters of 10-512 GPUs with data/tensor/pipeline parallelism.
We investigate how membership inference attack success covaries with neural network model size and overfitting. Using the shadow model approach of Shokri et al.
We investigate how adversarial robustness scales with model capacity in small neural networks. Using 2-layer ReLU MLPs with hidden widths from 16 to 512 neurons (354 to 265{,}218 parameters), we train on two synthetic 2D classification tasks (concentric circles and two moons) and evaluate robustness under FGSM and PGD attacks across five perturbation magnitudes (\varepsilon \in \{0.
For a fixed parameter budget, should one build a deep-narrow or shallow-wide MLP? We systematically sweep depth (1--8 hidden layers) against width across three parameter budgets (5K, 20K, 50K) on two contrasting tasks: sparse parity (a compositional boolean function) and smooth regression.
We re-analyze published benchmark data from BIG-Bench (8 tasks, 3 model families) and MMLU (13 models, 5 families) to test the claim by \citet{schaeffer2023} that emergent abilities in large language models are artifacts of discontinuous evaluation metrics. By applying both discontinuous (exact string match) and continuous (partial credit) metrics to the same published performance data, we quantify the \emph{Metric Sensitivity Index} (MSI) for each task and add deterministic bootstrap uncertainty estimates.
We re-analyze published benchmark data from BIG-Bench (8 tasks, 3 model families) and MMLU (13 models, 5 families) to test the claim by \citet{schaeffer2023} that emergent abilities in large language models are artifacts of discontinuous evaluation metrics. By applying both discontinuous (exact string match) and continuous (partial credit) metrics to the same published performance data, we quantify the \emph{Metric Sensitivity Index} (MSI) for each task and add deterministic bootstrap uncertainty estimates.