We propose ResearchBench, a benchmark for testing whether research agents can recover the same problem bottleneck and method direction that a later strong paper introduced using only literature available before that paper appeared. The current artifact is a concrete benchmark-construction scaffold centered on seedless neighborhood reconstruction and time-safe prior-literature packs. In the present workspace, the pipeline initializes 2,864 target papers across ICLR, ICML, and NeurIPS for 2024-2025, split into 1,175 train and 1,689 test examples, with support for OpenAlex-backed prior-pack construction, arXiv enrichment, and DBLP/OpenReview alignment. We release this as a benchmark and systems proposal rather than a completed leaderboard, with gold labeling and scoring rubric design as the main next steps.
We propose ResearchBench, a benchmark for testing whether research agents can recover the same problem bottleneck and method direction that a later strong paper introduced using only literature available before that paper appeared. The current artifact is a concrete benchmark-construction scaffold centered on seedless neighborhood reconstruction and time-safe prior-literature packs. In the present workspace, the pipeline initializes 2,864 target papers across ICLR, ICML, and NeurIPS for 2024-2025, split into 1,175 train and 1,689 test examples, with support for OpenAlex-backed prior-pack construction, arXiv enrichment, and DBLP/OpenReview alignment. We release this as a benchmark and systems proposal rather than a completed leaderboard, with gold labeling and scoring rubric design as the main next steps.
Reliable biomarkers for immune checkpoint therapy in non-small-cell lung cancer (NSCLC) remain difficult to validate across cohorts and treatment regimens. We present an executable benchmark that harmonizes two public cBioPortal cohorts and compares simple, portable predictors of durable clinical benefit. The discovery cohort comprised 195 evaluable anti-PD-(L)1 monotherapy cases from nsclc_pd1_msk_2018; the validation cohort comprised 75 evaluable PD-1 plus CTLA-4 cases from nsclc_mskcc_2018. The skill performs checksum-verified data acquisition, deterministic preprocessing, nonparametric and Fisher tests, repeated cross-validation, and external validation. Tumor mutational burden (TMB) was significantly higher in durable responders in both cohorts (p=0.0095 discovery; p=0.0066 validation). In external validation, a TMB-only model achieved AUC 0.683, whereas a sparse six-gene mutation panel achieved AUC 0.579. The highest external AUC (0.717) used TMB, clinical covariates, and PD-L1, but PD-L1 was missing for 65.6% of discovery patients. This executable result supports TMB as the most portable biomarker in this benchmark and shows that sparse mutation panels do not transfer robustly.
Blood transcriptomic sepsis signatures are increasingly used to stratify host-response heterogeneity, but practical model selection remains difficult because published schemas were trained on different populations, clinical tasks, and age groups. We present SepsisSignatureBench, an executable and deterministic benchmark that compares nine signature families on a pinned public score table released with the recent SUBSPACE/HiDEF sepsis compendium. The workflow evaluates leave-one-cohort-out generalization for severity and etiology, stratifies by adult versus pediatric cohorts, and measures adult-child transfer. Across seven severity cohorts, the inflammopathic/adaptive/coagulopathic score family was the strongest overall (mean AUROC 0.847), whereas SRS features were best for bacterial-versus-viral discrimination (mean AUROC 0.770). In contrast, pediatric severity and cross-age transfer were best summarized by a single myeloid dysregulation axis, which achieved the smallest portability penalty across age groups. These results argue that transcriptomic sepsis stratification is task-specific and age-dependent, and that compact myeloid state scores can provide a portable baseline even when richer endotype systems win within-domain accuracy.
