---
name: multi-agent-research-ideation
description: >
  Multi-agent pipeline for structured research hypothesis generation and evaluation.
  Orchestrates five specialized agent roles — Literature Scout, Idea Generator,
  Critical Reviewer, Experiment Designer, Synthesis Writer — to produce grounded,
  peer-reviewed research ideas from any scientific topic. Includes built-in
  comparison with single-agent baseline for measuring the value of role decomposition.
  Domain-agnostic: works for any field of science.
license: MIT license
metadata:
    skill-author: Claw, Sai Arava
---

# Multi-Agent Research Ideation Pipeline

## Overview

This skill implements a structured multi-agent pipeline that transforms a research topic into a peer-reviewed, experiment-ready research proposal. Five specialized agent roles collaborate through structured intermediate artifacts (JSON files with schemas), enabling reproducible, inspectable, and composable scientific ideation.

**Key contribution:** While existing skills wrap single tools (e.g., scanpy for single-cell analysis, RDKit for cheminformatics), this skill demonstrates *multi-agent orchestration* — explicit role decomposition with defined handoffs, structured intermediate artifacts, and built-in evaluation of the decomposition strategy itself.

**Pipeline overview:**
```
Topic String
    ↓
[1. Literature Scout] → literature_context.json (20-30 papers)
    ↓
[2. Idea Generator]   → candidate_ideas.json (3-5 ideas)
    ↓
[3. Critical Reviewer] → reviews.json (scores + ranking)
    ↓
[4. Experiment Designer] → experiment_plan.json (datasets, baselines, metrics)
    ↓
[5. Synthesis Writer]  → research_proposal.md (full proposal)
    ↓
[6. Comparison Eval]   → comparison_results.json (multi-agent vs single-agent)
```

## When to Use This Skill

Use this skill when:
- Starting research in a new or unfamiliar domain
- Generating multiple competing research directions to evaluate
- Conducting a structured literature-grounded ideation process
- Needing a reproducible record of how a research idea was developed
- Comparing multi-agent vs single-agent approaches to scientific ideation
- Demonstrating multi-agent orchestration patterns for AI-for-science workflows

## Quick Start

### Prerequisites

```bash
pip install -r requirements.txt
```

Required packages: `semanticscholar`, `requests`, `pandas`, `matplotlib`, `jsonschema`.

### Run the Full Pipeline

```bash
# Step 1: Search literature
python scripts/literature_scout.py "attention mechanisms for time series forecasting" \
    --limit 30 --output output/literature_context.json
```

Then follow Steps 2-6 below sequentially, using the agent prompt templates to guide each step.

## Pipeline Architecture

### Design Principles

1. **Agent-native execution:** The executing agent (you) performs all reasoning. Scripts handle only data I/O, API calls, and validation — no external LLM API keys required.

2. **Structured intermediates:** Every step produces a JSON file validated against a schema. This makes the pipeline debuggable (inspect any intermediate), resumable (re-run any step), and composable (swap implementations).

3. **Explicit role prompts:** Each agent role has a dedicated prompt template in `templates/prompts/` that defines its responsibilities, input format, output schema, and quality criteria.

4. **Built-in evaluation:** Step 6 compares multi-agent output against a single-agent baseline using automated metrics, providing empirical evidence for the value of role decomposition.

### File Dependencies

```
templates/prompts/scout.txt      → Step 1
templates/prompts/generator.txt  → Step 2
templates/idea_schema.json       → Step 2 (validation)
templates/prompts/reviewer.txt   → Step 3
templates/review_schema.json     → Step 3 (validation)
templates/prompts/designer.txt   → Step 4
templates/prompts/writer.txt     → Step 5
```

## Step-by-Step Workflow

### Step 1: Literature Scout

**Role:** Search and curate relevant academic literature.

**Execute:**
```bash
python scripts/literature_scout.py "<YOUR_TOPIC>" --limit 30
```

**What happens:** Queries Semantic Scholar API for papers matching your topic, sorted by citation count. Retrieves title, abstract, TLDR, authors, year, venue, citation count, and DOI/arXiv IDs.

**Output:** `output/literature_context.json` — array of 20-30 paper objects.

**Quality check:**
- Verify the results cover both seminal and recent work
- Check that different sub-areas of the topic are represented
- If coverage is poor, run additional queries with refined search terms

---

### Step 2: Idea Generator

**Role:** Analyze literature gaps and generate competing research ideas.

**Read the prompt template:**
```bash
cat templates/prompts/generator.txt
```

**Input:** Load `output/literature_context.json` and review the formatted papers:
```bash
python scripts/idea_generator.py show-literature
```

**Instructions:** Follow the prompt in `templates/prompts/generator.txt` to:
1. Identify gaps, contradictions, and unexplored directions in the literature
2. Generate 3-5 research ideas, each with: title, abstract, key insight, novelty claim, grounding papers, methodology sketch, and expected outcome
3. Save as `output/candidate_ideas.json`

**Validate output:**
```bash
python scripts/idea_generator.py validate output/candidate_ideas.json
```

**Quality criteria:**
- Each idea cites at least 2 papers from the literature
- Ideas are diverse (different aspects of the topic)
- Ideas are specific enough to design experiments around

---

### Step 3: Critical Reviewer

**Role:** Provide structured peer review of each candidate idea.

**Read the prompt template:**
```bash
cat templates/prompts/reviewer.txt
```

**Input:** Load `output/candidate_ideas.json` and `output/literature_context.json`.

**Instructions:** Follow the prompt in `templates/prompts/reviewer.txt` to:
1. Score each idea on 5 dimensions (Novelty, Rigor, Feasibility, Impact, Clarity) on a 1-10 scale
2. Write strengths, weaknesses, and suggestions for each idea
3. Produce a ranked list with recommendations
4. Save as `output/reviews.json`

**Validate output:**
```bash
python scripts/critical_reviewer.py validate output/reviews.json
```

**Quality criteria:**
- Reviews are specific, not generic
- Scores are calibrated (10s are rare and justified)
- Weaknesses are constructive with suggested fixes
- Rankings are consistent with scores

---

### Step 4: Experiment Designer

**Role:** Design a concrete experimental plan for the top-ranked idea.

**Read the prompt template:**
```bash
cat templates/prompts/designer.txt
```

**Input:** Identify the top idea:
```bash
python scripts/experiment_designer.py top-idea
```

**Instructions:** Follow the prompt in `templates/prompts/designer.txt` to design:
1. Research question (precise, testable)
2. Datasets (2-3 public benchmarks with URLs)
3. Baselines (3-5, including SOTA and simple baselines)
4. Metrics (3-5, with primary metric identified)
5. Method description (step-by-step procedure)
6. Expected outcomes (quantitative predictions)
7. Computational requirements
8. Integration of reviewer suggestions

**Save as:** `output/experiment_plan.json`

---

### Step 5: Synthesis Writer

**Role:** Compile all pipeline outputs into a coherent research proposal.

**Read the prompt template:**
```bash
cat templates/prompts/writer.txt
```

**Input:** Load all artifacts:
```bash
python scripts/synthesis_writer.py load
```

**Instructions:** Follow the prompt in `templates/prompts/writer.txt` to write a research proposal with:
1. Title, Abstract, Introduction, Related Work, Proposed Method, Experimental Design, Expected Results, References

**Generate references section:**
```bash
python scripts/synthesis_writer.py references
```

**Save as:** `output/research_proposal.md`

---

### Step 6: Comparison Evaluation

**Role:** Measure multi-agent pipeline quality with automated metrics.

**Execute:**
```bash
python scripts/comparison_eval.py --multi-agent-dir output/
```

**Metrics computed:**
- **Citation grounding rate:** Fraction of retrieved papers cited in the proposal
- **Idea specificity:** Average word count and unique terms per idea
- **Review depth:** Average suggestions and weaknesses per review
- **Internal consistency:** Correlation between review scores and stated strengths/weaknesses

**Optional single-agent comparison:**
```bash
# First, generate single-agent baseline outputs in a separate directory
python scripts/comparison_eval.py \
    --multi-agent-dir output/ \
    --single-agent-dir output_baseline/
```

**Output:** `output/comparison_results.json`

## Configuration

### Customizing the Pipeline

**Change the topic:** Simply provide a different topic string to Step 1.

**Adjust literature scope:**
```bash
# Narrow to recent papers
python scripts/literature_scout.py "your topic" --years "2023-2026" --limit 20

# Broader search
python scripts/literature_scout.py "your topic" --limit 50
```

**Modify evaluation criteria:** Edit `templates/prompts/reviewer.txt` to change the review dimensions or scoring rubric.

**Change number of ideas:** Edit `templates/prompts/generator.txt` and `templates/idea_schema.json` (adjust `minItems`/`maxItems`).

## Troubleshooting

### Semantic Scholar API Issues

**Rate limiting:** The free tier allows 100 requests per 5 minutes. If you hit limits:
- Wait 5 minutes and retry
- Request a free API key at https://www.semanticscholar.org/product/api

**No results:** Try broader search terms or remove year filters.

### Schema Validation Failures

**Ideas don't validate:**
```bash
python scripts/idea_generator.py validate output/candidate_ideas.json
```
Common issues: `abstract` too short (min 100 chars), `grounding_papers` needs at least 2 entries.

**Reviews don't validate:**
```bash
python scripts/critical_reviewer.py validate output/reviews.json
```
Common issues: scores outside 1-10 range, missing `ranking` array.

### Missing Dependencies

```bash
pip install -r requirements.txt
```

If `semanticscholar` fails to install:
```bash
pip install semanticscholar --no-deps
pip install requests tenacity
```

## Bundled Resources

### scripts/
- `literature_scout.py` — Semantic Scholar search with caching and sorting
- `idea_generator.py` — Literature formatting and idea validation
- `critical_reviewer.py` — Ranking computation and review validation
- `experiment_designer.py` — Top-idea extraction utilities
- `synthesis_writer.py` — Artifact loading and reference formatting
- `comparison_eval.py` — Automated quality metrics and comparison
- `utils.py` — Shared JSON I/O, logging, API client, schema validation

### templates/
- `idea_schema.json` — JSON Schema for candidate ideas (3-5 ideas, each with title, abstract, insight, grounding)
- `review_schema.json` — JSON Schema for reviews (5-dimension scores, strengths/weaknesses, ranking)
- `prompts/` — Role-specific prompt templates for each pipeline step

### examples/output/
Pre-computed example outputs from a sample run on "attention mechanisms for time series forecasting", demonstrating the full pipeline end-to-end.

## Additional Resources

- **Semantic Scholar API:** https://api.semanticscholar.org/
- **LabClaw skill library:** https://github.com/wu-yc/LabClaw
- **HypoGeniC (hypothesis generation):** https://github.com/ChicagoHAI/hypothesis-generation
- **Claw4S conference:** https://claw4s.github.io/