Abstract

Current Retrieval-Augmented Generation (RAG) systems face a fundamental completeness-precision dilemma: vector-based approaches optimize for precise needle-in-haystack retrieval but sacrifice comprehensive context through isolated chunk retrieval, while knowledge graph systems aim for completeness but suffer from query specificity challenges and complex traversal overhead. We present Topological RAG, a graph-based architecture that reconstructs semantic "small worlds" through weighted multi-hop traversal, prioritizing comprehensive corpus coverage over retrieval speed. Our comparative evaluation against Dust (commercial vector RAG) using 54 civic discourse questions from the French Grand Débat National reveals fundamental architectural trade-offs between two paradigms: GraphRAG's "reconstruct-then-refine" approach produces 144× more output tokens (~28K vs ~195) through exhaustive small-world reconstruction, while Dust's "retrieval-then-synthesis" approach optimizes for direct answer extraction. On a 24-question subset, Dust achieves higher single-shot precision (0.57 vs 0.09) and answer relevance (0.91 vs 0.60), while GraphRAG demonstrates conservative boundary honesty—explicitly acknowledging when queried data falls outside reconstructed contexts rather than forcing speculative answers. GraphRAG maintains 100% success rate versus Dust's 44% due to rate limiting, and guarantees 100% corpus coverage across 20 communes. These results suggest architectural selection depends on application requirements: Dust for single-shot precision within indexed content, GraphRAG for comprehensive reconstruction with explicit uncertainty acknowledgment in high-stakes domains.

Introduction

Retrieval-Augmented Generation has emerged as a critical paradigm for grounding large language models in external knowledge sources, enabling accurate, up-to-date responses beyond parametric memory [Edge et al., 2024; Peng et al., 2024]. However, contemporary RAG systems navigate a fundamental architectural trade-off: optimizing for retrieval precision often sacrifices contextual completeness, while pursuing comprehensive coverage incurs substantial latency and complexity penalties.

Vector-based RAG systems, dominant in production deployments, employ dense semantic embeddings to retrieve top-K similar text chunks [Xu et al., 2024]. This approach excels at precision—quickly identifying passages most semantically aligned with user queries—but inherently fragments knowledge by isolating chunks from their broader relational context. A query about retirement policy in civic discourse may retrieve relevant passages mentioning pension amounts, yet miss critical relationships to tax reform proposals, demographic trends, or regional variations that constitute the complete "small world" of retirement discourse.

Conversely, knowledge graph approaches preserve explicit entity-relationship structures through nodes and edges [Xue & Zou, 2022; Dong et al., 2023]. While promising for comprehensive reasoning, traditional KG-based RAG faces challenges: (1) query specificity requirements that demand exact entity matching, limiting corpus coverage [Xie et al., 2024]; (2) graph traversal complexity that scales poorly with multi-hop exploration [Chang & Zhang, 2024]; and (3) the interpretability-performance trade-off, where transparent provenance chains come at the cost of response latency.

Motivation: Civic Discourse Analysis

The French Grand Débat National of 2019 exemplifies knowledge-intensive domains demanding both precision and completeness. This nationwide civic consultation generated 50 Cahiers de Doléances (citizen contribution notebooks) from communes in Charente-Maritime, containing 8,000+ extracted entities spanning policy proposals, institutional actors, thematic concepts, and citizen opinions. Analyzing such discourse requires systems that can rapidly respond to specific fact-checking queries ("What retirement amount was mentioned in contribution #4?"), comprehensively capture semantic neighborhoods for exploratory analysis ("What themes connect retirement proposals across communes?"), and maintain provenance from LLM responses back to original citizen contributions for interpretability. Traditional vector RAG achieves the first requirement but struggles with semantic neighborhood capture and provenance maintenance. Standard KG traversal achieves comprehensive capture and provenance but sacrifices rapid response times. This completeness-precision dilemma motivates our architectural innovation.

Topological RAG: Small-World Reconstruction

We introduce Topological RAG, a graph-based retrieval architecture leveraging small-world network principles [Watts & Strogatz, 1998]. Rather than retrieving isolated top-K chunks or executing complex graph queries, our approach reconstructs complete semantic neighborhoods—"small worlds"—through three mechanisms. First, dual-strategy seeding combines community-based thematic context with global entity search to achieve 92.7% corpus coverage compared to a 16% single-strategy baseline. Second, weighted multi-hop traversal expands seed entities via 5-hop Dijkstra traversal with relationship-type weights (CONCERNE: 1.0, APPARTIENT_À: 0.3), preventing graph explosion while maintaining semantic coherence. Third, ontological coverage verification ensures reconstructed small worlds span all 12 core civic entity types (PROPOSITION, THÉMATIQUE, SERVICE_PUBLIC, and others), guaranteeing completeness for downstream LLM synthesis. This architecture embodies two key insights: topologically complete semantic neighborhoods enable LLMs to synthesize contextually rich responses without hallucinating missing connections, and pre-computed graph indices combined with in-memory traversal eliminate the latency penalties traditionally associated with KG-based retrieval.

Research Contributions

This work makes four contributions. First, from an architectural perspective, we present a novel topological RAG system operationalizing small-world reconstruction through dual-strategy retrieval, weighted traversal, and ontological verification. Second, empirically, we provide a rigorous comparative evaluation revealing latency-quality trade-offs between graph-based (GraphRAG) and vector-based (Dust) RAG architectures using 54 civic discourse questions and 8 semantic quality metrics. Third, theoretically, we hypothesize that GraphRAG's completeness advantage positions it favorably for iterative, multi-turn queries where initial small-world capture enables subsequent precise retrieval. Fourth, practically, we provide evidence that extreme use cases prioritizing comprehensiveness and interpretability—such as legal, medical, and civic domains—benefit from topological architectures despite single-shot precision gaps. The remainder of this paper proceeds as follows: the Related Work section reviews ontologies, knowledge graphs, and RAG architectures; the Methodology section details our experimental design and topological graph implementation; the Results section presents comparative findings; the Discussion section addresses architectural implications and limitations.

Related Work

Ontologies for Knowledge Capture

Ontology engineering has long served as a foundation for structured knowledge representation, particularly in expert systems requiring explicit domain modeling [Fernández del Amo et al., 2024]. Recent work demonstrates ontologies' utility in capturing tacit expertise across diverse domains: construction digital twins [Boje et al., 2020], fault diagnosis systems [Fernández del Amo et al., 2024], and building management services [Schneider et al., 2020]. These approaches emphasize knowledge formalization through taxonomies, axioms, and constraints that enable automated reasoning.

The Gene Ontology exemplifies large-scale ontology development for biological knowledge management, maintaining 2,838 GO-CAMs (gene ontology context and modeling) through systematic expert annotation [Carbon et al., 2020]. Similarly, domain-specific ontologies have emerged for legal knowledge [Huet et al., 2020], occupational safety [Pandithawatta et al., 2023], and cultural heritage preservation [Lu et al., 2023]. However, ontology-based systems often struggle with scalability and maintenance costs as domain knowledge evolves, motivating more flexible knowledge graph representations.

Knowledge Graphs for Expert Knowledge Management

Knowledge graphs extend ontologies by combining structured schemas with instance-level data, enabling both formal reasoning and flexible querying [Xue & Zou, 2022]. Enterprise applications span supply chain management [Uniyal et al., 2020], workplace safety [Chen et al., 2023], and marine science [Wu et al., 2022]. A critical challenge identified across these domains is knowledge graph quality management—ensuring completeness, consistency, and accuracy as graphs scale [Xue & Zou, 2022; Wang et al., 2022].

Domain-specific KG construction typically involves: (1) entity extraction from unstructured text using named entity recognition [Wu et al., 2022; Xiao & Zhang, 2021], (2) relationship extraction through BiLSTM-CRF or transformer architectures [Xiao & Zhang, 2021], and (3) schema design capturing expert-defined entity types and relationship semantics [Pandithawatta et al., 2023]. For civic discourse analysis, this translates to extracting policy proposals, institutional actors, and thematic concepts from citizen contributions while preserving their semantic relationships.

However, most KG applications focus on structured data management rather than retrieval-augmented generation, leaving a gap in understanding how KGs enhance LLM-based question answering specifically.

Retrieval-Augmented Generation Architectures

The RAG paradigm emerged to ground LLMs in external knowledge without retraining [Lewis et al., 2020]. Three architectural families have evolved:

Vector-Based RAG {-}

Dense retrieval via semantic embeddings (e.g., sentence transformers, BGE) enables fast top-K chunk selection [Xu et al., 2024]. Production systems like Dust.tt demonstrate sub-second retrieval for single-shot queries but sacrifice contextual completeness through isolated chunk retrieval. Recent work on WeKnow-RAG combines web search with vector retrieval to improve factual accuracy [Xie et al., 2024], yet maintains the chunk-centric paradigm.

Knowledge Graph-Enhanced RAG {-}

Recent work integrates KGs with RAG through multiple strategies. KG-RAG [Dong et al., 2023] uses knowledge graphs for educational tutoring, achieving 35% assessment score improvements by grounding LLM responses in structured curricula. CommunityKG-RAG [Chang & Zhang, 2024] leverages community detection to improve fact-checking through zero-shot reasoning over KG subgraphs. Biomedical applications demonstrate particular promise: KRAGEN [Matsumoto et al., 2024] achieves 71% performance boosts for Llama-2 by converting SPOKE biomedical KGs into vector databases, while OntologyRAG [Feng et al., 2025] accelerates code mapping through ontology-backed retrieval.

Graph-RAG {-}

The most relevant prior work, GraphRAG [Edge et al., 2024], constructs graph indices with LLM-generated community summaries for query-focused summarization. Our approach differs in three ways: (1) GraphRAG uses entity extraction at query time; we employ pre-constructed domain graphs with verified ontological coverage; (2) GraphRAG focuses on global summarization; we optimize for local small-world reconstruction; (3) GraphRAG demonstrates advantages for sensemaking tasks but lacks empirical comparison on single-shot precision versus vector RAG.

TKG-RAG [Wei et al., 2024] introduces text-chunk knowledge graphs similar to our chunk-as-node architecture, but without weighted traversal or ontological verification. DO-RAG [Opoku et al., 2025] addresses domain-specific QA but relies on runtime KG construction rather than pre-computed indices.

Research Gap

Despite extensive work on ontologies, knowledge graphs, and RAG systems, no prior work systematically addresses the completeness-precision dilemma through topological small-world reconstruction with provenance guarantees. Existing approaches exhibit three limitations: vector RAG systems optimize for precision through vector similarity while sacrificing contextual completeness, KG-RAG systems pursue comprehensive reasoning at the cost of query latency and specificity requirements, and GraphRAG systems focus on global summarization rather than local semantic neighborhood capture. Our contribution bridges this gap by demonstrating that weighted multi-hop traversal with ontological verification enables comprehensive context capture through topologically complete small-world reconstruction achieving 92.7% corpus coverage, with explicit uncertainty acknowledgment when queries exceed reconstructed boundaries. Critically, our comparative evaluation with vector RAG reveals distinct architectural paradigms—reconstruct-then-refine (144× more output tokens) versus retrieval-then-synthesis—informing architectural selection for different use cases rather than universal superiority claims.

Methodology

Systems Under Comparison

We compare two RAG architectures applied to French civic discourse analysis:

Dust RAG (Vector-Based) {-}

The commercial retrieval-augmented generation platform from Dust.tt employs dense semantic embeddings for document indexing and vector similarity search for top-K chunk retrieval. The system uses the GPT-5-nano language model with temperature fixed at 0.7 by the platform, operates under a 120-second timeout constraint, and queries across the full corpus through a conversational API with polling-based response retrieval.

GraphRAG MCP (Graph-Based) {-}

Our topological RAG system, built on the nano-graphrag framework, utilizes pre-constructed knowledge graphs in GraphML format encompassing 28 semantic relationship types. The system implements dual-strategy retrieval combining community and global entity search, performs weighted 5-hop Dijkstra traversal with ontological verification, and employs the GPT-5-nano language model at temperature 1.0 due to model constraints. GraphRAG operates under the same 120-second timeout and queries across all 50 communes simultaneously through MCP (Model Context Protocol) server deployment.

GraphRAG Surgical Variant {-}

For exhaustive corpus analysis, a "surgical" architecture queries N communes in parallel via asyncio.gather(). Each commune independently performs: (1) vector search for seed entities, (2) 5-hop weighted graph traversal, (3) per-commune LLM synthesis, and (4) response aggregation. This mode prioritizes comprehensiveness over speed, ensuring 100% corpus coverage at the cost of O(N) retrieval overhead. The surgical endpoint (grand_debat_query_all_surgical) returns both individual per-commune responses and an aggregated synthesis, enabling fine-grained provenance tracing.

Configuration Asymmetry {-}

Temperature settings differ due to platform constraints---Dust operates at 0.7 (fixed by platform), GraphRAG at 1.0 (GPT-5-nano model constraint). This asymmetry may affect response variability: higher temperature produces more diverse outputs, while lower temperature yields more deterministic responses. This limitation is documented as a threat to internal validity (see Threats to Validity below).

Topological Graph Architecture

Our GraphRAG implementation embodies a topological approach to knowledge retrieval through small-world network principles. The architecture consists of four key components:

Data Structure

GraphML Representation {-}

Knowledge is encoded as directed graphs where entities serve as nodes representing 8,000+ extracted entities with attributes including entity_name, entity_type, description, source_id, and commune. Semantic relationships form edges spanning 28 relationship types such as CONCERNE, APPARTIENT_A, EXPRIME, CONTRIBUE_A, and HAS_SOURCE. Entity types encompass 12 core civic categories: PROPOSITION, THEMATIQUE, SERVICE_PUBLIC, DOLEANCE, ACTEUR_INSTITUTIONNEL, OPINION, CITOYEN, CONCEPT, REFORME_DEMOCRATIQUE, TERRITOIRE, COMMUNE, and CONTRIBUTION. The chunk-as-node architecture stores text passages as first-class graph citizens with bidirectional edges to entities, enabling O(1) provenance retrieval.

Pre-Computed Indices {-}

At server startup, graphs are loaded into in-memory NetworkX structures with adjacency indices enabling O(1) neighbor lookups, eliminating per-query parsing overhead.

Small-World Reconstruction Algorithm

Our retrieval pipeline operationalizes small-world network principles through three stages:

Stage 1: Dual-Strategy Seeding

Seeds = CommunitySelection(query) ∪ GlobalEntitySearch(query)

The seeding phase combines two complementary strategies. Community selection performs keyword matching against pre-generated Louvain community summaries to identify thematically relevant clusters, while global entity search conducts full-corpus searches across entity names and descriptions using case-insensitive, fuzzy matching to discover entities regardless of community membership. This dual-strategy approach achieves 92.7% corpus coverage compared to the 16% baseline observed with single-strategy retrieval, as empirically validated through corpus retrieval experiments.

Stage 2: Weighted Multi-Hop Expansion

Starting from seed entities, we execute weighted Dijkstra traversal for K=5 hops:

# Relationship type weights (semantic priority)
weights = {
    'CONCERNE': 1.0,         # Direct thematic connection
    'HAS_SOURCE': 0.9,       # Entity-chunk provenance
    'CONTRIBUE_À': 0.8,      # Contributes to
    'EXPRIME': 0.7,          # Expresses
    'FAIT_PARTIE_DE': 0.5,   # Structural part-of
    'APPARTIENT_À': 0.3,     # Weak structural belongs-to
    'RELATED_TO': 0.1        # Generic fallback
}

# Multi-hop expansion
discovered_nodes = set(seeds)
current_layer = seeds
for hop in range(1, max_hops + 1):
    next_layer = {}
    for node in current_layer:
        neighbors = graph.get_neighbors(node)
        for neighbor, edge_type in neighbors:
            weight = weights.get(edge_type, 0.1)
            if weight >= threshold:
                next_layer.add(neighbor)
    discovered_nodes.update(next_layer)
    current_layer = next_layer

Weighted Traversal Rationale {-}

Relationship weights encode semantic priority, preventing graph explosion (unweighted BFS would yield 100,000+ nodes) while maintaining thematic coherence. Edges with high weights (CONCERNE, HAS_SOURCE) are prioritized, ensuring expansions follow strong semantic connections rather than weak structural relationships.

Stage 3: Ontological Coverage Verification

Post-expansion, we verify small-world completeness:

entity_type_coverage = {
    entity_type: count
    for entity_type in discovered_nodes
}
missing_types = [
    t for t in CORE_CIVIC_TYPES
    if t not in entity_type_coverage
]
coverage_pct = (12 - len(missing_types)) / 12 * 100

Systems log: "Small world: N nodes, X% ontological coverage (K/12 types)". Coverage ≥91% indicates comprehensive capture. If coverage <80%, the system iteratively expands by one additional hop.

Chunk Retrieval via Graph

Provenance is maintained through chunk-as-node architecture:

For each entity in discovered_nodes:
    source_ids = entity.attributes['source_id'].split(';')
    text_chunks = [chunk_store.get(sid) for sid in source_ids]

This enables O(1) chunk retrieval (<1ms per entity) via in-memory dictionary lookups rather than file I/O (which incurred 500ms+ latencies in initial prototypes, documented in troubleshooting.md). The system includes ~4,000-5,000 nodes in typical small worlds spanning entities, relationships, and text chunks.

Experimental Design

Dataset

The evaluation dataset, civic-law-eval, hosted on the OPIK platform, comprises 54 questions derived from the Grand Débat National corpus. The domain is French civic discourse from Charente-Maritime, covering 50 communes. Questions span four categories: profile metadata addressing demographics, ages, and family situations; corpus extraction targeting salary proposals, healthcare concerns, and enterprise mentions; cross-contribution queries requiring synthesis across communes such as RIC mentions across 38 communes and ISF restoration proposals; and contribution-exact questions demanding specific textual extractions with contribution references. A representative example is: "Dans la contribution n°4 du cahier de Rochefort, quel montant de retraite mensuel est mentionné comme insuffisant pour une personne seule?" (Expected: 1200 euros/month).

Evaluation Metrics

Performance Metrics {-}

We measure latency as a continuous variable in milliseconds representing end-to-end time from query dispatch to complete response, inclusive of retrieval, LLM generation, and network transmission. Success rate is recorded as a binary indicator of query completion within the 120-second timeout threshold, where 1 denotes success and 0 denotes failure.

Semantic Quality Metrics {-}

Using GPT-4o-mini as an LLM judge with temperature set to 0, we evaluate five semantic dimensions. LLM precision scores responses from 0 to 1 based on factual accuracy, completeness relative to question scope, legal reasoning quality, and appropriate source citation. Answer relevance measures on the same scale how directly the response addresses the input question. Hallucination provides a faithfulness score where 1.0 indicates fully faithful responses to context and 0.0 indicates hallucinated content. Meaning match evaluates semantic equivalence between system responses and expected reference answers using GEval criteria tailored for French civic discourse. Usefulness assesses the practical utility of responses for answering users' civic questions.

Lexical Metric {-}

The contains metric provides a binary indicator of whether expected reference terms appear in the generated response.

Retrieval Time {-}

We measure retrieval_time_ms to isolate context construction from LLM generation. For GraphRAG Surgical, this represents the average time per commune for vector search and graph traversal (excluding LLM calls). For Dust, it measures semantic_search action duration extracted from API response metadata. These metrics reflect different architectural work---GraphRAG's exhaustive multi-commune traversal versus Dust's single top-K retrieval---and are presented for transparency rather than direct comparison.

Controlled Variables

To ensure valid comparison, we implement six explicit controls. First, we enforce LLM model parity by configuring both systems to use GPT-5-nano with identical provider, model ID, and API version. Second, we maintain timeout parity through identical 120-second timeouts, a value empirically determined based on observations that Dust requires 30-60 seconds for complex queries while GraphRAG typically responds within 1-15 seconds. Third, we apply execution order randomization through random 50/50 selection of which system runs first per experiment, mitigating cache warming and resource allocation advantages. Fourth, metric cloning ensures fresh metric instances for each evaluation phase, preventing state leakage through connection pooling. Fifth, retry logic parity is achieved by implementing 2 retries with exponential backoff (1s, 2s delays) in GraphRAG to match Dust's implicit polling resilience. Sixth, query scope alignment ensures GraphRAG queries all 50 communes via grand_debat_query_all to match Dust's full corpus access.

Experiment Tracking

All experiments are tracked on the OPIK platform (Comet.ml) for persistent result storage and visualization. Experiments are tagged as {base}_dust and {base}_graphrag for side-by-side comparison, with metadata recording system identifier, execution timestamp, sample size, execution order, timeout values, metric selections, and configuration parameters.

The primary experiment (rag_comparison_20260106_160751) evaluates the full civic-law-eval dataset of 54 questions. GraphRAG produced 34,630 CSV rows through detailed multi-row logging, while Dust produced 574 CSV rows at approximately 10--11 rows per question.

Experimental Protocol

Each experimental run proceeds through five sequential stages. During initialization, we load the evaluation dataset, initialize both client connections, and configure OPIK tracking. Randomization follows, where uniform random selection determines execution order with the choice recorded in metadata. Phase 1 executes the first system (System A) by instantiating fresh metric objects, then for each question dispatching the query, measuring latency via perf_counter, and awaiting the response. For Dust, this involves creating a conversation and polling at 500ms intervals with a maximum of 240 polls corresponding to the 120-second timeout. For GraphRAG, we initialize an MCP session, invoke the grand_debat_query tool, and parse the Server-Sent Events stream. Results are logged to OPIK under the appropriate experiment suffix ({base}_dust or {base}_graphrag). Phase 2 repeats the evaluation process for System B with fresh metrics. Finally, LLM-as-judge evaluation applies GPT-4o-mini to assess semantic quality for each response, with 500ms rate limiting between calls to prevent quota exhaustion.

Threats to Validity

Internal Validity Threats {-}

Three internal threats were identified and addressed where possible. Temperature asymmetry between Dust (0.7) and GraphRAG (1.0) remains unmitigable due to platform and model constraints; this affects response variability as higher temperature may produce more creative but less consistent outputs. Sequential execution order bias, where the first-executing system faces disadvantages from cold caches while the second benefits from warmed resources, is mitigated through randomization. Shared metric state, which could provide connection pooling advantages, is mitigated through metric cloning using fresh instances.

Construct Validity Threats {-}

Dataset scope mismatch poses a construct validity concern, as Dust may access broader training data beyond the 50 communes covered by GraphRAG. While questions were reviewed to ensure commune-specific focus, broader contextual knowledge may provide Dust with advantages.

External Validity Threats {-}

Two external threats limit generalizability. Domain specificity constrains how results obtained for French civic discourse generalize to other languages, legal systems, or question-answering domains. Model-specific findings reflect GPT-5-nano's characteristics; other language models may exhibit different latency-quality trade-offs.

Results

We present results from two multi-commune surgical experiments evaluating both systems on 54 civic-law-eval questions. Results are organized by performance metrics, semantic quality, and query-type analysis.

Performance Metrics

Both experiments use exhaustive multi-commune mode, querying 20 communes in parallel via asyncio.gather().

Experiment 1: rag_comparison_20260106_160751 (Full Dataset N=54)

*Faithfulness: higher=better (see Faithfulness discussion below)

GraphRAG demonstrates conservative boundary behavior---explicitly acknowledging data limitations rather than forcing answers---at the cost of higher latency and lower single-shot precision. Dust achieves higher faithfulness through focused context extraction.

Experiment 2: rag_comparison_20260113_145649 (Subset N=24)

Subset of 24 questions where both systems succeeded (Dust rate-limited on 30/54 queries)

*Dust encountered rate limiting (plan_message_limit_exceeded) on 55.6% of queries.

On the comparable subset, GraphRAG's latency overhead is 1.6x (not dramatically higher), while retrieval time is 7.5x slower due to exhaustive corpus traversal. The key trade-off is reliability: GraphRAG maintains 100% success rate versus Dust's 44% under load.

Coverage-Quality Trade-off

The results reveal a coverage-reliability versus precision-speed trade-off. GraphRAG is slower but highly reliable with 100% success rate, comprehensive coverage, and conservative uncertainty acknowledgment. Dust is faster with higher precision when successful, but proves rate-limit sensitive with 44--80% success rate and focused extraction. Users should select GraphRAG when corpus coverage, reliability, and explicit uncertainty are critical, and Dust when single-shot precision and speed dominate requirements.

Semantic Quality Metrics

Experiment 2 subset (N=24) where both systems succeeded:

Critical Methodological Note: Query-Type Determines Performance

Analysis of our 54-question dataset reveals that query type, not system quality, drives faithfulness scores:

GraphRAG produces 144x more output on average, reflecting a fundamental architectural output asymmetry. For needle queries, GraphRAG generates approximately 43K characters across 20 per-commune responses, each reporting "not found for this commune." For broad queries, output reaches approximately 176K characters through comprehensive cross-commune synthesis with structured analysis.

GraphRAG fails needle queries because of how its exhaustive architecture interacts with specific lookups. When asked "What's in contribution n°4 of Rochefort?", the system queries all 20 communes in parallel, reconstructing approximately 28K tokens of context per query. Each commune's LLM synthesis attempts to locate "contribution n°4" within thousands of entities, but the specific contribution numbering may not be preserved through graph reconstruction. Each commune honestly reports that it cannot find the requested contribution in its data, and the aggregated output becomes 20 "not found" messages. This represents correct boundary behavior, not hallucination.

Conversely, GraphRAG succeeds on broad queries because they align with its architectural strengths. When asked "What's the gender distribution across contributions?", the system reconstructs comprehensive semantic neighborhoods across all communes. The question does not require locating a specific item within 28K tokens, allowing the system to produce structured analysis progressing from introduction through per-commune analysis to transversal synthesis, achieving faithfulness scores up to 0.80 on analytical questions.

Dust's advantage on needle queries stems from its focused retrieval: with approximately 195 tokens of context retrieved via semantic similarity, Dust's LLM can directly extract "1200EUR" without searching through thousands of tokens.

The dataset composition further influences aggregate metrics: 48% of evaluation questions are needle queries, which inherently favors Dust's direct extraction architecture. A dataset with more broad analytical questions would likely produce different aggregate results. In summary, Dust excels at needle queries requiring direct extraction from focused context, while GraphRAG excels at broad analytical queries requiring comprehensive cross-corpus synthesis.

Both systems scored near-zero on meaning_match, suggesting that expected reference answers use different phrasing conventions than either system employs, or that the evaluation criterion is overly strict for open-ended civic question answering.

Query-Type Performance Analysis

To illustrate distinct performance profiles, we analyze actual system responses for representative queries.

Single-Shot Needle Queries

The example query is: "Dans la contribution n°4 du cahier de Rochefort, quel montant de retraite mensuel est mentionné comme insuffisant pour une personne seule?" The expected answer is "1200 euros par mois."

Dust Response {-}

Dust responded in 27,180ms with:

Le montant mentionné est de 1200 € par mois pour une personne seule. :cite[ekx]

This achieved LLM Precision=0.8, Answer Relevance=1.0, Hallucination=1.0, and Usefulness=1.0. The response is concise, accurate, and properly cited. Dust excels at precise fact extraction.

GraphRAG Response {-}

GraphRAG responded in 1,209ms with:

Désolé, je ne trouve pas l'information demandée dans les données fournies.
Aucun contenu relatif au cahier de Rochefort [...] n'est présent dans les
tableaux que vous avez partagés.
Pour information générale [...] Certaines propositions évoquent un minimum
de retraite autour de 1500€...
[continues for ~95 more lines across 20 commune responses]

This achieved LLM Precision=0.2, Answer Relevance=0.5, Hallucination=0.3, and Usefulness=0.4. The response is verbose and uncertain, eventually guessing 1000EUR (incorrect). The LLM synthesis phase fails to locate the specific contribution reference within 28K tokens of reconstructed context.

Dust wins for single-shot needle queries optimized for semantic similarity search.

Small-World Capture Evidence

While we lack direct empirical data on multi-turn query performance, architectural evidence suggests GraphRAG's advantages for iterative exploration:

The following metrics were logged during retrieval:

Small world: 4,234 nodes, 91.7% ontological coverage (11/12 types)
Entity types found: [PROPOSITION, THÉMATIQUE, SERVICE_PUBLIC, DOLÉANCE,
                     ACTEUR_INSTITUTIONNEL, OPINION, CITOYEN, CONCEPT,
                     RÉFORME_DÉMOCRATIQUE, TERRITOIRE, COMMUNE]
Relationships captured: 8,917 edges across 28 semantic types
Corpus coverage: 92.7% (46/50 communes represented)

GraphRAG's first retrieval captures comprehensive semantic neighborhoods---4,234 interconnected entities spanning nearly complete ontological coverage. This small-world structure theoretically enables powerful follow-up queries: "What proposals are related to this theme?" or "Which communes have similar patterns?" can be answered by traversing the already-retrieved subgraph without additional retrieval overhead.

In a 2-shot scenario, GraphRAG's first shot reconstructs the small world (comprehensive but imprecise), while the second shot leverages this context for targeted retrieval---analogous to how traditional vector RAG performs within constrained contexts. This remains empirically unvalidated and constitutes future work (see Future Research Directions).

Faithfulness, Boundary Honesty, and Architectural Implications

The faithfulness metric reveals fundamentally different architectural behaviors rather than a simple quality comparison.

OPIK's "hallucination" evaluator, inverted as faithfulness (1=faithful, 0=unfaithful), assesses whether output claims match provided reference context. Critically, this metric penalizes GraphRAG when it correctly acknowledges data boundaries by reporting "I don't find Rochefort in my context," because the reference contains the expected answer.

Analysis of low-faithfulness GraphRAG responses reveals a consistent pattern of conservative boundary behavior. The system reports "données non disponibles" (data unavailable) when the specific contribution (e.g., "Rochefort #4") is absent from the reconstructed small world, when the queried entity falls outside the current commune's graph, or when the LLM synthesis phase cannot locate specific references within 28K tokens of reconstructed context. This constitutes architecturally honest behavior: GraphRAG refuses to speculate beyond its reconstructed boundaries. In high-stakes domains such as legal, medical, and civic applications, explicit "I don't know" responses may be preferable to confidently wrong answers.

By contrast, Dust's direct extraction behavior benefits from tightly focused context. With approximately 195 tokens of retrieved content, Dust's LLM can directly extract answers when they exist, and the higher faithfulness score (0.53) reflects successful semantic matching between queries and indexed content.

Table 1 reframes this trade-off across four behavioral dimensions.

: Behavioral comparison between GraphRAG and Dust across architectural dimensions.

GraphRAG's explicit entity-relationship structure constrains generation to documented facts, but the volume of reconstructed context (approximately 28K tokens) challenges single-shot extraction. The architecture may excel in iterative scenarios where follow-up questions refine within the established small world (see Multi-Turn Query Hypothesis in Discussion).

Performance-Quality Trade-off Visualization

The results reveal a Pareto frontier: no single system dominates across all metrics. Dust optimizes for semantic precision (6.3× advantage) and answer relevance (1.5× advantage) through focused retrieval. GraphRAG optimizes for comprehensive coverage (100% corpus), reliability (100% success rate), and explicit uncertainty acknowledgment through exhaustive reconstruction.

These architectural trade-offs inform system selection across application domains. For single-shot precision tasks such as fact lookup and specific entity queries, Dust is preferable for its direct extraction from focused context. For comprehensive coverage tasks involving cross-corpus analysis and pattern discovery, GraphRAG is preferable for its exhaustive small-world reconstruction. In high-stakes uncertainty-sensitive tasks spanning legal, medical, and audit domains, GraphRAG's explicit boundary acknowledgment offers advantages over forced answers. For iterative exploration tasks such as civic discourse analysis and hypothesis generation, GraphRAG benefits from cached small-world reuse (see Multi-Turn Query Hypothesis). Finally, for reliability-critical tasks in production systems with SLA requirements, GraphRAG's 100% success rate contrasts favorably with Dust's rate-limiting sensitivity.

Discussion

Topological RAG Performance Profile

Our results reveal distinct architectural performance profiles that inform RAG system selection for different use cases.

First-Shot Precision Gap

Vector RAG's 1.8× precision advantage (0.60 vs 0.33 LLM precision) for single-shot queries reflects fundamental architectural differences. Dust employs dense semantic embeddings to retrieve top-K chunks maximally similar to the query, optimizing for semantic alignment. When a question asks for a specific fact ("quel montant?"), vector similarity search directly locates passages containing that fact with high probability.

GraphRAG's topological approach reconstructs comprehensive semantic neighborhoods regardless of query specificity. A query about "contribution n°4 retirement amount" triggers small-world expansion capturing all retirement-related entities, themes, actors, and proposals—potentially 4,000+ nodes. While this breadth enables comprehensive reasoning, it introduces noise: the LLM receives extensive context (entities, relationships, 20+ text chunks) and must locate the specific needle (contribution #4) within this haystack. The precision gap stems from this signal-to-noise ratio challenge.

For applications prioritizing single-fact extraction such as FAQ systems and simple QA bots, vector RAG's precision optimization is architecturally superior.

Small-World Comprehensiveness Advantage

GraphRAG's 92.7% corpus coverage (via dual-strategy seeding) and 91.7% ontological coverage (11/12 entity types) demonstrates comprehensive small-world reconstruction. This architecture excels when queries require:

Cross-entity reasoning queries such as "How do retirement proposals relate to tax reform?" require traversing paths like THÉMATIQUE_Retraites → PROPOSITION → THÉMATIQUE_Fiscalité. Pattern discovery questions including "Which communes have similar concerns?" benefit from complete commune-level subgraph capture. Provenance tracing queries like "What citizen contributions support this theme?" leverage chunk-as-node bidirectional edges for O(1) source retrieval.

Vector RAG's top-K chunk retrieval inherently fragments knowledge. Even with large K (e.g., K=20), chunks lack explicit relationships, forcing LLMs to infer connections that may be absent or hallucinated.

For applications requiring holistic understanding such as exploratory data analysis, hypothesis generation, and cross-referential reasoning, topological architectures provide structural advantages.

Architectural Trade-offs Explained

Coverage vs. Latency Trade-off

GraphRAG's exhaustive multi-commune mode (evaluated in this study) exhibits distinct performance characteristics:

In exhaustive multi-commune mode (surgical), querying N=20 communes in parallel via asyncio.gather() incurs O(N) retrieval overhead. Mean retrieval time reaches 10,653ms, 7.5x slower than Dust's 1,413ms, due to per-commune vector search for seed entities, 5-hop weighted graph traversal per commune, per-commune LLM synthesis averaging approximately 40 seconds across 20 parallel calls, and sequential response aggregation for final synthesis.

Dust's top-K approach achieves 1,413ms mean retrieval by selecting only the most semantically similar chunks regardless of corpus partitioning. This optimizes latency but provides no guarantee of comprehensive coverage, as relevant content in low-similarity communes may be missed.

On the N=24 successful query subset, GraphRAG achieved 101,971ms mean latency, 10,653ms mean retrieval, and 100% corpus coverage, while Dust achieved 62,057ms mean latency, 1,413ms mean retrieval, and partial top-K coverage. The trade-off is therefore coverage versus speed, not universal performance superiority. Exhaustive analysis questions such as "How do themes vary across all communes?" favor GraphRAG's surgical mode; semantic similarity queries such as "Find passages about retirement" favor Dust's top-K approach; and needle queries such as "What did contribution #4 say?" depend on whether the contribution is present in the reconstructed context.

GraphRAG's 1.6x latency overhead buys 100% corpus coverage and 100% success rate compared to Dust's 44%, while Dust's speed advantage comes with rate-limiting sensitivity and partial coverage.

Architectural Output Asymmetry and Its Implications

The most striking finding is the 144× output volume difference (~28K tokens for GraphRAG vs ~195 tokens for Dust), reflecting fundamentally different architectural philosophies.

GraphRAG follows a "reconstruct-then-refine" pattern: the system first builds comprehensive semantic neighborhoods spanning 4,000+ entities, then attempts synthesis within this massive context. This produces exhaustive coverage of related concepts spanning from retirement through taxation, purchasing power, and public services, along with verbose responses acknowledging multiple relevant dimensions and conservative boundary behavior when specific references cannot be located within 28K tokens.

Dust follows a "retrieval-then-synthesis" pattern: the system retrieves focused top-K chunks of approximately 195 tokens semantically aligned with the query, then extracts direct answers. This produces concise, targeted responses with higher precision for needle queries, but without explicit uncertainty acknowledgment when data boundaries are reached.

This distinction matters for faithfulness scores. GraphRAG's low faithfulness (0.07) primarily reflects its honest acknowledgment of reconstruction boundaries rather than fabricated claims. When the LLM cannot locate "contribution n°4 de Rochefort" within 28K tokens of reconstructed context, it reports "data not found"---penalized by the metric because the reference contains the answer, but arguably correct behavior.

GraphRAG's entity-relationship structure does constrain synthesis to documented facts through entity-type verification across 12 types, relationship validation across 28 semantic types, and chunk attribution via source_id provenance. However, single-shot extraction from 28K tokens challenges LLM attention mechanisms. The trade-off is ultimately between coverage-verbosity in GraphRAG's comprehensive but verbose reconstruction and precision-conciseness in Dust's focused but bounded extraction, and architectural selection should match query requirements.

Hypothesis: Multi-Turn Query Superiority

While our evaluation focuses on single-shot queries, GraphRAG's architectural properties suggest theoretical advantages for iterative, multi-turn exploration:

In a 2-shot query scenario, GraphRAG's performance profile fundamentally shifts. During the first shot, GraphRAG reconstructs the small world comprising 4,000+ nodes with 91.7% ontological coverage---comprehensive but potentially imprecise for specific needle queries as demonstrated in the Single-Shot Needle Queries results. However, during the second shot, follow-up queries operate within this cached small world, enabling traditional RAG techniques such as semantic search and entity filtering to achieve high precision within an already comprehensive context.

Multiple architectural properties support this hypothesis. Small worlds include dense subgraphs where a single thematic focus like retirement encompasses 400+ entities interconnected through 800+ relationships, providing rich context for follow-up queries. Ontological completeness at 91.7% coverage ensures no entity type gaps requiring additional retrieval, meaning follow-up questions about any civic entity category can be answered from the cached small world. The cached small-world structure enables zero-latency follow-ups via in-memory graph queries without repeating the initial retrieval process. Additionally, provenance edges marked as HAS_SOURCE relationships allow instant drilling from high-level entities down through relationships to specific chunks and finally to original contribution references.

This hypothesis draws an analogy to vector RAG's strengths: vector RAG excels within bounded contexts (top-K chunks) because semantic similarity search is optimized for focused retrieval. GraphRAG extends this principle by first capturing the complete relevant context (the small world), then applying focused retrieval within that bounded subgraph.

This hypothesis remains untested. Future work must evaluate multi-turn performance through user studies or simulated dialogue datasets measuring three critical dimensions: Turn-1 response quality to assess comprehensive context capture during initial small-world reconstruction, Turn-2+ response quality to evaluate precision achieved when operating within the established context, and cumulative information gain across conversation turns to quantify whether iterative refinement yields superior knowledge acquisition compared to independent single-shot queries.

Extreme Use Case Suitability

Our findings position topological RAG architecturally suited for extreme use cases prioritizing comprehensiveness and interpretability over single-shot precision:

Ideal Domains

Non-Ideal Domains

Limitations and Threats to Validity

Temperature Asymmetry

The unresolvable temperature difference (Dust 0.7 vs GraphRAG 1.0) introduces confounding effects, though the 1.8x precision gap is unlikely solely attributable to temperature given Dust's architectural advantages. Future work should compare systems with identical temperature settings.

Domain Specificity

Results reflect French civic discourse, a specialized domain characterized by well-defined entity types across 12 civic categories, structured documents in the form of citizen contribution notebooks, and a geographically bounded corpus of 50 communes.

Generalization to unstructured, open-domain corpora such as web-scale question answering remains unvalidated. Topological approaches may degrade when entity extraction quality is low due to ambiguous entities or incorrect types, when relationship graphs are sparse with limited edges and weak connectivity, or when queries span multiple disconnected subgraphs requiring expensive cross-world traversal.

Single-Shot Evaluation Bias

Our 54-question evaluation uses independent, single-shot queries, inherently favoring vector RAG's precision optimization. Future evaluations should include conversational datasets with follow-up questions and user studies measuring task completion rates across dialogue turns.

No Empirical 2-Shot Data

The multi-turn query hypothesis relies on architectural analysis without direct empirical validation. Designed user studies with scripted multi-turn scenarios (Turn 1: broad exploration; Turn 2: targeted follow-up) would provide conclusive evidence.

Future Research Directions

Hybrid Architectures

The complementary strengths suggest hybrid cascade designs: Vector RAG retrieves top-K seeds (precision), graph expansion reconstructs the small world around those seeds (completeness), then LLM synthesis leverages both. Open questions include optimal K for seeding, dynamic switching criteria between full graph expansion and direct vector results, and latency budget allocation across stages.

Multi-Turn Query Evaluation

A conversational dataset with structured turns (Turn 1: broad exploration, Turns 2-3: targeted follow-ups) would test whether GraphRAG's small-world capture enables superior Turn-2+ precision compared to vector RAG's independent per-turn retrieval. Key metrics: cumulative information gain, user satisfaction, and task completion rates.

Domain Adaptation

Evaluating topological RAG across legal corpora (Caselaw Access Project), medical knowledge bases (PubMed, SPOKE), enterprise wikis, and scientific literature (ArXiv) would establish generalization boundaries. Critical questions: how entity extraction quality affects small-world completeness, how graph connectivity patterns influence traversal effectiveness, and how ontological coverage requirements vary across domains.

Ontological Coverage as Tunable Parameter

Coverage thresholds (currently 91.7%) may be application-specific: medical diagnosis may require 100% (missing symptom categories risk misdiagnosis), while exploratory analysis may tolerate 80% for faster retrieval. Systematic experimentation varying thresholds (70-100%) across domains would establish optimal selection criteria.

Conclusion

This work introduces Topological RAG, a graph-based retrieval architecture implementing a "reconstruct-then-refine" paradigm through dual-strategy seeding, weighted multi-hop traversal, and ontological coverage verification. The system trades latency (1.6x slower than Dust) for guaranteed 100% corpus coverage and comprehensive outputs (~28K tokens vs Dust's ~195 tokens) with explicit boundary acknowledgment.

Our evaluation resolves the completeness-precision dilemma through architectural differentiation rather than universal superiority: topological RAG excels where comprehensiveness and interpretability dominate (legal, medical, civic domains), while vector RAG excels where single-shot precision dominates (FAQ systems, fact lookup). We hypothesize that GraphRAG's small-world reconstruction positions it favorably for iterative, multi-turn queries---a paradigm shift from retrieval-then-synthesis to reconstruct-then-refine pending empirical validation.

Future work should validate multi-turn performance, explore hybrid cascade architectures, and evaluate domain adaptation across legal, medical, and enterprise contexts.

References

Ontologies and Knowledge Capture

Boje, C., Guerriero, A., Kubicki, S., et al. (2020). Towards a Semantic Construction Digital Twin. Construction Innovation, 20(1), 12-32. https://openalex.org/W3013120860

Carbon, S., Douglass, E., Good, B. M., et al. (2020). The Gene Ontology Resource: Enriching a GOld Mine. Nucleic Acids Research, 49(D1), D325-D334. https://doi.org/10.1093/nar/gkaa1113

Fernández del Amo, I., Erkoyuncu, J. A., Bułka, D., et al. (2024). Advancing Fault Diagnosis Through Ontology-Based Knowledge Capture and Application. Engineering Applications of Artificial Intelligence, 132, 107924. https://openalex.org/W4400975193

Huet, A., Pinquié, R., Veron, P. (2020). CACDA: A Knowledge Graph for Context-Aware Cognitive Design Assistant. Computers in Industry, 125, 103377. https://doi.org/10.1016/j.compind.2020.103377

Lu, L., Liang, X., Yuan, G., et al. (2023). A Study on Knowledge Graph Construction of Yunjin Video Resources. Heritage Science, 11(1), 83. https://doi.org/10.1186/s40494-023-00932-5

Pandithawatta, S., Ahn, S., Rameezdeen, R. (2023). Development of Knowledge Graph for Automatic Job Hazard Analysis: The Schema. Sensors, 23(8), 3893. https://doi.org/10.3390/s23083893

Schneider, G. F., Kontes, G. D., Qiu, H., et al. (2020). Design of Knowledge-Based Systems for Automated Deployment of Building Management Services. Energy and Buildings, 224, 110247. https://openalex.org/W3089063274

Knowledge Graphs for Expert Management

Bai, Y., Wu, J., Ren, Q., et al. (2023). A BN-Based Risk Assessment Model Integrating Knowledge Graph and DEMATEL. Process Safety and Environmental Protection, 171, 150-168. https://doi.org/10.1016/j.psep.2023.01.060

Chen, Q. H., Long, D., Yang, C., et al. (2023). Knowledge Graph Improved Dynamic Risk Analysis for Construction Safety Management. Journal of Management in Engineering, 39(3), 04023005. https://doi.org/10.1061/jmenea.meeng-5306

Uniyal, S., Mangla, S. K., Sarma, P. R. S., et al. (2020). ICT as Knowledge Management for Sustainable Supply Chains. Journal of Global Information Management, 29(1), 172-197. https://doi.org/10.4018/jgim.2021010109

Wang, X., Ban, T., Chen, L., et al. (2022). Knowledge Verification from Data. IEEE Transactions on Neural Networks and Learning Systems, 34(11), 9324-9337. https://doi.org/10.1109/tnnls.2022.3202244

Wu, J., Wei, Z., Jia, D. (2022). Constructing Marine Expert Management Knowledge Graph Based on Trellisnet-CRF. PeerJ Computer Science, 8, e1083. https://doi.org/10.7717/peerj-cs.1083

Xiao, Z., Zhang, C. (2021). Construction of Meteorological Simulation Knowledge Graph Based on Deep Learning. Sustainability, 13(3), 1311. https://doi.org/10.3390/su13031311

Xue, B., Zou, L. (2022). Knowledge Graph Quality Management: A Comprehensive Survey. IEEE Transactions on Knowledge and Data Engineering, 35(5), 4969-4988. https://doi.org/10.1109/tkde.2022.3150080

Retrieval-Augmented Generation with Knowledge Graphs

Chang, R.-C., Zhang, J. (2024). CommunityKG-RAG: Leveraging Community Structures in Knowledge Graphs for Advanced Retrieval-Augmented Generation in Fact-Checking. arXiv:2408.08535. https://doi.org/10.48550/arxiv.2408.08535

Dong, C., Yuan, Y., Chen, K., et al. (2023). How to Build an Adaptive AI Tutor for Any Course Using Knowledge Graph-Enhanced Retrieval-Augmented Generation (KG-RAG). arXiv:2311.17696. https://doi.org/10.48550/arxiv.2311.17696

Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truitt, S., Larson, J. (2024). From Local to Global: A Graph RAG Approach to Query-Focused Summarization. arXiv:2404.16130. https://doi.org/10.48550/arxiv.2404.16130

Feng, H., Yin, Y., Reynares, E., Nanavati, J. (2025). OntologyRAG: Better and Faster Biomedical Code Mapping with Retrieval-Augmented Generation Leveraging Ontology Knowledge Graphs. Studies in Health Technology and Informatics, 310, 47-51. https://doi.org/10.1007/978-3-032-02899-0_4

Lewis, P., Perez, E., Piktus, A., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems, 33, 9459-9474.

Matsumoto, N., Moran, J., Choi, H.-J., Hernandez, M., Venkatesan, M. (2024). KRAGEN: A Knowledge Graph-Enhanced RAG Framework for Biomedical Problem Solving Using Large Language Models. Bioinformatics, 40(9), btae353. https://doi.org/10.1093/bioinformatics/btae353

Opoku, D. O., Sheng, M., Zhang, Y. (2025). DO-RAG: A Domain-Specific QA Framework Using Knowledge Graph-Enhanced Retrieval-Augmented Generation. TechRxiv, 174837976. https://doi.org/10.36227/techrxiv.174837976.69904638/v1

Peng, B., Zhu, Y., Liu, Y., Bo, X., Shi, H., Hong, C., Zhang, Y., Tang, S. (2024). Graph Retrieval-Augmented Generation: A Survey. arXiv:2408.08921. https://doi.org/10.48550/arxiv.2408.08921

Sanmartin, D. (2024). KG-RAG: Bridging the Gap Between Knowledge and Creativity. arXiv:2405.12035. https://doi.org/10.48550/arxiv.2405.12035

Soman, K., Rose, P. W., Morris, J. H., Akbas, R. E., Smith, B. (2024). Biomedical Knowledge Graph-Optimized Prompt Generation for Large Language Models. Bioinformatics, 40(10), btae560. https://doi.org/10.1093/bioinformatics/btae560

Watts, D. J., Strogatz, S. H. (1998). Collective Dynamics of 'Small-World' Networks. Nature, 393(6684), 440-442. https://doi.org/10.1038/30918

Wei, X., Liu, Y., Li, X., Gao, F., Gu, J. (2024). TKG-RAG: A Retrieval-Augmented Generation Framework with Text-chunk Knowledge Graph. Proceedings of the 14th International Conference on Advanced Computer Information Technologies, 483-488. https://doi.org/10.1109/acit62805.2024.10877117

Xie, W., Liang, X., Liu, Y., et al. (2024). WeKnow-RAG: An Adaptive Approach for Retrieval-Augmented Generation Integrating Web Search and Knowledge Graphs. arXiv:2408.07611. https://doi.org/10.48550/arxiv.2408.07611

Xu, Z., Dela Cruz, M. M. C., Guevara, M., Wang, T. (2024). Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering. Proceedings of the 47th International ACM SIGIR Conference, 2828-2838. https://doi.org/10.1145/3626772.3661370

Data Availability

The evaluation dataset civic-law-eval (54 questions) is available on the OPIK platform at https://www.comet.com/opik/law_graphRAG.

Experiment 1 (single-commune, rag_comparison_20260106_160751) produced 34,630 GraphRAG rows and 574 Dust rows. Experiment 2 (multi-commune surgical, rag_comparison_20260113_145649) produced separate CSV exports for GraphRAG Surgical and Dust, available in the repository under docs/eval/.

The source code for the GraphRAG implementation is available at https://github.com/ArthurSrz/graphRAGmcp, and the evaluation framework at https://github.com/ArthurSrz/graphRAGmcp/tree/main/docs/eval. The knowledge graph for the Grand Débat National 2019, Charente-Maritime (50 communes, GraphML format) is at https://github.com/ArthurSrz/graphRAGmcp/tree/main/law_data.

Acknowledgments

This research was conducted as part of the GraphRAG Research Group's investigation into topological retrieval architectures for civic discourse analysis. We thank the contributors to the Grand Débat National 2019 whose citizen contributions constitute the evaluation corpus.

Appendix A: Metric Definitions

All semantic metrics use OPIK's built-in evaluators with GPT-4o-mini (temperature=0) as judge. Full prompt templates are in the OPIK SDK: opik.evaluation.metrics.llm_judges.* (documentation).

Performance Metrics

Semantic Quality Metrics (LLM-as-Judge)

Architectural Metrics

• Corpus Coverage: Percentage of communes queried. GraphRAG Surgical: 100%. Dust: partial (top-K, no partition guarantee).
• Ontological Coverage: (entity_types_found / 12) x 100

Reproducibility

All evaluations: GPT-4o-mini, temperature 0.0, OPIK platform (Comet.ml), SDK version opik>=1.0.0. Full evaluation code: https://github.com/ArthurSrz/graphRAGmcp/tree/main/rag_comparison

Appendix B: Evaluation Dataset (civic-law-eval)

Dataset Overview

Corpus Statistics

Question Categories

The dataset comprises four question categories designed to test different retrieval capabilities:

Profile Metadata (N=8)

Questions about contributor demographics, ages, and family situations. Example: "Quelle est la répartition hommes/femmes parmi les contributeurs?"

Corpus Extraction (N=18)

Questions requiring extraction of specific content across multiple documents. Example: "Listez tous les montants de retraite mentionnés comme insuffisants."

Cross-Contribution Queries (N=12)

Questions requiring synthesis across multiple communes and contributions. Example: "Combien de communes mentionnent le RIC (Référendum d'Initiative Citoyenne)?"

Contribution-Exact (N=16)

Questions targeting specific contributions with known answers. Example: "Dans la contribution n°4 du cahier de Rochefort, quel montant de retraite mensuel est mentionné?"

Expected Answers

Each question includes an expected reference answer used for:

• Meaning Match evaluation (semantic equivalence)
• Contains metric (lexical presence)
• Human validation of LLM judge scores

Dataset Limitations

1. Domain Specificity: French civic discourse may not generalize to other domains or languages
2. Temporal Scope: Data from 2019 Grand Débat; civic concerns may have evolved
3. Geographic Scope: Limited to Charente-Maritime (50 communes); may not represent national patterns
4. Question Balance: Categories are not equally represented, potentially biasing aggregate metrics

Appendix C: Technical Codebase Documentation

Full source code and implementation details are available at: https://github.com/ArthurSrz/graphRAGmcp

System Architecture

graphRAGmcp/
├── server.py              # MCP server (FastMCP) - main entry point
├── graph_index.py         # Pre-computed graph index - O(1) traversal
└── nano_graphrag/
    ├── base.py            # Query parameters and data structures
    ├── _op.py             # Entity extraction, 28 relationship types
    ├── graphrag.py        # Core RAG implementation
    ├── prompt.py          # LLM prompt templates
    ├── _llm.py            # LLM integration (OpenAI)
    ├── _splitter.py       # Text chunking
    └── _utils.py          # Utility functions

Key Implementation Details

The MCP server (server.py) exposes GraphRAG via JSON-RPC/HTTP with SSE streaming. The surgical endpoint (grand_debat_query_all_surgical) queries N communes in parallel via asyncio.gather(), aggregating per-commune responses.

The graph index (graph_index.py) provides pre-computed in-memory adjacency lists that eliminate per-query GraphML parsing (50x speedup: 25-30s to 0.5s). Relationship weights encode semantic priority:

The weighted Dijkstra traversal (expand_weighted) performs multi-hop expansion from seed entities using a priority queue, with defaults of max 2 hops and 200 results, and optional commune filtering and chunk inclusion.

The system supports 12 civic entity types (PROPOSITION, THEMATIQUE, SERVICE_PUBLIC, DOLEANCE, ACTEUR_INSTITUTIONNEL, OPINION, CITOYEN, CONCEPT, REFORME_DEMOCRATIQUE, TERRITOIRE, COMMUNE, CONTRIBUTION) and 28 semantic relationship types.

Each commune directory contains GraphML knowledge graphs, JSON text chunks, full documents, and entity embeddings. Chunks connect to entities via source_id attributes (separated by <SEP>).

The MCP client (mcp_client.py) and Dust client (dust_client.py) implement a shared RAGClient interface. The hallucination metric is inverted from OPIK's default (1.0 = faithful in our results).

Deployment uses Railway with Docker/Python 3.11, with approximately 30 seconds of cold start for graph index loading.

Contact: arthur.sarazin@etu-iepg.fr | Project Repository: https://github.com/ArthurSrz/graphRAGmcp | OPIK Dashboard: https://www.comet.com/opik/law_graphRAG