GraphScout

Multi-hop questions — "which university did the founder of the company that acquired DeepMind attend?" — break naive RAG, because the second hop is invisible from the surface question. Plenty of projects assert that graph RAG or an agent fixes it. GraphScout measures how much each fix actually buys, with confidence intervals.

Three arms share one corpus, one eval harness, and one set of metrics: a dense-vector baseline, a graph-RAG arm (hybrid retrieval → rerank → knowledge graph expansion), and an agentic arm that decomposes the question and routes sub-questions across vector, graph, and community tools before verifying its answer.

The infrastructure is the point as much as the result: a versioned eval harness, a calibrated LLM-as-judge, per-request tracing in Langfuse, and a CI gate that blocks quality regressions. Total infrastructure cost: $0. Total LLM spend: ~$30, kept there by model tiering, batch processing, prompt caching, and response caching.

_{*The graph arm's scored run hit a fully warm response cache. Every row traces to a committed, provenance-stamped run record — all three at the same git SHA, corpus and split.}

Scored as chunk-level coverage: a gold paragraph counts as retrieved only if some chunk covering it lands in the top-8, over a 50-candidate pool.

Agentic retrieval is what closes the multi-hop gap. Recall@8 nearly doubles over the dense baseline, and all-support@8 — every gold chunk for a question present in the top-8, which is what multi-hop answering actually requires — goes from 14% to 65%. Neither single-shot arm could touch that, and it took decomposition and iteration (6.2 tool rounds per question on average) rather than a smarter single query.

And retrieval was never the binding constraint on answers. Answer F1 moves 0.149 → 0.231 while recall moves 0.39 points, and exact match is actually lower than the graph arm (0.127 vs 0.140). Grounding sits at 0.613 and only 65.3% of agent runs finish cleanly. So the honest reading is that the agent now finds the evidence and still often fails to compose it into the gold answer — the next work is answer synthesis and verification, not more retrieval. That half is stated because a 2× recall headline sitting on top of a flat F1 is exactly the result a portfolio is tempted to crop.

Within single-shot retrieval, an earlier free ablation isolates why arm 2 helps at all: both single-shot arms plateau at the same ~0.62 recall@50, so hybrid retrieval and reranking don't find evidence dense search missed — they pull known-good evidence higher (MRR 0.68 → 0.77). Single-shot retrieval gets all gold chunks into the top-8 for only 19% of questions, which is what makes the agentic arm's 65% the interesting number.