Four code-intelligence tools for AI coding agents, plus a no-MCP grep baseline, run through the same six end-to-end exploration scenarios on six real repositories. Two of the four MCP servers scored below plain grep. Here is the full scorecard, the method, and where every tool loses, including the one I built.
Disclosure, up front. I am the author of Sense, one of the five entries below. So the scorecard is built to be checked, not believed: open code, frozen held-out scenarios with hand-graded reference scores, a Spearman ≥ 0.85 anchor against my own grading, and a published list of every place Sense loses. Don't take my word for any single number. Audit the whole thing →
Ranked by fairness: a weighted composite of answer quality, citation grounding, efficiency, and a keyword smoke test (formula below). Averaged across all six scenarios. Best value in each column is highlighted.
| Rank | Tool | Fairness | Judge quality | Grounding | Avg tokens | Avg time | Total cost | vs baseline |
|---|---|---|---|---|---|---|---|---|
| 🥇 #1 | sensesymbol graph + embeddings over MCP | 81.3% | 85.4% | 89.2% | 10,896 | 141.5s | $6.22 | +17.8% |
| 🥈 #2 | probestateless tree-sitter + ripgrep | 77.7% | 84.8% | 72.8% | 12,119 | 162.7s | $6.23 | +17.7% |
| 🥉 #3 | baselineClaude Code · grep / find / Read · no MCP | 77.2% | 84.2% | 80.8% | 12,716 | 185.4s | $7.57 | — |
| #4 | serenaLSP-driven symbol cache | 75.2% below grep | 83.4% | 61.9% | 14,800 | 191.4s | $7.57 | 0.0% |
| #5 | gitnexuscode knowledge graph · Cypher-style | 74.9% below grep | 84.5% | 76.9% | 12,964 | 173.8s | $6.87 | +9.2% |
The finding that should give every team pause: attaching serena or gitnexus, as configured out of the box, made Claude Code worse than no MCP at all on the headline fairness score. An MCP server is not free. It has to earn back the friction it adds. Per-scenario tables for all six repos (axum, discourse, flask, gin, javalin, next.js) are in the full report.
Five ways an AI agent can navigate an unfamiliar codebase. Same Claude Code harness, same six repos pinned at the same commits. Only the MCP configuration changes between runs.
Pre-built symbol graph plus embeddings, served over four MCP tools: search, graph (callers/callees), blast (impact), conventions. Local Go binary.
No index. Every query parses touched files on the fly with tree-sitter and ripgrep, with MCP scoping. Zero setup, stateless.
No MCP at all. grep, find, Read, the same primitives a human dev uses. The floor every real tool has to beat.
Spins up real language servers (Pyright, gopls, rust-analyzer), caches their symbol graphs, exposes find_symbol / find_references over MCP.
Builds a graph database from the repo and exposes Cypher-like queries plus a cross-repo registry over MCP.
The field will grow. This is a living scorecard, and the harness can score any code-intel MCP that runs as a drop-in server in Claude Code. New tools get added as they appear.
One judge score is not a benchmark. The other axes exist to catch what a single quality score misses: a beautiful but unverifiable answer is penalized on grounding, a slow drift through grep is penalized on efficiency.
Every file:line reference in an answer is checked against the repo at the pinned commit. A line past end-of-file is flagged as a hallucination, not paraphrase. Sense grounded 89.2% of its citations; serena, the lowest, 61.9%.
How often the agent reaches for the MCP is computed separately and never folded into fairness. Otherwise the no-MCP baseline takes a structural zero. That separation is how baseline can, and does, beat two of the MCP servers.
Three held-out scenarios with hand-graded scores, SHA256-locked. The scoring is only trusted when it correlates above 0.85 (Spearman) with my own grading. Goodhart's Law applies to your own benchmark too.
Keyword hits are searched in the agent's answer text, never its tool-call inputs. Otherwise grepping for a symbol would score a "hit" even when grep returned nothing, a tax on tools that don't use grep.
Full scoring definitions: SCORING.md.
No tool wins everywhere, including the one that wins on aggregate. If a benchmark only lists where the author's tool shines, it's a press release. This is the honest column.
sense scan before the first query (minutes on most repos,
about 20 on a 75k-symbol monorepo like Next.js). On tiny codebases where the agent can
just read the file, the index earns nothing. Line numbers drifted on one Rust citation.
The .sense/ index lives on disk per repo, worth sizing inside small CI
containers.
Nothing here can drift from the raw data. Every cell is derived from per-run transcripts, scores, and judge outputs committed in the repo. The whole benchmark replays end to end.