Planner and evaluation harness
TokenSmith Query Decomposition
Across three passes on 19 questions, the planner reported a 4.04 vs 3.93 mean judge score, 0.580 vs 0.579 keyword recall, and 23.3s vs 14.7s mean request latency.
- Public planner/evaluation snapshot
- Project status not reported
01 / Overview
The problem and contribution
- Problem
- TokenSmith's documented baseline uses single-pass retrieval and generation. The project explores whether a planner can route complex questions through decomposition and evidence synthesis, then compares that approach with the baseline.
- Intended audience
- The public snapshot is presented as a course-project artifact for reviewers evaluating multi-hop retrieval and its measurement.
- Contribution
- The public README describes this as my project and identifies the planner layer: simple-versus-complex classification, dependency-aware decomposition of complex questions into two to five subqueries, and evidence synthesis.
- Implementation
- The public snapshot contains the planner files, evaluation scripts, benchmark materials, and aggregate outputs. The documented flow classifies a question, decomposes complex questions, then merges, deduplicates, and reranks evidence before answer generation. The evaluation scripts use a separate local TokenSmith backend and locally configured GGUF models; that backend is not included in this snapshot.
02 / Evidence preview
What the public materials show
No reviewed screenshots are published for this case study.
The architecture summary below is the available visual overview.
Planner and evaluation flow
- Classify the question as simple or complex.
- For complex questions, create two to five dependency-aware subqueries.
- Merge, deduplicate, and rerank evidence before answer generation.
03 / System design
Technical decisions and tradeoffs
Route questions through a simple-versus-complex classifier, and decompose complex questions into dependency-aware subqueries.
The public README documents selective decomposition, but does not publish a more specific rationale for the routing boundary.
Tradeoff: The planner adds a classification step. The reported classifier accuracy is 68.4% overall, with lower accuracy for complex questions (54.5%).
Documented baseline: TokenSmith's documented single-pass retrieval-and-generation baseline.
Synthesize evidence by merging, deduplicating, and reranking it before generating the final answer.
This is the evidence-synthesis sequence described in the public planner snapshot; a separate rationale is not published.
Tradeoff: The three-pass aggregate reports a modest change in judge score and essentially unchanged keyword recall, alongside higher mean request latency.
Documented baseline: The documented single-pass baseline.
04 / Evaluation
Measured results and limits
Three-pass aggregate
The reported comparison aggregates three passes over the same 19-question benchmark. It uses a small local judge model; keyword recall is based on string matching; latency includes request overhead.
These results describe this benchmark and setup only. They do not establish broad superiority or statistical significance.
| Metric | Baseline | Planner | Interpretation |
|---|---|---|---|
| Mean judge scorescore / 5 | 3.93 / 5 | 4.04 / 5 | Small measured gain in this evaluation. |
| Keyword recallrecall | 0.579 | 0.580 | Essentially unchanged. |
| Mean request latencyseconds | 14.7 s | 23.3 s | Approximately 58% higher with the planner. |
| Classifier accuracyaccuracy | — | 68.4% | Overall routing accuracy. |
| Simple-question classificationaccuracy | — | 87.5% | Reported simple-query classification accuracy. |
| Complex-question classificationaccuracy | — | 54.5% | Reported complex-query classification accuracy. |
- Three passes reuse the same 19-question benchmark.
- The judge is a small local model, and keyword recall uses string matching.
- Latency includes request overhead and is higher for the planner in this aggregate.
- The figures do not establish broad superiority or statistical significance.
05 / Reflection
Lessons and current limitations
- Read retrieval quality, recall, and latency together: the reported judge score changes slightly, recall is effectively flat, and latency increases.
- Classifier performance, especially on complex questions, is a material limit on a planner that routes work by query type.
- The public repository is a planner and evaluation snapshot, not the full TokenSmith backend.
- Standalone reproduction depends on a separately configured backend and local models.
- The benchmark is small and reused across its three passes; its metrics have the methodology limits described above.
- The public materials do not provide a separate rationale for each design choice.
06 / Public presentation boundary
What this page represents
Public planner/evaluation snapshot Planner and evaluation snapshot; separate backend not included.
The linked repository is a public planner/evaluation snapshot. It does not include the separate TokenSmith backend. A repository license was not verified, so this page calls it a public snapshot rather than open source.
Content and evidence reviewed 2026-10-01.
07 / Walkthrough
Video overview
Video not published. Request a technical walkthrough.
Evidence references
Public sources
- TokenSmith public planner/evaluation snapshot README Reviewed 2026-10-01
- Three-pass aggregate results Reviewed 2026-10-01