One Million Tokens, Four Needles

A million-token context window can hold roughly eight novels or 50,000 lines of code. That is a capacity claim. It is not a reliability claim.

The demo most often used to collapse those two ideas is needle in a haystack: hide one fact in a long prompt, ask for it back, celebrate near-perfect retrieval. Google’s own long-context guide warns that single-needle performance does not carry over unchanged when several target facts are hidden in the same context. OpenAI made the same distinction when it released MRCR, an evaluation with two, four, or eight near-identical requests buried among distractors.

Real work is full of four-needle problems. Find the latest budget, not the superseded one. Join it to the right project owner. Notice that the deadline moved in a different document. Cite the lines that support the answer.

This drill tests that job. It takes about an hour to build, then becomes a regression suite you can run whenever a model, retrieval policy, or prompt changes.

What we are testing

The question is not “Can the model find a string?” It is:

Can this system recover several related facts from a long, confusing corpus, combine them correctly, show its evidence, and do so at an acceptable cost and latency?

Five variables matter:

1. Context size.
2. Number and position of target facts.
3. Similarity of distractors.
4. Complexity of the requested join.
5. Context policy: full prompt, retrieval, or just-in-time search.

Keep the model and question fixed while changing one of those variables. Otherwise you are comparing stories, not systems.

Step 1: make facts the model cannot remember

Create four synthetic records. Each needs a unique project, owner, deadline, and budget. Store the exact correct answer as JSON.

[
  { "project": "Juniper", "owner": "Mara Lee", "deadline": "2027-02-12", "budget": 480000 },
  { "project": "Lattice", "owner": "Owen Park", "deadline": "2026-11-03", "budget": 725000 },
  { "project": "Northstar", "owner": "Inez Shah", "deadline": "2027-01-19", "budget": 315000 },
  { "project": "Sable", "owner": "Theo Lim", "deadline": "2026-12-08", "budget": 560000 }
]

These are test fixtures, not findings. Synthetic facts prevent a model from answering from training data and give you an exact gold label.

Step 2: build distractors that can win

Easy distractors are random prose. Useful distractors are almost right.

Add a “Juniper Labs” project, a former owner named Maria Lee, an older budget of $408,000, and a meeting note that proposes February 21 without approving it. Put superseded values in plausible documents. Give two people the same surname. Make the model select evidence, not match keywords.

Use documents shaped like production: long sections, tables, version histories, and duplicated boilerplate. Synthetic needles inside clean filler will overstate performance.

Step 3: create four corpus sizes

Start around 8,000, 32,000, and 128,000 tokens. Make the fourth band the largest length your application could actually afford. Do not pay for one million tokens because the model card printed the number.

Rotate each target independently across the beginning, quarter points, middle, and end. Put the question at the end for the baseline; Google recommends that placement for long prompts. Then run a query-at-start variant to measure how much your particular workflow depends on order.

Step 4: ask three kinds of question

Run a single-fact lookup first. Then request all four records in one response. Finally, require a join:

Return projects with an approved budget above $450,000 whose current deadline falls in 2026. Include owner, budget, deadline, source document, and a supporting span for every field.

Require strict JSON. Ask for source identifiers and short supporting spans. Score the values deterministically, then check whether each cited span actually supports the field.

This prevents a fluent answer with invented evidence from passing.

Step 5: compare three context policies

Run the same questions three ways.

• Full context: send the entire corpus.
• Retrieved top-k: use a simple lexical or embedding search, then send only the highest-ranked chunks.
• Just-in-time search: let an agent issue targeted searches and fetch more context as needed.

Keep the model snapshot, sampling settings, question, and grader fixed. Repeat each condition at least five times when output is non-deterministic.

Anthropic’s context-engineering guide describes context as finite and subject to diminishing returns. More tokens can produce “context rot”: retrieval and long-range reasoning lose precision as the window fills. The goal is the smallest high-signal context that reliably completes the task, not the fullest prompt.

Step 6: show the bill

For every run, save:

• Raw request and response.
• Model snapshot and parameters.
• Input, output, and cached tokens.
• Time to first token and total latency.
• Tool and infrastructure errors.
• Structured answer and evidence score.
• Total cost.

Google exposes cache hits through usage.total_cached_tokens and recommends placing shared material at the beginning of a prompt. Cache behavior can turn two identical-looking experiments into different economic systems, so log it.

Publish the result as cost per fully correct answer:

Policy	Context tokens	Four-fact joint accuracy	Evidence accuracy	Median latency	Cost / correct run
Full context	measured	measured	measured	measured	measured
Retrieved top-k	measured	measured	measured	measured	measured
Just-in-time search	measured	measured	measured	measured	measured

Do not fill this table with one lucky run.

The thing to ship

Turn the winning condition into a context budget:

• Maximum full-context size.
• Retrieval trigger.
• Compaction trigger.
• Required evidence format.
• Latency and cost ceiling.
• Regression threshold.

A million-token window is useful when the task needs it. This drill tells you when it does—and when retrieval is faster, cheaper, and more accurate.

Run it before your next model migration. The most important number will probably not be maximum context. It will be the number of tokens your system can avoid.

---

This article specifies an experiment; it does not report a Supasifu model benchmark. Exact tokenization, caching, pricing, and context limits vary by model and provider.