TL;DR

Different queries need different chunk sizes, but RAG systems commit to one size upfront. In this blog, we show that indexing the same corpus at multiple chunk sizes (e.g., 100, 200, 500 tokens) and aggregating results with Reciprocal Rank Fusion (RRF) improves retrieval by 1–37% across benchmarks, without retraining models. Oracle experiments reveal 20-40% headroom when selecting optimal chunk size per query. [See it in code]

Is chunk size query-dependent?

Every RAG system faces the same chunking dilemma: smaller chunks preserve fine-grained details but lose context; larger chunks capture broader meaning but dilute specific facts. The conventional solution is to find a “sweet spot” chunk size (typically 500-800 tokens) that performs well on average.

But what if there is no universal sweet spot? What if different queries fundamentally need different chunk sizes?

This tradeoff is fundamental to fixed-dimensional embeddings. As chunk size grows, a fixed-size vector must compress increasingly diverse information, inevitably losing fine-grained details. Conversely, smaller chunks preserve specificity but sacrifice the surrounding context needed to understand meaning.

This raises a natural question:
Is there a single chunk size that serves all queries equally well?

As we show below, the answer is no.

To investigate this, we evaluated retrieval performance across multiple chunk sizes and datasets. The results reveal a consistent pattern: the best-performing chunk size varies substantially across queries, even when querying the same corpus.

This observation motivates a shift in perspective. Instead of committing to a single segmentation strategy, we ask whether multiple representations of the same text can be combined in a principled way to improve retrieval robustness.

Prior work

Several approaches have tackled this problem. Anthropic’s contextual retrieval enriches chunks with document-level context. Jina AI’s late chunking captures both coarse and fine-grained signals by chunking in the latent space. RAPTOR builds hierarchical summaries over fixed-size chunks.

However, all these methods still commit to a fixed chunk size. They also add complexity: some require retraining embedding models; others use LLMs to generate synthetic context, potentially introducing noise.

In contrast, our approach keeps chunking simple and instead exposes multiple resolutions at retrieval time.

The hypothesis

Standard practice treats chunk size as a one-time tuning parameter: choose a value, measure average performance, iterate until satisfied, then deploy. This optimization targets the mean. But averages can be misleading.

If queries truly have different optimal chunk sizes, then optimizing for average performance necessarily means underperforming on many individual queries. The question becomes: how much performance are we leaving on the table?

This, combined with the inherent limitations of embedding models, led us to the following hypothesis:

Different queries benefit from different chunk sizes, and selecting an appropriate chunk size at inference time can substantially improve retrieval performance.

Before proposing a practical method, we first test whether this hypothesis holds empirically.

Oracle experiments: is chunk size query-dependent?

To isolate the effect of chunk size, we ran a controlled experiment across several retrieval benchmarks.

Experimental setup

We evaluated on three diverse benchmarks:

– QMSum: Meeting transcripts with queries about specific discussion points. Documents average 5,000+ tokens with information distributed across speakers and topics.

– NarrativeQA: Full-length stories and book chapters where queries require understanding interconnected plot elements at varying levels of detail.

– Seinfeld (custom): Trivia questions over TV episode transcripts, testing retrieval of both specific facts and broader contextual understanding.

Chunk Size Is Query-Dependent: A Simple Multi-Scale Approach to RAG Retrieval

These datasets span structured meeting notes, narrative fiction, and conversational dialogue, representing different information distributions and query types.

For each dataset, we:

1. Created multiple indices of the same corpus, each with a different sliding-window chunk size (e.g., 50, 100, 200, 500, 1000, 2000 tokens).

2. For each query, retrieved top-k chunks independently from each index.

3. Evaluated document-level recall@K: did any chunk from the correct document appear in the top-k results? This metric serves as a proxy for downstream success: retrieval must first surface the correct document before any generation step can succeed.

4. Introduced an oracle aggregator that, for each query, selects whichever chunk size achieved the highest recall (assuming access to ground truth).

The oracle represents an upper bound: if we could perfectly predict the best chunk size per query, what performance could we achieve?

Results

The results strongly support our hypothesis. Across all datasets, a consistent pattern emerges: 

Chunk Size Is Query-Dependent: A Simple Multi-Scale Approach to RAG Retrieval
Figure 1: Chunk size performance vs. oracle upper bound. Oracle’s superior performance across all K values and across all datasets confirms that optimal chunk size varies by query.
Figure 1: Chunk size performance vs. oracle upper bound. Oracle's superior performance across all K values and across all datasets confirms that optimal chunk size varies by query.
Figure 1: Chunk size performance vs. oracle upper bound. Oracle's superior performance across all K values and across all datasets confirms that optimal chunk size varies by query.
Figure 1: Chunk size performance vs. oracle upper bound. Oracle's superior performance across all K values and across all datasets confirms that optimal chunk size varies by query.
Figure 1: Chunk size performance vs. oracle upper bound. Oracle’s superior performance across all K values and across all datasets confirms that optimal chunk size varies by query.

Key findings:

  • As can be seen from the chart, no single chunk size dominates: the “Oracle” is better than any single chunk size. This supports the notion that what works for one question often fails for another.
  • The oracle substantially outperforms all fixed choices. Gaps of 20-30% in recall@1 are common, reaching 40%+ in some datasets.
  • This headroom is systematic, not dataset-specific. The pattern holds across very different text types: meeting transcripts (QMSum), TV scripts (Seinfeld), and narrative stories (NarrativeQA). 

This confirms the hypothesis: optimal chunk size is strongly query-dependent, and the choice materially affects retrieval quality.

The oracle experiment is deliberately unrealistic. At inference time, we do not know which chunk size will work best for a given query.

This leads to a practical question:
Can we approximate the oracle without explicitly predicting chunk size, retraining embedding models, or introducing query-specific heuristics?

Method: Multi-scale indexing with rank-based aggregation

Figure 2: Multi-scale indexing and aggregation pipeline. At ingestion time, documents are chunked at multiple window sizes (w₁, w₂, ..., wₙ) and indexed separately. At inference time, the query is issued to all indices in parallel, retrieving top-k chunks from each. Chunks vote for their parent documents, and final rankings are determined via Reciprocal Rank Fusion (RRF).
Figure 2: Multi-scale indexing and aggregation pipeline. At ingestion time, documents are chunked at multiple window sizes (w₁, w₂, …, wₙ) and indexed separately. At inference time, the query is issued to all indices in parallel, retrieving top-k chunks from each. Chunks vote for their parent documents, and final rankings are determined via Reciprocal Rank Fusion (RRF).
Figure 2: Multi-scale indexing and aggregation pipeline. At ingestion time, documents are chunked at multiple window sizes (w₁, w₂, ..., wₙ) and indexed separately. At inference time, the query is issued to all indices in parallel, retrieving top-k chunks from each. Chunks vote for their parent documents, and final rankings are determined via Reciprocal Rank Fusion (RRF).
Figure 2: Multi-scale indexing and aggregation pipeline. At ingestion time, documents are chunked at multiple window sizes (w₁, w₂, …, wₙ) and indexed separately. At inference time, the query is issued to all indices in parallel, retrieving top-k chunks from each. Chunks vote for their parent documents, and final rankings are determined via Reciprocal Rank Fusion (RRF).

To answer the question above, we tried to come up with a simple and cheap method that considers several indices (corresponding to chunk sizes) in the retrieval phase. The ingestion phase remains the same and mirrors the oracle setup. We apply simple sliding-window chunking at multiple window sizes and build a separate index for each size. No learned chunking, no special preprocessing.

At inference time, the query is issued to all indices in parallel, producing multiple ranked lists of chunks (one per chunk size).

The remaining step is aggregation.

From chunks to documents

Our approach treats retrieval as a voting process. Each retrieved chunk “votes” for its parent document, and we aggregate these votes across all chunk sizes to produce a final document ranking.

Rather than using embedding similarity scores, which aren’t directly comparable across different chunk sizes, we use Reciprocal Rank Fusion (RRF), a model-agnostic rank aggregation method.

Why RRF? It’s simple, robust, and doesn’t require score calibration across indices. A chunk’s vote weight depends only on its rank position, not its absolute similarity score.

The score for document I is:

SI=∑w∈W∑i∈Iw1riw+1S_I = \sum_{w\in W}\sum_{i\in I_w} \frac{1}{r_{iw}+1}

where:

  • WW is the set of chunk sizes (e.g., {50, 100, 200, 500, 1000})
  • IwI_{w} is the set of chunks from document I retrieved at window size ww  
  • riwr_{iw} is the rank of chunk ii in index ww (higher rank = higher vote weight)

Intuitively, this promotes documents that are consistently represented: either through many hits within a single chunk size or across multiple chunk sizes.

The main limitation of our proposed method is the indexing cost – the number of chunks to embed and store is multiplied by a factor of 2-5 across our experiments (corresponding to the amount of windows).

Results

MTEB (retrieval subset)

We evaluated the method on a subset of MTEB retrieval tasks using two off-the-shelf embedding models: intfloat-multilingual-e5-small and nomic-embed-text-v1.

For each configuration, we compare:

  • Naive: standard single-chunk-size indexing (baseline)
  • RRF by size: our multi-scale approach with RRF aggregation 

Across 7 out of 8 configurations, multi-scale retrieval with RRF improves over the naive single-chunk baseline. While the absolute gains may appear modest (1–3%), they are meaningful in the context of mature retrieval benchmarks, where improvements between state-of-the-art models are often measured in fractions of a percent.

MTEB (retrieval subset) results
Figure 3: results on MTEB

Notably, these gains are achieved:

  • without retraining the embedding model.
  • without changing the retriever.
  • using only inference-time aggregation.

The dramatic improvement on TRECCOVID (+36.7% for E5-small) suggests some datasets benefit particularly from multi-scale representations, possibly due to query diversity or document structure.

Other benchmarks

We observe similar trends on FinanceBench, NarrativeQA, QMSum, and the Seinfeld dataset. In all cases, multi-scale aggregation matches or exceeds the best fixed chunk size and often approaches the oracle’s performance envelope.

We observe similar trends on FinanceBench, NarrativeQA, QMSum, and the Seinfeld dataset. In all cases, multi-scale aggregation matches or exceeds the best fixed chunk size and often approaches the oracle’s performance envelope.
Figure 4: consistent results on various datasets.
We observe similar trends on FinanceBench, NarrativeQA, QMSum, and the Seinfeld dataset. In all cases, multi-scale aggregation matches or exceeds the best fixed chunk size and often approaches the oracle’s performance envelope.
We observe similar trends on FinanceBench, NarrativeQA, QMSum, and the Seinfeld dataset. In all cases, multi-scale aggregation matches or exceeds the best fixed chunk size and often approaches the oracle’s performance envelope.
We observe similar trends on FinanceBench, NarrativeQA, QMSum, and the Seinfeld dataset. In all cases, multi-scale aggregation matches or exceeds the best fixed chunk size and often approaches the oracle’s performance envelope.
We observe similar trends on FinanceBench, NarrativeQA, QMSum, and the Seinfeld dataset. In all cases, multi-scale aggregation matches or exceeds the best fixed chunk size and often approaches the oracle’s performance envelope.
Figure 4: consistent results on various datasets.

Summary

Chunk size is often treated as a fixed preprocessing choice in retrieval. Our experiments suggest a different view: chunk size interacts with the query, and different questions benefit from different resolutions of the same document. 

Oracle experiments across multiple benchmarks show that selecting the best chunk size per query can significantly improve document-level recall. To approximate this behavior in practice, we index the corpus at multiple chunk sizes and aggregate retrieval results at inference time using Reciprocal Rank Fusion. 

This approach is simple, model-agnostic, and requires no retraining. The observed gains are comparable to those typically reported when switching embedding models, yet they come entirely from rethinking representation and aggregation. 

Rather than searching for a single globally optimal chunk size, multi-scale retrieval allows systems to benefit from multiple representations simultaneously, deferring the choice to inference time where query context is available. 

We encourage you to experiment with the example code and the Seinfeld dataset here.

Acknowledgements

This work was developed by Alan Arazi, Ido Weiss, Roee Hendel, and Niv Granot.