Page-Chunked Classification Is Cheap and Predictable. It's Blind to the Table That Crosses a Page.

The cost model is a page. The document isn't.

The last post on batch-classifying 400 million documents got a sharp question in the comments: Gemini's native PDF handling gives you a fixed, predictable token cost per page — 258 tokens per page is the actual number Google publishes — so a 50-page document costs roughly 50x a 1-page one. That's what makes costing 400 million documents a multiplication instead of a guess. The question was: what happens to a table that starts on page 12 and ends on page 13, once you're chunking and classifying by page?

Nothing good, by default. And the failure is quiet — it doesn't throw an error, it just returns a confident, wrong label.

A page break is a rendering artifact, not a semantic one

Nothing in the PDF format guarantees that a table, a clause, or even a sentence finishes before the page does. A contract's indemnification table can run four rows on page 12 and six more on page 13 with no repeated header row — it's one table to a human, and two unrelated fragments to a classifier that only ever sees one page at a time.

Page 13 in isolation looks like six rows of numbers with no column headers. A classifier asked "what kind of clause is this?" has nothing to anchor on, so it either falls back to a default label or, worse, pattern-matches on the numbers themselves and produces something confident and wrong. The error doesn't look like an error. It looks like a label.

Classifying page-by-page Page 12 Item | Qty | Price (table continues...) label: "pricing schedule" ✓ Page 13 (seen alone) no header, no context label: "uncategorized" ✗ (same table, wrong answer)

The fix: carry context across the break, and merge tables structurally

Two separate mechanisms, because they solve two different versions of the problem.

For prose — a sentence or clause that just happens to end mid-page — a small overlap window is enough: carry the tail of page N into the prompt for page N+1, the same sliding-window technique used for chunking long documents generally. A typical overlap is 10–20% of the chunk's token budget; for a 258-token page, that's on the order of 25–50 tokens of trailing context, not a second copy of the page.

def chunk_pages_with_overlap(pages: list[str], overlap_tokens: int = 40) -> list[str]:
    """pages: per-page text, in order. Each chunk gets the tail of the
    previous page prepended, so a fragment that started before the
    break has the context it needs to be classified correctly."""
    chunks = []
    for i, page in enumerate(pages):
        if i == 0:
            chunks.append(page)
            continue
        prev_tail = tokenize(pages[i - 1])[-overlap_tokens:]
        chunks.append(detokenize(prev_tail) + "\n" + page)
    return chunks

Tables don't degrade gracefully the same way prose does — a truncated sentence still reads as a sentence, but six rows with no header row don't read as anything. For tables specifically, overlap isn't the right tool. What works is detecting the continuation and reconciling it structurally before classification ever sees it:

def merge_spanning_tables(pages: list[PageLayout]) -> list[PageLayout]:
    """If a page's layout ends mid-table and the next page opens with
    rows that match the same column count but no header row, treat it
    as one table split across the break and stitch it back together."""
    for i in range(len(pages) - 1):
        last_table = pages[i].tables[-1] if pages[i].tables else None
        first_block = pages[i + 1].tables[0] if pages[i + 1].tables else None
        if last_table and first_block and is_continuation(last_table, first_block):
            last_table.rows.extend(first_block.rows)
            pages[i + 1].tables.pop(0)
    return pages

def is_continuation(table, next_block) -> bool:
    # same column count, and the next block has no header-shaped first row
    return (
        table.column_count == next_block.column_count
        and not looks_like_header(next_block.rows[0])
    )

Once merged, the full table — headers included — is what gets classified, not an isolated fragment of it. This is the same principle as the dependency graph in the RCA-agent post from a few weeks back: give the model the structure it needs to reason correctly, instead of asking it to infer structure from an incomplete view.

Why this doesn't break the cost model

The overlap window and the table merge both add a handful of extra tokens per chunk — not a second pass over the document, not a bigger model. And the classification instructions themselves (the label set, the schema, the system prompt) are identical across every chunk in the batch, which is exactly what prefix caching is for: the shared prefix is computed once and its KV cache is reused for every subsequent chunk that starts with it, rather than recomputed per request. Real-world hit rates on a shared, static prefix like a classification prompt are commonly in the 60–85% range, which is most of why the per-page cost stays close to the fixed number rather than creeping up every time you add a safety mechanism.

So the honest cost model isn't "pages times 258 tokens." It's "pages times 258 tokens, plus a small, mostly-cached overlap." The second number is close enough to the first to still be worth budgeting as a multiplication — it just isn't exactly one.

Lesson

Fixed-cost-per-page is a billing abstraction. It tells you what the bill looks like, not what the document looks like. Anything that can span a page — a table, a clause, a continued sentence — needs an explicit reconciliation step before classification, because the model never gets a vote on where the page breaks; the renderer already decided that, for reasons that have nothing to do with meaning.


Hitting something like this in production? I help teams with performance engineering, SRE/observability, and AI-driven root cause analysis — work with me.

Comments

Popular posts from this blog

Performance Testing 102: Little's Law and It's usage in Performance Testing

Performance Testing 104: Workload Modelling Designing & Process

Boosting Apache Web Server Performance: A Comprehensive Guide to Tuning and Optimizing