pdfgeo — geometry-aware PDF text reconstruction and phrase search on PDFium character boxes What it is One Python module (pdfgeo.py) with a CLI. It reads every character PDFium can see on every page (Unicode, tight glyph box, loose advance box, rotation angle, font name, generated flag, unicode-map-error flag), rebuilds words, lines, columns and blocks from that geometry, classifies page furniture (line-number gutters, timestamp lanes, running headers/footers/stamps, page numbers, off-page text), and writes readable text plus per-page structured metadata. A second command searches phrases block by block and returns page coordinates. Nothing is discarded: every classified item is kept in the metadata with its text, box, source character range and the reason. It does NOT use PDFium's or Poppler's own text order as the algorithm; those are only baselines. It does no OCR and calls no model. Text that exists only as pixels is not extracted. Pages and image regions without glyph coverage are flagged by a heuristic on page objects (see Signals); the flags are not an exhaustive census. Requirements Python 3.10+ and pypdfium2 (see requirements.txt). Pillow only for `overlay`. Usage python3 pdfgeo.py extract [--chars] out_dir must not exist. Writes: text.txt readable text; each page starts with "===== page N | body_glyphs=.. blocks=.. removed=.. ocr_risk=.. signals=.. =====" and blocks separated by blank lines. A page with no body text prints "[no body text in native layer on this page]". pages.jsonl one JSON record per page (schema below). summary.json totals, per-phase timing, pypdfium2/pdfium versions, sha256 of the pdf and of the module, coordinate convention. chars.jsonl (--chars) every PDFium character: page, index, char, generated flag, loose box, tight box, angle, font, unicode-map-error. Provenance only. python3 pdfgeo.py search "phrase" [--regex] [--include-removed] [--json] Case-insensitive; whitespace in the query matches any run of whitespace; curly quotes/dashes are normalised on both sides. A hit never crosses an inferred block; when block inference itself merges two regions (see Limits), a hit can still join them. Each hit gives page, block id, 1-based line numbers inside the block, one bbox per line, context, the matching variant (raw / dehyphenated) and the page's ocr_risk. --include-removed also searches classified furniture. python3 pdfgeo.py overlay out.png [--scale S] Renders the page with blocks (green), lines (blue), gutter tokens (red), running header/footer/stamps (orange), page numbers (purple), off-page text (black) and column separators (magenta). python3 selfcheck.py [probe phrase] Deterministic invariants over a finished run (see Selfcheck). Exit 1 on failure. Coordinates PDF user space points, origin bottom-left, box = [left, bottom, right, top]. Word/line/block boxes are unions of PDFium loose character boxes (advance boxes, full line height), so they are slightly larger than the ink. Algorithm (pdfgeo.py, function names in brackets) 1. Primitives [page_primitives, page_images]. For each PDFium character: unicode, FPDFText_IsGenerated (PDFium-inserted spaces and CR/LF), FPDFText_GetCharBox, FPDFText_GetLooseCharBox, FPDFText_GetCharAngle, FPDFText_GetFontInfo, FPDFText_HasUnicodeMapError. The C return codes are asserted for every non-generated character. Image objects are enumerated recursively through form XObjects to get image regions in page space. 2. Runs [build_runs]. Generated characters and controls are dropped from geometry (kept in chars.jsonl). Characters are grouped in PDFium index order into runs while they stay on one baseline (loose-box vertical overlap >= 50% of the smaller height), advance left-to-right and do not jump more than 3 em. Runs are the unit that protects against x-sorting two overprinted strings into each other. 3. Words [run_words]: split at real space glyphs or at a horizontal gap > 0.12 em. 4. Off-page [page_layout]: a run with any word centred outside the page box is classified "offpage" (clipped web-print headers etc.). 5. Gutter lanes [detect_gutters, gutter_value]. Candidate tokens are short numerals (<= 4 digits) or hh:mm[:ss] times. They are clustered into vertical bands by x centre (1.5 em). A band is a gutter only if ALL of: >= 4 tokens; the longest top-to-bottom increasing subsequence covers >= 80% of them; numeral steps are multiples of one common step (1 for transcripts, 5 for patents); the band spans >= 20% of the page height; at most 10% of text rows have a word crossing the band's channel; and, for a left/right band, at most 20% of its tokens have any text on their outer side (this is what separates a transcript gutter from a docket table's number column). Rejected bands are reported with the failing test in gutter_candidates_rejected. 6. Column separators [column_separators]. On a grid of x positions across the middle 70% of the page, a position is a separator if >= 8 text rows have a whitespace gap (> 0.9 em) covering it, that is >= 60% of the rows in its span, and the span is >= 30% of the page height. Adjacent accepted positions merge. 7. Segments (physical lines) [words_to_segments, split_at_separators]. Runs are placed in rows by vertical overlap; on a row, adjacent pieces merge when the gap is <= 1.5 em (<= 3 em when the left piece is <= 3 characters, which keeps "Q", "A", "1." with their line), and never when they overlap by more than half a word (overprint). Segments are then cut at column separators inside the separator's vertical span when the gap at the cut is >= 0.9 em, or, for strong separators (>= 20 rows) outside the margins, at a 1-2 digit numeral sitting in the channel. Rotated runs become their own segments and are excluded from reading order. 8. Furniture [classify_furniture]. Segments whose centre lies in the top or bottom 12% of the page are keyed by digit-masked normalised text and y band (2% of page height). Pass 1: a key seen on >= 20 pages is furniture regardless of context. Pass 2: after linking the remaining segments into blocks, a segment in a block of <= 3 lines is furniture if its key recurs on >= 3 pages (>= 10 when the key has no letters, to spare table dates), a lone numeral/roman numeral in the margin is a page number, and a small block anywhere on the page whose key equals a >= 20-page header/footer is a "stamp_outside_margin". 9. Blocks [link_blocks]. A segment links to the segment directly below it when the next row starts within 2.4 em, horizontal overlap is >= 30% of the narrower one, heights are within 60%, and the pairing is unique in both directions (a line that has two candidates below it, or a line with two above, does not link — this keeps full-width headings from gluing two columns together). 10. Reading order [xy_order]: recursive XY-cut over block boxes. Vertical cuts are taken first when they produce tall parts (columns), otherwise horizontal cuts, otherwise top-to-bottom, left-to-right. 11. Signals [page_signals]: no_body_text, image_dominant_low_text, image_dominant_with_text (a text layer under a full-page scan — usually OCR), image_region_without_text (an image >= 5% of the page containing almost no body glyphs: pasted figures, equations, screenshots), image_region_with_text, unicode_map_errors, low_alpha_ratio, irregular_tokens, rotated_text_present, overprint_present, offpage_text. ocr_risk folds them into high / medium / low / none. It is a risk signal derived from object geometry, not a visual census. 12. Search [block_variants, search]. Each block yields a "raw" stream (lines joined by spaces) and, only when a line ends in a hyphen after a lowercase letter and the next line starts lowercase, a "dehyphenated" stream. Every character of a stream maps back to (line, word), so hits come back with per-line boxes. pages.jsonl record page, width, height, counts: pdfium_chars, generated, glyphs, words, odd_words, segments, body_segments, body_glyphs, blocks, removed image_area_fraction, image_regions: [{box, area_fraction, body_glyphs_inside, text_covered}] signals: [...], ocr_risk: high|medium|low|none gutters: [{side, x_center, count, values, kind, step, median_pitch_pt, channel_crossing_rows, outliers_kept}] gutter_candidates_rejected: [{side, x_center, count, values, reason}] column_separators: [{x, x_range, rows_supporting, rows_in_span, y_span}] blocks: [{id, box, lines: [{text, box, h, words: [{t, box, chars: [first, last]}]}]}] removed: [{kind, text, box, chars: [first, last], reason}] `chars` are PDFium character indices on that page (0-based), the join key into chars.jsonl. Selfcheck (selfcheck.py) pages.jsonl covers 1..N in order; recorded PDFium char counts match the file; every word and removed item has a valid char range and a non-inverted box inside its block; body words + removed words == words seen on the page (nothing lost or double-counted); gutter values are monotone and no gutter token survives in the body at the gutter x; zero-body pages carry the explicit marker and are not ocr_risk=none over an image; and every search hit's boxes lie inside one block. Limits, stated plainly - Text present only as pixels (scans without a text layer, drawings, pasted equations, screenshots) is not extracted. Such pages/regions are flagged. - Where a scan carries an OCR text layer, output quality is that of the embedded OCR (dropped hyphens, wrong glyphs, misplaced boxes). Pages are flagged medium. - Reading order inside tables is column-major within the table region; a multi-line cell stays together but the table is not reconstructed as rows. - Tight table columns whose inter-column gap is under ~1 em are not separated. - A gutter lane that is only partially present in an OCR layer (missing numerals) can leave a stray numeral inside a merged line near the page margins. - Furniture classification needs recurrence: a header that appears on one or two pages stays in the body. A page number printed as "Page 214" is only removed when the masked key recurs. - Centre-aligned cover pages with stacked short lines are ordered by geometry, which can differ from the author's visual grouping. - Every threshold above was tuned on one document (the 373-page nonconfidential joint appendix in EcoFactor v. Google, CAFC 23-1101, ECF 15). No accuracy figure is claimed and no generalisation is claimed. Measured findings are reported on the accompanying public report and evaluation.json. The source.zip bundle also carries portable-evaluation-final.zip with the evaluator's runnable controls and versioned evaluation artifacts. outputs.zip is the accompanying complete all-page output archive.