pdfgeo — geometry-aware PDF text reconstruction and phrase search on PDFium character boxes

What it is
  One Python module (pdfgeo.py) with a CLI. It reads every character PDFium can see on
  every page (Unicode, tight glyph box, loose advance box, rotation angle, font name,
  generated flag, unicode-map-error flag), rebuilds words, lines, columns and blocks
  from that geometry, classifies page furniture (line-number gutters, timestamp lanes,
  running headers/footers/stamps, page numbers, off-page text), and writes readable
  text plus per-page structured metadata. A second command searches phrases block by
  block and returns page coordinates. Nothing is discarded: every classified item is
  kept in the metadata with its text, box, source character range and the reason.

  It does NOT use PDFium's or Poppler's own text order as the algorithm; those are
  only baselines. It does no OCR and calls no model. Text that exists only as pixels
  is not extracted. Pages and image regions without glyph coverage are flagged by a
  heuristic on page objects (see Signals); the flags are not an exhaustive census.

Requirements
  Python 3.10+ and pypdfium2 (see requirements.txt). Pillow only for `overlay`.

Usage
  python3 pdfgeo.py extract <file.pdf> <out_dir> [--chars]
      out_dir must not exist. Writes:
        text.txt      readable text; each page starts with
                      "===== page N | body_glyphs=.. blocks=.. removed=.. ocr_risk=.. signals=.. ====="
                      and blocks separated by blank lines. A page with no body text
                      prints "[no body text in native layer on this page]".
        pages.jsonl   one JSON record per page (schema below).
        summary.json  totals, per-phase timing, pypdfium2/pdfium versions,
                      sha256 of the pdf and of the module, coordinate convention.
        chars.jsonl   (--chars) every PDFium character: page, index, char, generated flag,
                      loose box, tight box, angle, font, unicode-map-error. Provenance only.
  python3 pdfgeo.py search <out_dir> "phrase" [--regex] [--include-removed] [--json]
      Case-insensitive; whitespace in the query matches any run of whitespace; curly
      quotes/dashes are normalised on both sides. A hit never crosses an inferred
      block; when block inference itself merges two regions (see Limits), a hit can
      still join them. Each hit gives
      page, block id, 1-based line numbers inside the block, one bbox per line,
      context, the matching variant (raw / dehyphenated) and the page's ocr_risk.
      --include-removed also searches classified furniture.
  python3 pdfgeo.py overlay <file.pdf> <out_dir> <page> out.png [--scale S]
      Renders the page with blocks (green), lines (blue), gutter tokens (red),
      running header/footer/stamps (orange), page numbers (purple), off-page text
      (black) and column separators (magenta).
  python3 selfcheck.py <file.pdf> <out_dir> [probe phrase]
      Deterministic invariants over a finished run (see Selfcheck). Exit 1 on failure.

Coordinates
  PDF user space points, origin bottom-left, box = [left, bottom, right, top].
  Word/line/block boxes are unions of PDFium loose character boxes (advance boxes,
  full line height), so they are slightly larger than the ink.

Algorithm (pdfgeo.py, function names in brackets)
  1. Primitives [page_primitives, page_images]. For each PDFium character: unicode,
     FPDFText_IsGenerated (PDFium-inserted spaces and CR/LF), FPDFText_GetCharBox,
     FPDFText_GetLooseCharBox, FPDFText_GetCharAngle, FPDFText_GetFontInfo,
     FPDFText_HasUnicodeMapError. The C return codes are asserted for every
     non-generated character. Image objects are enumerated recursively through form
     XObjects to get image regions in page space.
  2. Runs [build_runs]. Generated characters and controls are dropped from geometry
     (kept in chars.jsonl). Characters are grouped in PDFium index order into runs
     while they stay on one baseline (loose-box vertical overlap >= 50% of the
     smaller height), advance left-to-right and do not jump more than 3 em. Runs are
     the unit that protects against x-sorting two overprinted strings into each other.
  3. Words [run_words]: split at real space glyphs or at a horizontal gap > 0.12 em.
  4. Off-page [page_layout]: a run with any word centred outside the page box is
     classified "offpage" (clipped web-print headers etc.).
  5. Gutter lanes [detect_gutters, gutter_value]. Candidate tokens are short numerals
     (<= 4 digits) or hh:mm[:ss] times. They are clustered into vertical bands by x
     centre (1.5 em). A band is a gutter only if ALL of: >= 4 tokens; the longest
     top-to-bottom increasing subsequence covers >= 80% of them; numeral steps are
     multiples of one common step (1 for transcripts, 5 for patents); the band spans
     >= 20% of the page height; at most 10% of text rows have a word crossing the
     band's channel; and, for a left/right band, at most 20% of its tokens have any
     text on their outer side (this is what separates a transcript gutter from a
     docket table's number column). Rejected bands are reported with the failing
     test in gutter_candidates_rejected.
  6. Column separators [column_separators]. On a grid of x positions across the
     middle 70% of the page, a position is a separator if >= 8 text rows have a
     whitespace gap (> 0.9 em) covering it, that is >= 60% of the rows in its span,
     and the span is >= 30% of the page height. Adjacent accepted positions merge.
  7. Segments (physical lines) [words_to_segments, split_at_separators]. Runs are
     placed in rows by vertical overlap; on a row, adjacent pieces merge when the gap
     is <= 1.5 em (<= 3 em when the left piece is <= 3 characters, which keeps "Q",
     "A", "1." with their line), and never when they overlap by more than half a word
     (overprint). Segments are then cut at column separators inside the separator's
     vertical span when the gap at the cut is >= 0.9 em, or, for strong separators
     (>= 20 rows) outside the margins, at a 1-2 digit numeral sitting in the channel.
     Rotated runs become their own segments and are excluded from reading order.
  8. Furniture [classify_furniture]. Segments whose centre lies in the top or bottom
     12% of the page are keyed by digit-masked normalised text and y band (2% of
     page height). Pass 1: a key seen on >= 20 pages is furniture regardless of
     context. Pass 2: after linking the remaining segments into blocks, a segment in a
     block of <= 3 lines is furniture if its key recurs on >= 3 pages (>= 10 when the
     key has no letters, to spare table dates), a lone numeral/roman numeral in the
     margin is a page number, and a small block anywhere on the page whose key equals
     a >= 20-page header/footer is a "stamp_outside_margin".
  9. Blocks [link_blocks]. A segment links to the segment directly below it when the
     next row starts within 2.4 em, horizontal overlap is >= 30% of the narrower one,
     heights are within 60%, and the pairing is unique in both directions (a line
     that has two candidates below it, or a line with two above, does not link —
     this keeps full-width headings from gluing two columns together).
 10. Reading order [xy_order]: recursive XY-cut over block boxes. Vertical cuts are
     taken first when they produce tall parts (columns), otherwise horizontal cuts,
     otherwise top-to-bottom, left-to-right.
 11. Signals [page_signals]: no_body_text, image_dominant_low_text,
     image_dominant_with_text (a text layer under a full-page scan — usually OCR),
     image_region_without_text (an image >= 5% of the page containing almost no body
     glyphs: pasted figures, equations, screenshots), image_region_with_text,
     unicode_map_errors, low_alpha_ratio, irregular_tokens, rotated_text_present,
     overprint_present, offpage_text. ocr_risk folds them into high / medium / low /
     none. It is a risk signal derived from object geometry, not a visual census.
 12. Search [block_variants, search]. Each block yields a "raw" stream (lines joined
     by spaces) and, only when a line ends in a hyphen after a lowercase letter and
     the next line starts lowercase, a "dehyphenated" stream. Every character of a
     stream maps back to (line, word), so hits come back with per-line boxes.

pages.jsonl record
  page, width, height,
  counts: pdfium_chars, generated, glyphs, words, odd_words, segments, body_segments,
          body_glyphs, blocks, removed
  image_area_fraction, image_regions: [{box, area_fraction, body_glyphs_inside, text_covered}]
  signals: [...], ocr_risk: high|medium|low|none
  gutters: [{side, x_center, count, values, kind, step, median_pitch_pt,
             channel_crossing_rows, outliers_kept}]
  gutter_candidates_rejected: [{side, x_center, count, values, reason}]
  column_separators: [{x, x_range, rows_supporting, rows_in_span, y_span}]
  blocks: [{id, box, lines: [{text, box, h, words: [{t, box, chars: [first, last]}]}]}]
  removed: [{kind, text, box, chars: [first, last], reason}]
  `chars` are PDFium character indices on that page (0-based), the join key into
  chars.jsonl.

Selfcheck (selfcheck.py)
  pages.jsonl covers 1..N in order; recorded PDFium char counts match the file; every
  word and removed item has a valid char range and a non-inverted box inside its
  block; body words + removed words == words seen on the page (nothing lost or
  double-counted); gutter values are monotone and no gutter token survives in the body
  at the gutter x; zero-body pages carry the explicit marker and are not
  ocr_risk=none over an image; and every search hit's boxes lie inside one block.

Limits, stated plainly
  - Text present only as pixels (scans without a text layer, drawings, pasted
    equations, screenshots) is not extracted. Such pages/regions are flagged.
  - Where a scan carries an OCR text layer, output quality is that of the embedded
    OCR (dropped hyphens, wrong glyphs, misplaced boxes). Pages are flagged medium.
  - Reading order inside tables is column-major within the table region; a
    multi-line cell stays together but the table is not reconstructed as rows.
  - Tight table columns whose inter-column gap is under ~1 em are not separated.
  - A gutter lane that is only partially present in an OCR layer (missing numerals)
    can leave a stray numeral inside a merged line near the page margins.
  - Furniture classification needs recurrence: a header that appears on one or two
    pages stays in the body. A page number printed as "Page 214" is only removed
    when the masked key recurs.
  - Centre-aligned cover pages with stacked short lines are ordered by geometry,
    which can differ from the author's visual grouping.
  - Every threshold above was tuned on one document (the 373-page nonconfidential
    joint appendix in EcoFactor v. Google, CAFC 23-1101, ECF 15). No accuracy figure
    is claimed and no generalisation is claimed. Measured findings are reported on the accompanying public report and evaluation.json.


The source.zip bundle also carries portable-evaluation-final.zip with the evaluator's runnable controls and versioned evaluation artifacts. outputs.zip is the accompanying complete all-page output archive.
