The NLRB Stats Archive publishes the statistical tables from the National Labor Relations Board’s annual reports, fiscal years 1936 to 2009, as machine-readable data. The source is 74 scanned PDFs, and most of the work is getting from a scanned page to a table whose cells are right. The first version of the site went up in June (launch post). In September I archived everything except the PDFs and rebuilt the pipeline from scratch. This post explains why, and lays out the architecture of the version that replaced it.
Why I rebuilt it
The original reason I even bothered to make this is outlined here. Honestly, since then the motivation has transformed into about finding a way to learn more about agentic engineering. I’ve alway professed to be a “data dog”, and agentic workflows have enabled me to do more of that.
Early September, I decided I wanted to update the database by swapping out some models and streamlining the data pipeline… and, then, I realized that this would be impossible. The first version was built through just repeatedly prompting. The site worked, but I couldn’t describe the architecture.
To be honest, I’d care a lot less about understanding, if I only cared about having the database work. Unfortunately, I really wanted to learn more about things like:
- How to architect a system with different LLMs stepping in and out at various stage,
- Benchmarking different VLMs at table extraction
- Producing reproducible workflows for future projects
Even then, I don’t really need to understand the code, but I do need to understand the system architecture. Beyond just producing overwhelming amounts of code, AI coding agents can produce incomprehensible codebases for complex projects.
(This situation is very reminiscent of the “Software Crisis” where increasing computational complexity meant that long linear programming scripts were no longer feasible, leading to the development of many of the norms of “good coding” which we now take for granted. Same applies here, but at a higher-level of abstraction. )
For this rebuild, I wanted a process with:
- humans-in-the-loop
- modularity of tasks for LLMs into stages, and
- deterministic code to manage inputs into and process the output from LLMs
None of this is really technical, but neither is writing good code.
Now that I’ve implemented this procedure, I feel more confident in my ability feature test. I’m now benchmarking open-weight models at the table-reading stage, where the PDFs are already preprocessed and a vision model reads each cropped table into rows and columns. That stage is the most expensive part of the pipeline, so testing cheaper open-weight models on that one task is the biggest cost saving available to me. That task is also much harder than skimming the PDFs to find where the tables and figures are. I currently use Opus for that skimming too, and I suspect it’s overkill.
Below is the rough architecture, but keep an eye out for the benchmarking document.
Find the tables
Which pages hold a table or figure, and which pages belong together.
%%{init: {"flowchart": {"wrappingWidth": 300, "nodeSpacing": 30, "rankSpacing": 40}}}%% flowchart TD PDF["the scanned report"] S0["<b>0 · ingest</b><br/>each page as an image and as text;<br/>sideways pages turned upright"] S1["<b>1 · inventory</b><br/>the report's own list of tables"] D2["<b>2 · classify</b> · docling<br/>which pages hold a table or figure"] C2["<b>2 · classify</b> · Claude<br/>which pages hold a table or figure"] D3["<b>3 · headings</b> · docling<br/>the caption, from the page text"] C3["<b>3 · headings</b> · Claude<br/>the caption, and whether the page<br/>starts a table or continues one"] D4["<b>4 · group</b><br/>docling's pages into items"] C4["<b>4 · group</b><br/>Claude's pages into items"] M4["<b>4 · merge</b><br/>agreements pass straight through;<br/>disagreements go to a queue"] ARB["<b>arbiter</b> · Claude<br/>a suggested verdict per dispute,<br/>the readers blinded as A and B"] H4["human: type the verdicts"] ITEMS["<b>the items</b><br/>one per printed table or figure,<br/>with its pages"] N1["on to: cut and read"] PDF --> S0 S0 --> S1 S0 --> D2 --> D3 --> D4 --> M4 S0 --> C2 --> C3 --> C4 --> M4 S1 -.->|a check| M4 M4 -->|disputes| ARB --> H4 H4 -.->|run the merge again| M4 M4 --> ITEMS --> N1 classDef code fill:#dde8f7,stroke:#4a6fa5,color:#111111 classDef claude fill:#e9e0f5,stroke:#7a5ba6,color:#111111 classDef file fill:#f6f6f6,stroke:#999999,color:#222222 classDef human fill:#fbeccd,stroke:#c48a12,color:#111111 classDef hand fill:#fafafa,stroke:#4a6fa5,color:#333333,stroke-dasharray:4 3 class PDF,ITEMS file class S0,S1,D2,D3,D4,C4,M4 code class C2,C3,ARB claude class H4 human class N1 hand
The reports aren’t uniform. Seventy-four years of government printing means typeset pages from the 1930s, tables scanned sideways, faded scans and born-digital PDFs from the 2000s. One table can run across thirty pages, one page can hold three tables, and many tables carry no number at all.
Ingest turns every page into two things: a small image and its text. On some scans the embedded text is pure noise, usually because the table was scanned sideways. Those pages are re-read with OCR at each of four rotations, and their images are saved upright. After this, no later stage needs to know that a page was ever sideways.
Inventory reads the report’s own printed list of tables, where there is one. No model ever sees it. It serves only as a check: if the report lists a Table 14 and the pipeline never finds one, that shows up.
Next, two readers go through the pages independently. One is docling, an open-source document-layout model that runs locally. The other is Claude, which looks at the page images in batches. Each reader marks which pages hold a table or figure, then reads the heading on each flagged page. Claude also reports what sits at the top of each table page: the start of a new table, rows continuing from the page before, or neither. That matters, because many continuation pages print no heading at all.
From those headings, plain code groups the pages into items. A heading opens a table, a “Continued” marker or an unheaded table page extends it, and anything else closes it. This happens once per reader, and then the two groupings are merged. Where both readers put the same pages into the same item, it passes straight through. Everything else, whether the page ranges differ or only one reader found the item, goes into a queue for me.
Before I read the queue, Claude reads it as an arbiter. For each dispute it sees the page images and the two groupings, labeled only “Reader A” and “Reader B,” and suggests a verdict with a confidence and a reason. The disputes are then ranked so the ones that need my attention come first. The arbiter only suggests: I type the verdicts, and the merge runs again with them. This is the review step that changes the end product the most. What comes out is the final list of items, one per printed table or figure, each with its pages.
Cut and read
Crop each table from its pages, then read it into rows and columns.
%%{init: {"flowchart": {"wrappingWidth": 300, "nodeSpacing": 30, "rankSpacing": 40}}}%% flowchart TD IN["the items, from Stage 4"] S5["<b>5 · cut</b><br/>a layout model finds each table's box;<br/>the crop takes in caption and footnotes"] CROPS["the crops, one per item page"] SWP["<b>sweep</b><br/>flags crops that look<br/>sideways or clipped"] H5["human: fix the bad crops<br/>with a new box or a turn"] S6["<b>6 · extract</b> · Claude<br/>each crop read into rows and columns:<br/>a fresh session, a frozen prompt,<br/>a fixed answer shape"] OTH["<b>other readers</b><br/>docling and open-weight models,<br/>on a held-out sample"] CLEAN["<b>clean</b><br/>the same rules for every reader;<br/>no digit or letter is ever changed"] ST["<b>stitch</b><br/>an item's pages joined<br/>into the one table they print"] SITE["the site's data:<br/>whole tables and page images"] N2["on to: check; name and publish"] IN --> S5 --> CROPS CROPS --> SWP --> H5 H5 -.->|run the cut again| S5 CROPS --> S6 --> CLEAN CROPS --> OTH --> CLEAN CLEAN --> ST ST -->|Claude's tables| SITE --> N2 classDef code fill:#dde8f7,stroke:#4a6fa5,color:#111111 classDef claude fill:#e9e0f5,stroke:#7a5ba6,color:#111111 classDef file fill:#f6f6f6,stroke:#999999,color:#222222 classDef human fill:#fbeccd,stroke:#c48a12,color:#111111 classDef hand fill:#fafafa,stroke:#4a6fa5,color:#333333,stroke-dasharray:4 3 class CROPS,SITE file class S5,SWP,OTH,CLEAN,ST code class S6 claude class H5 human class IN,N2 hand
Cut crops each item out of its pages. A layout model finds the table’s box on each page, and the boxes are matched to items from top to bottom. Each box is extended to take in the caption above and the footnotes below, and the crop is rendered from the original PDF at the scan’s full resolution. From this point on, the crop is the unit of work, and nothing downstream reads the PDF.
A sweep then runs OCR over every crop and flags two kinds of failure. The first is a crop that reads better when rotated, meaning it’s sideways. The second is a table crop whose left edge is full of numbers rather than row labels, meaning the box cut off the row labels. The flags, plus a contact sheet showing every crop, tell me which ones to open. I write each fix, a new box or a rotation, into a corrections file and run the stage again. Crops are cheap to fix here and expensive to fix later, because a re-cut crop has to be read again.
Extract is where models are swapped in and out. Claude gets each crop in a fresh session with no memory of other pages, the same frozen prompt every time, and a required answer shape: heading rows, body rows, footnotes, notes on doubtful cells and a confidence. An answer that isn’t a clean rectangular grid is rejected and requested once more. It is never padded into shape, because padding shifts values between columns.
Every reader’s output then goes through the same cleaning code: leader dots are dropped, stray spaces inside numbers are closed up, and side-by-side panels are restacked into one. None of these rules ever changes a digit or a letter. Finally, an item’s pages are stitched into the one table they print. Repeated headings are dropped, and a table whose pages disagree on width is held back rather than forced into line. The stitched tables and page images are what the website is built from.
The same stage runs the other readers on a held-out sample: docling’s table model, a second Claude model and open-weight vision models running locally. They all get identical crops and identical cleaning, so the reading is the only thing that differs between them.
Check
Correct a sample of tables by hand, and score each reader against them.
%%{init: {"flowchart": {"wrappingWidth": 300, "nodeSpacing": 30, "rankSpacing": 40}}}%% flowchart TD IN["from Stages 5 and 6:<br/>the crops and every reader's tables"] EXP["<b>export</b><br/>a checking kit for a sample of tables;<br/>each table's own sums show where to look"] KIT["the kit: Claude's table to correct,<br/>its page images, every reader's version"] H["human: correct the cells,<br/>mark each table checked"] S7["<b>7 · score</b><br/>tables produced, and cells right,<br/>for every reader"] OUT["accuracy by reader and year"] N3["on to the site: a checked table<br/>is published as corrected"] IN --> EXP --> KIT --> H --> S7 IN -->|every reader's tables| S7 S7 --> OUT H --> N3 classDef code fill:#dde8f7,stroke:#4a6fa5,color:#111111 classDef file fill:#f6f6f6,stroke:#999999,color:#222222 classDef human fill:#fbeccd,stroke:#c48a12,color:#111111 classDef hand fill:#fafafa,stroke:#4a6fa5,color:#333333,stroke-dasharray:4 3 class KIT,OUT file class EXP,S7 code class H human class IN,N3 hand
Accuracy is measured against tables I corrected by hand. An export step builds a checking kit for a sample of tables. For each table, the kit holds Claude’s reading, its page images and every other reader’s version. It also lists where to look first, using the table’s own arithmetic. The exporter works out which columns and rows are sums of others, then flags the cells that break a relationship the rest of the table obeys. No model is involved in that check.
I correct the cells against the page images and mark each table as checked. Score then compares every reader to the checked tables and keeps two questions apart: did the reader produce a whole table, and, if it did, how many of its cells are right? Producing nothing and reading wrongly are different failures, and averaging them together would hide both. Once a table has been checked, the site publishes the corrected version in place of the model’s reading.
Name and publish
Give every table and figure a title, then build the site.
%%{init: {"flowchart": {"wrappingWidth": 300, "nodeSpacing": 30, "rankSpacing": 40}}}%% flowchart TD IN["the tables, crops and headings<br/>from earlier stages"] CAP["<b>8 · captions</b><br/>the printed caption is the title<br/>once a second reading confirms it"] RR["<b>re-read</b> · Claude<br/>reads the caption blind<br/>where nothing confirms it"] GRP["<b>8 · grouping</b> · Claude<br/>names items with no caption;<br/>places every item in a series"] H8["human: settle disputed captions,<br/>vet the generated titles"] T["the titles and series"] S9["<b>9 · site</b><br/>a page per table or figure,<br/>year listings, search"] PRE["the website, as a local preview"] DEP["deploy: still being rebuilt"] IN --> CAP CAP -->|confirmed| T CAP -->|unconfirmed| RR RR -->|readings agree| T RR -->|readings differ| H8 IN --> GRP --> H8 --> T T --> S9 IN --> S9 S9 --> PRE -.-> DEP classDef code fill:#dde8f7,stroke:#4a6fa5,color:#111111 classDef claude fill:#e9e0f5,stroke:#7a5ba6,color:#111111 classDef file fill:#f6f6f6,stroke:#999999,color:#222222 classDef human fill:#fbeccd,stroke:#c48a12,color:#111111 classDef hand fill:#fafafa,stroke:#4a6fa5,color:#333333,stroke-dasharray:4 3 class T,PRE file class CAP,S9 code class RR,GRP claude class H8 human class IN,DEP hand
Titles works in two passes. Most tables print a caption, and the caption becomes the title once a second, independent reading confirms it. That second reading can come from the other reader’s heading or from the page’s own text. Where neither confirms the caption, Claude re-reads it blind from the crop, without seeing what the earlier stage read. If the two readings agree, the caption becomes the title. If they differ, the item comes to me.
In the second pass, Claude names the items that print no caption at all, in a consistent house style. It also places every item in a series, meaning the same table recurring across years. I settle the disputed captions and vet the generated titles.
Site builds every page of the website from the stitched tables, the crops and the titles. Each table gets a page with its grid, a CSV download, its footnotes and the source images; each figure gets a page too, and there are year listings and a search. The build refuses to run if a listing disagrees with what the earlier stages produced. Deployment is a separate step that I’m still rebuilding for this version.
What it produced
- 74 reports and about 3,650 items, of which about 2,800 are tables.
- Claude read all 4,420 table pages in a single run of about two days. At API prices that would have cost roughly $1,500; in practice it was drawn from subscription quota rather than billed.
- On 40 held-out tables (112 pages from four years never used for tuning), Claude’s reading matched the hand-checked tables on 25,191 of 25,194 cells, and 38 of the 40 tables were exactly right. One caveat: I corrected those tables starting from Claude’s reading rather than typing them blind, so a plausible-looking error could have survived the check.
- The arithmetic check doesn’t share that weakness. Across the whole corpus, 94% of the whole-number cells sit inside a row or column sum that works out, and 0.18% break one. That 0.18% includes misprints in the reports themselves, so it’s a ceiling on misreads rather than a count of them.
- 98% of table items stitch cleanly across their pages.
- My own review time, across all the stages, came to about one full working day.
What’s next: a fair test of open-weight models
The first comparison at the extract stage ran six open-weight vision models locally, alongside docling’s table model. The best of them, GLM-OCR, got 96.5% of cells right, but only on the 25 of 40 test tables it managed to complete. The others fell short mostly on coverage, for three main reasons: output limits on wide tables, a single failed page sinking a multi-page table, and OCR-tuned models that returned the prompt’s worked example instead of reading the page. Claude Sonnet produced all 40 tables at 98.3% of cells, but its errors were the dangerous kind: well-formed numbers that are simply wrong.
That comparison wasn’t on equal footing. The Claude readers see each image shrunk to 2,000 pixels on its long edge, while the local models read the crop at full size, and the local models also got a more forgiving parser. A fair re-test would equalize those conditions. It’s exactly the experiment this structure was built for: because every reader gets the same crops, swapping the reader changes one thing and nothing else.