Following through on my last post, Rebuilding the NLRB Stats Archive, where I ended by promising a test of open-weight models.
I focus the test on the most expensive part of my data pipeline, where Opus 5 reads the archive’s 4,420 table pages. I should’ve done a better job measuring usage, but it is hard when you are running multiple sessions for different projects. Roughly, I would say this used half of my weekly Claude Max allowance. That’s fine, but open-weight models are effectively free and I love free.
Model swapping
Okay. So everyone said swapping models is frictionless.
NOT TRUE! My theory is that this is a classic selection problem:
- Early adopters are the ones who observe the lowest frictions to adopting a thing
- Early adopters tell late adopters, “You gotta try this thing! It is so easy and fun.”
- Late adopters try to adopt and tell early adopters, “You are a liar. This thing is neither easy nor fun.”
- Early adopters explain that they either have spent 3 years working on related-things or just enjoy suffering in their free time
But also implementing open-weight models really isn’t too hard and is pretty interesting. I’ll leave it to other people (or your agent) to explain…
Basically, assuming you can even get these models to just run, you still need to address the practical question of how to get them to run well, and this is where I experienced great pain.
One day, I will understand all these settings—but not this day. Instead, I turned to Andrej Karpathy’s autoresearch skill. I then selected five open-weight models, and assigned each model a budget of 13.25 GPU hours. Then, each model was assigned an agent that read the model’s documentation and applied a simple greedy search to maximize accuracy.
This wasn’t a one-to-one port, since Karpathy designed autoresearch for training LLMs (not inference), and honestly I could’ve just used a /loop here as well. Like Karpathy, I’ve set up a pretty extensive Obsidian vault where my agents dump reports about how each round of iterations is going, allowing me to track progress and design decisions. In the future, I’ll probably read more about Monte Carlo tree search and other, more structured methods.
Throwing compute basically works
Here are the results of that process:
Figure 1. Share of cells each model recovered, all tables.
All models are measured relative to Claude Opus 5 (my baseline production model). I separately benchmarked Claude, so I know that it has near perfect accuracy in my setting (after a lot of wrangling), but you can imagine the above as upper bounds on the true error rate for these models.
In gray, you see the models’ accuracy after a single round of work. The open-weight models are pretty abysmal. I actually initially stopped after the first round, because I was so depressed by these results and did not want to waste more usage. Thankfully, I decided to push harder, and you can see in blue that after expending the full budget, the models saw massive improvements. For example, DeepSeek-OCR-2 went from about 4% of cells to 86%.
A natural concern is overfitting. In yellow, I report the accuracy of these models on a holdout sample of tables (again Claude Opus 5), and show that my training process works out-of-sample.
What drives these gains? It comes back to the friction I described earlier: integrating an open-weight model into a complex data pipeline is hard enough that even frontier models can’t do it in one shot. Here are two examples I had my agents trace:
-
Take GLM-OCR. It’s trained to respond to three fixed commands, one of them
Table Recognition:. I didn’t know that at first and sent it the same 800-word prompt as every other model. It copied out the example table from my prompt instead of reading the page. An agent reviewing my setup against the model’s documentation caught the mistake later. Once the prompt was cut down toTable Recognition:, GLM-OCR read 522 numbers on five test pages without a single misread, and it ended up one of the best models here. -
Take DeepSeek-OCR-2. It reads a page as a set of fixed-size tiles and decides how many to use from the page’s shape alone. Despite following the default methods, our approach assigned a wide table too few tiles to keep its columns apart and produced none of the 15 wide tables in my benchmark. Over several iterations, an agent found that padding its pages gets them cut into more tiles. Remaining issues were resolved by reading each page several ways and keeping the data which most of the reads agree on.
The main takeaway is that throwing more compute at this scale works.
Open-weight models are good enough for some tasks
These reports have two kinds of table. Wide ones have 14 or more columns and run over two or more pages, like this one:
1985, Table 6B, the first of its three pages.
Everything else I’ll call conventional:
1985, Table 11D.
Conventional tables are about 80% of the tables in the archive and 45% of the cells.
Figure 2. Cells right in the tables each model produced.
Figure 2 looks only at the tables a model actually produced, and counts how many of their cells it got right. For both wide and conventional tables, the models do exceptionally well conditional on being able to process the table.
This is great news! Based on these results, I should try to direct my tables first to a cheaper model, and then assign the harder tables to frontier models. Personally, I still think the current accuracy rates are too low, so I need to do more looping/autoresearching or try some other models. In addition, while it is natural to define “harder tables” as the ones which the model fails to read entirely, a better solution would be to use LLM-as-a-Judge.
What I learned, and what’s next
What I learned:
- Do not one-shot the implementation of open-weight models
- A simple agent-driven greedy search with a fixed time budget goes a long way
- Most of the gap to frontier models is their ability to read complex table formatting, not cell-level fidelity
The main question I have after this is, “More compute or more structured search?” In terms of more compute, my laptop is the main bottleneck since I cannot run models on it 24/7 and it does not have enough RAM for larger models. Obviously, there are cloud computing resources, but that undermines the whole point of saving money on these projects.
In terms of more structured search, I got exposed to a ton of methods through this process. I mentioned earlier that I could use a more structured search algorithm such as Monte Carlo tree search and EvalGen. I could also try to refine the autoresearch approaches being used by my agents through system prompts, or develop an LLM-judge using AI observability/eval platforms like Braintrust that assesses whether proposed revisions should be adopted or raised for human review. I could also play around with fine-tuning open-weight models.