TLDR: Good data is everywhere, but old data is hard to use. Here’s a rough attempt at some solutions.
As part of my dissertation, I studied the threat effects of hospital unionization following the 1974 Health Care Amendments. I had to do some incredibly intensive data work where I purchased archival copies of the American Hospital Association’s Annual Survey. You can read more about the Annual Survey here, but it was extremely painful to get this data into a format where I could use it.
I then linked it to NLRB union election records, which thankfully* has already been gathered and cleaned by other researchers. In fact, if your only focus is studying labor unions in the 21st century, there is a TON of easy-to-use data.
But I was studying the expansion of collective bargaining to hospital workers in 1973, so I routinely felt hampered by the lack of other data sources.
Of course, there is good data everywhere, but old data is hard to use. They exist in formats poorly adapted for computers to read (since computers did not like exist then). This is why a lot of smart people have developed really advanced techniques for converting coarse document data into machine-readable formats.
Since I have some free time, I wanted to see how far I could push existing agentic workflows in conjunction with these methods. That’s where NLRB Stats Archive comes into play.
I went on the NLRB website and downloaded their Annual Reports from 1936 to 2003. I had Claude use Docling to identify tables and figures, separated them, and ran them through Opus. Then, I produced this website to publicly document the outcome.
NLRB Stats Archive IS NOT: This is absolutely not a finished and perfectly accurate dataset of all the tables and figures contained in the Annual Reports. It honestly does a pretty good job, but if you click through enough you will find repeated errors.
NLRB Stats Archive IS: This is absolutely a public display of my active progress in trying to learn how to convert messy data into something which can be used by computers, humans, and agents. It’s also my way of learning how to code “stuff”, not just regressions.
As the name implies, this is not a project I want contained to just NLRB Annual Reports. In principal, there are many other sources of data that would be worth excavating scattered across the Internet or in the cardboard boxes of labor libraries across the country.
We’ll see how this evolves! Ping me if you have advice or want to support this project.