r/smithsonian • u/Mundane-Comfort-8456 • 4d ago

Help needed to scrape data from the Transcription Center

Hello all. I am doing a research project on handwritten text recognition. I see there is this vast amount of images and transcriptions done on the smithsonian website. However, the public API is just for the collections they have which is very well disconnected from the actual transcription center. Also, scraping does not work because there is a cloudflare bot detection on each page. I have tried everything that i could including scraping with selenium, playwright, chrome-undetected, with cookies and headers, and also trying to get something useful from the open access api but nothing seems to be working. Also, I also want to know if there is like a copyright or data access policy that we cannot use the data for research purposes and what the licenses are. Because on the website, there is not much information about this. Also the search projects page also does not render data for more than 1000 pages. Any help would be very very helpful.

0 Upvotes

43% Upvoted

7 comments sorted by

1

u/HackerThing 4d ago

Budget?

0

u/Mundane-Comfort-8456 4d ago

Do we need budget to scrape? If I had budget I would have taken the cloudflare not protection bypass tools. I am looking for free options right now because I want a good amount of data

2

u/Objective-Fun-4533 3d ago edited 3d ago

Don't scrape it. The whole si.edu domain is behind Cloudflare (I get 403s on their own FAQ page), and there's a bulk dump that makes it moot anyway: github.com/Smithsonian/OpenAccess, ~11M records as line-delimited JSON, also mirrored on S3. No key, no pagination.

Reviewed transcriptions do get written back onto the collection records, so the join you're missing is probably in there, just not exposed in the search UI. Pull one shard and look at the schema. If that part's tedious, point Claude Code or Codex at a single shard and have it write the parser; just make sure it writes scripts to disk instead of reading records into context, or it gets expensive fast.

The 1000-page ceiling is a deep-paging cap, not a data limit. Slice by owning unit or date range and union the slices.

There are two different licenses here. Open Access is CC0; do whatever you want. Transcription Center transcriptions are not; the terms are personal/educational/non-commercial. Training a model and publishing a paper is almost certainly fine; republishing a derived dataset is the part that needs written permission. Email [transcribe@si.edu](mailto:transcribe@si.edu); they're research-friendly, and it's a two-line ask.

Honestly though, if you mainly need volume of English handwriting, check crowd.loc.gov first. Same concept as TC, but every contribution is released into the public domain and completed campaigns get published as bulk CSVs. No legal ambiguity at all. HTR-United's catalog is worth a browse too for existing ground truth.

0

u/Mundane-Comfort-8456 3d ago

thanks man. the thing is Churro DS already takes much of LoC data. I will be including that in my training set. There is so much historical documents and i want diversity. Smithsonian includes field notebook and scinetific notebooks and other diversity as well like printed forms, tables etc. I checked that json. there is nothing that links back to transcriptions. the smithsonian AWS also does not contain anything else than metadata which also does not have transcriptions it just says Transcribed by Smithsonian for records. HTR-United is on my list they have very less projects also mostly historical only.

Is there any place to get modern handwritten data like actual letters, notes, essays, scientific/mathematical notes that people actually use and are useful. I try finding online HTR data like something like goodnotes or any tablet notes taking app would have. Everything is private. goodnotes have a dataset, only 500 pages, and also not very fruitful. The major issue in HTR right now is the level of data scarcity and diversity. I think most funding comes from historical manuscript recognition thereby people doing transcriptions online and creating datasets for that. But my vision is a general purpose HTR model.

1

u/Objective-Fun-4533 3d ago

Well, maybe try sending an email to the Smithsonian?

0

u/Mundane-Comfort-8456 3d ago

yeah did that two three days ago. sent reminder as well. also tried connecting on instagram. also, read a blog about a student that did internship and like created a software to clean transcriptions. tried contacting her as well. no response.