The open Library · 10,000 documents · CC BY 4.0
Ten thousand full pages, cleaned and readable.
A public sample of the corpus: complete documents, not snippets, each with its source, score and counts. Read them here or download all ten thousand as one file.
10,000documents
—words
8shelves
Every document here is one line of all.jsonl.gz; the reader shows an excerpt and loads the full text on request.
For agents and tools
Machine-readable by design.
Every document is also a Markdown file with front-matter, and the whole Library is one JSONL download. No keys, no rate limits beyond the CDN's.
One file, all of it
Ten thousand complete documents, one JSON object per line, gzip-compressed.
curl -O https://dirbear.com/library/all.jsonl.gz
zcat all.jsonl.gz | head -1 | jq '{title, domain, bear_rank, shelf, tokens}'llms.txt
A plain-text map of the site, the shelves and the record schema for language models.
# Dir Bear
> The web, distilled for AI.
## Library (open, CC BY 4.0)
- /library/all.jsonl.gz: all 10,000 documents
- /library/{slug}.md: one document
## Shelves
science-engineering · software-docs · health ·
finance · education · home-craft ·
history-culture · civic-lawLoad it in Python
Straight into a dataframe, no SDK needed.
import pandas as pd
df = pd.read_json("https://dirbear.com/library/all.jsonl.gz",
lines=True, compression="gzip")
df.groupby("shelf").tokens.sum()