DnD-WebScraper Puts a Tiny Markup Language Inside Every Spreadsheet Cell
Two Python scripts turn a D&D wiki into Google Sheets rows. The clever part is the grammar the author invented so a character sheet app could read the data back.
- Every cell this scraper writes is a small re-parseable document, not flattened text.
- The parser produces that language by rewriting the HTML tree in place, swapping lists and links for markup strings before it reads the text.
- The two scripts hold opposite philosophies: the race scraper invents a grammar, the spell scraper slices by fixed character offsets.
- The format's real ceiling is the absence of escaping, which lets ordinary punctuation corrupt downstream parsing.
The Cell Is Not a Cell
Open the exported CSV and one cell looks like a mistake. It is not a sentence. It is a string stitched together from four different delimiters, with brackets, pipes, and semicolons doing specific jobs.
Spell List ||| name || school;;\nFireball || Evocation;\n{item:Bat guano};\n[Light;\nPrestidigitation]
Read that slowly and a structure appears. `Spell List` is the table's name, set off by a triple pipe. `name` and `school` are its columns, split by a double pipe. The double semicolon and newline close the header. Each body row ends with a single semicolon and a newline. Inside one row sits a typed link, `{item:Bat guano}`, and a nested list, `[Light;\nPrestidigitation]`, whose items are separated by a semicolon and a newline. Nothing here is accidental. The delimiters get finer as the structure gets deeper, which makes the raw string legible to a human long before a parser touches it.
The repository's README documents this as a format, with a specifier for links, a separator for list items, and a layered set of tokens for tables. The samples below come straight from that spec.
| Token | Meaning | Example |
|---|---|---|
| {type:name} | Typed link | {feat:Alert} |
| [item;\nitem] | List with item separator | [Light;\nPrestidigitation] |
| ||| | Table name separator | Spell List ||| ... |
| || | Column separator | name || school |
| ;;\n | Header to body separator | ...;;\nrow... |
| ;\n | Row or item separator | Fireball || Evocation;\n... |
| Name | Description;\n | Feature | Alert | ...;\n |
That is the whole idea, and it is the reason this otherwise small personal project is worth reading. The cells are not text. They are documents with a grammar, written by the scraper and designed to be re-parsed by the app on the other side. This is a serialization format whose storage medium happens to be a spreadsheet.
Why a Character Sheet Needed a Database
The author maintains a browser-based Character Sheet app. It needs reference data (races, spells, feats) that it can query at runtime without a backend. There is no server to host a database, and no budget for one.
Google Sheets fills that gap. It is queryable, editable by hand, and free. The scraper's job is to keep the sheet current, pulling from dnd5e.wikidot.com and writing rows the app can consume. That constraint explains everything downstream. If the database is a spreadsheet and the consumer is a character sheet, then the cell has to carry enough structure for the app to rebuild lists, tables, and links on its own.
The Grammar, Unfolded
A flat string is easy to read and hard to feel. The diagram below takes the cell from earlier and lets you take it apart. Hover a delimiter and its partner lights up, bracket-matcher style. Step through the four node types and watch one line of text become a tree with a link, a list, a table, and a feature.
The dispatch that decides what becomes a column and what becomes markup is a single `match` statement. Core mechanical fields (ability score, age, speed, size) get their own spreadsheet columns. Everything else collapses into a generic feature string that reuses the same custom syntax.
match featureName:
case "Ability Score Increase": data.ability_score = featureData
case "Age": data.age = featureData
...
case _: data.features[featureName] = f"{featureName} | {featureData};\n"
That split is the schema boundary. Filterable columns for the fields a character sheet sorts and looks up, one catch-all feature blob for everything else, still serialized in the same language so the consumer app parses it identically.
Parsing by Rewriting the Page
Producing that language required an unusual move. Instead of walking the HTML tree to extract text, the scraper rewrites the tree in place, replacing nested lists and links with markup strings before it ever reads the text.
while elmnt.ul: # has list
elmnt.ul.replace_with(list_formatter(elmnt.ul).strip('\n'))
while elmnt.a: # has link
elmnt.a.replace_with(link_formatter(elmnt.a))
The `while` loops, not `if`, matter. Each `replace_with` changes the tree, so the next match has to be re-queried. The effect is to turn a recursive tree-walking problem into a flat string-processing problem. Nested lists and links become annotated text before the scraper reads `elmnt.strings`.
One helper, `FilterStrings`, handles the mess BeautifulSoup leaves behind. Inline links come out as separate fragments, so the function re-joins them using a shape heuristic: a string that begins and ends with braces is treated as a link and merged into the previous fragment, unless that fragment ends with a closing bracket, which would mean a list or table just closed. The comment in the code spells out the problem: fragments like `["some text with ", "{link}", " in the sentence"]` become `["some text with {link} in the sentence"]`.
The trade-off is a side effect. Because the function mutates its input, it is not idempotent. Running it twice on the same soup would produce a different result the second time. For a one-shot scraper that never needed to run twice on the same tree, that was a cost worth paying.
The Second Script Disagrees With the First
The repository also contains a spell scraper, and it shares almost nothing with the race scraper. No grammar. No shared module. It parses by fixed character offsets, assuming the page's markup is exactly where the author last saw it.
additional_data["sources"] = target_elements[1].text[8:].strip()
additional_data["school"] = split[1].strip().capitalize()
additional_data["cast time"] = text[len("Casting Time:") + castIdx + 1: rangeIdx].strip()
It slices labels by their known length and positions elements by their known index. There is a layout branch for a second page shape, and the flag guarding it is named `broken_extra_block`, which tells you the author had already mapped the wiki's irregularities and chose to handle each one explicitly rather than generalize. The URL join is just as brittle: `link = url[:-7] + link` assumes the index URL ends in exactly seven characters, the length of `/spells`. Change the URL and the join breaks without a word.
| Races script | Spells script | |
|---|---|---|
| Output method | stdlib csv plus custom markup | pandas DataFrame |
| Parsing strategy | DOM rewrite and closure state machine | positional offsets and fixed slices |
| Failure behavior | empty entries discarded at commit | one bad spell can drop a whole table |
| Shared code | none | none |
Two philosophies, one repo, zero shared code. Read together, the scripts make the race scraper's choices look deliberate rather than accidental. When the author cared about a reusable format, he built one. When he only needed a page he had already mapped, he hard-coded the offsets and moved on. Brittleness traded for speed on a target that was not going to move.
What It Is Not Competing With
This project does not compete with Scrapy, Crawlee, or Playwright on crawling. It has no scheduler, no retry policy, no concurrency, no robots handling. A few hundred sequential requests is the whole engine. Where it differs is the output shape. General scrapers hand you Markdown, JSON, or a text blob. This one hands you cells designed to be read back.
| Project | Language | Crawl engine | Output format | Designed to be re-parsed? |
|---|---|---|---|---|
| Scrapy | Python | Yes | Items / JSON / CSV | No |
| Crawlee | JS and Python | Yes | No | |
| Playwright | TypeScript | No, browser only | Raw page / DOM | No |
| trafilatura | Python | No, extraction only | Text / metadata | No |
| Crawl4AI | Python | Yes | Markdown for LLMs | No |
| DnD-WebScraper | Python | No | Custom markup in CSV | Yes |
Only the last row answers yes to the decisive column. The others are built to collect information. This one is built to hand information forward in a shape the next program already knows how to read.
It is worth being honest about scale. The project has no stars, no forks, one contributor, and no issues or pull requests. It is public because the author's character sheet app depends on it, not because it is trying to attract anyone. The README is the spec, and the spec lives there because there is no library to import it from.
The Cost of Inventing a Language
A custom format is a promise you have to keep everywhere. The delimiters here are ordinary punctuation, and nothing in the visible code escapes them. A feature description containing a single pipe, or a list item containing a closing bracket, can corrupt the parse on the far side.
| Decision | Benefit | Cost |
|---|---|---|
| Custom markup over JSON | Human-readable, editable in a cell, nestable | No escaping; punctuation in prose can break parsing |
| Tree mutation in place | Turns tree-walking into string work | Side effects; not idempotent |
| Offset parsing in spells | Precise on a known layout | Breaks silently when the wiki changes |
| Sheets as the database | Free, queryable, hand-editable | No schema enforcement, no transactions |
| Two standalone scripts | Each runs on its own | Duplicated logic; spec and implementation drift |
There are no tests, no fixtures, and no dependency manifest. The parsing logic is only verifiable by running it against the live wiki, which means a change on dnd5e.wikidot.com would surface as a wrong cell rather than a failing build. That is the price of a personal dependency that nobody else has to install.
And yet the central lesson holds. When the consumer on the other side is a spreadsheet cell, the format is the architecture. The scraper is not really a scraper. It is the compiler for a tiny language, and the spreadsheet is where the output runs.





