DnD-WebScraper Puts a Tiny Markup Language Inside Every Spreadsheet Cell

Two Python scripts turn a D&D wiki into Google Sheets rows. The clever part is the grammar the author invented so a character sheet app could read the data back.

8 min read • View on GitHub • More from PantheraDigital

A wide editorial illustration of a spreadsheet grid drawn in perspective, with most cells flat and empty and one foreground cell pulled open like a drawer. Nested inside the opened cell are smaller structured objects: a bracketed list, a small ruled table, and a labeled link token shaped like a tag on a string. It illustrates the article's thesis that each spreadsheet cell contains a re-parseable document rather than plain text.
One cell, opened. The contents are not prose that flattened. They are a small grammar built to be read back by another program.
Key Takeaways

The Cell Is Not a Cell

Open the exported CSV and one cell looks like a mistake. It is not a sentence. It is a string stitched together from four different delimiters, with brackets, pipes, and semicolons doing specific jobs.

Spell List ||| name || school;;\nFireball || Evocation;\n{item:Bat guano};\n[Light;\nPrestidigitation]

Read that slowly and a structure appears. `Spell List` is the table's name, set off by a triple pipe. `name` and `school` are its columns, split by a double pipe. The double semicolon and newline close the header. Each body row ends with a single semicolon and a newline. Inside one row sits a typed link, `{item:Bat guano}`, and a nested list, `[Light;\nPrestidigitation]`, whose items are separated by a semicolon and a newline. Nothing here is accidental. The delimiters get finer as the structure gets deeper, which makes the raw string legible to a human long before a parser touches it.

The repository's README documents this as a format, with a specifier for links, a separator for list items, and a layered set of tokens for tables. The samples below come straight from that spec.

TokenMeaningExample
{type:name}Typed link{feat:Alert}
[item;\nitem]List with item separator[Light;\nPrestidigitation]
|||Table name separatorSpell List ||| ...
||Column separatorname || school
;;\nHeader to body separator...;;\nrow...
;\nRow or item separatorFireball || Evocation;\n...
Name | Description;\nFeatureAlert | ...;\n

That is the whole idea, and it is the reason this otherwise small personal project is worth reading. The cells are not text. They are documents with a grammar, written by the scraper and designed to be re-parsed by the app on the other side. This is a serialization format whose storage medium happens to be a spreadsheet.

Why a Character Sheet Needed a Database

The author maintains a browser-based Character Sheet app. It needs reference data (races, spells, feats) that it can query at runtime without a backend. There is no server to host a database, and no budget for one.

Google Sheets fills that gap. It is queryable, editable by hand, and free. The scraper's job is to keep the sheet current, pulling from dnd5e.wikidot.com and writing rows the app can consume. That constraint explains everything downstream. If the database is a spreadsheet and the consumer is a character sheet, then the cell has to carry enough structure for the app to rebuild lists, tables, and links on its own.

The pipeline, end to end. The serialization step in the middle is where the project earns its keep.

The Grammar, Unfolded

A flat string is easy to read and hard to feel. The diagram below takes the cell from earlier and lets you take it apart. Hover a delimiter and its partner lights up, bracket-matcher style. Step through the four node types and watch one line of text become a tree with a link, a list, a table, and a feature.

The dispatch that decides what becomes a column and what becomes markup is a single `match` statement. Core mechanical fields (ability score, age, speed, size) get their own spreadsheet columns. Everything else collapses into a generic feature string that reuses the same custom syntax.

match featureName:
    case "Ability Score Increase": data.ability_score = featureData
    case "Age": data.age = featureData
    ...
    case _: data.features[featureName] = f"{featureName} | {featureData};\n"

That split is the schema boundary. Filterable columns for the fields a character sheet sorts and looks up, one catch-all feature blob for everything else, still serialized in the same language so the consumer app parses it identically.

Parsing by Rewriting the Page

Producing that language required an unusual move. Instead of walking the HTML tree to extract text, the scraper rewrites the tree in place, replacing nested lists and links with markup strings before it ever reads the text.

A close-up editorial illustration of a single tree branch representing an HTML DOM node, clipped cleanly at the stem, with a flat written tag grafted into the cut and a small pair of shears resting against it. Beside it, the same branch remains whole and unclipped. It shows the tree-mutation parsing trick, where structure is removed and a markup string is grafted in its place before the text is read.
Structure out, markup in, in place. The node is cut and a written tag is grafted onto the stem before anyone reads what the branch says.
while elmnt.ul:  # has list
    elmnt.ul.replace_with(list_formatter(elmnt.ul).strip('\n'))

while elmnt.a:   # has link
    elmnt.a.replace_with(link_formatter(elmnt.a))

The `while` loops, not `if`, matter. Each `replace_with` changes the tree, so the next match has to be re-queried. The effect is to turn a recursive tree-walking problem into a flat string-processing problem. Nested lists and links become annotated text before the scraper reads `elmnt.strings`.

One helper, `FilterStrings`, handles the mess BeautifulSoup leaves behind. Inline links come out as separate fragments, so the function re-joins them using a shape heuristic: a string that begins and ends with braces is treated as a link and merged into the previous fragment, unless that fragment ends with a closing bracket, which would mean a list or table just closed. The comment in the code spells out the problem: fragments like `["some text with ", "{link}", " in the sentence"]` become `["some text with {link} in the sentence"]`.

The trade-off is a side effect. Because the function mutates its input, it is not idempotent. Running it twice on the same soup would produce a different result the second time. For a one-shot scraper that never needed to run twice on the same tree, that was a cost worth paying.

The Second Script Disagrees With the First

The repository also contains a spell scraper, and it shares almost nothing with the race scraper. No grammar. No shared module. It parses by fixed character offsets, assuming the page's markup is exactly where the author last saw it.

additional_data["sources"] = target_elements[1].text[8:].strip()
additional_data["school"] = split[1].strip().capitalize()
additional_data["cast time"] = text[len("Casting Time:") + castIdx + 1: rangeIdx].strip()

It slices labels by their known length and positions elements by their known index. There is a layout branch for a second page shape, and the flag guarding it is named `broken_extra_block`, which tells you the author had already mapped the wiki's irregularities and chose to handle each one explicitly rather than generalize. The URL join is just as brittle: `link = url[:-7] + link` assumes the index URL ends in exactly seven characters, the length of `/spells`. Change the URL and the join breaks without a word.

Races scriptSpells script
Output methodstdlib csv plus custom markuppandas DataFrame
Parsing strategyDOM rewrite and closure state machinepositional offsets and fixed slices
Failure behaviorempty entries discarded at commitone bad spell can drop a whole table
Shared codenonenone

Two philosophies, one repo, zero shared code. Read together, the scripts make the race scraper's choices look deliberate rather than accidental. When the author cared about a reusable format, he built one. When he only needed a page he had already mapped, he hard-coded the offsets and moved on. Brittleness traded for speed on a target that was not going to move.

What It Is Not Competing With

This project does not compete with Scrapy, Crawlee, or Playwright on crawling. It has no scheduler, no retry policy, no concurrency, no robots handling. A few hundred sequential requests is the whole engine. Where it differs is the output shape. General scrapers hand you Markdown, JSON, or a text blob. This one hands you cells designed to be read back.

ProjectLanguageCrawl engineOutput formatDesigned to be re-parsed?
ScrapyPythonYesItems / JSON / CSVNo
CrawleeJS and PythonYesNo
PlaywrightTypeScriptNo, browser onlyRaw page / DOMNo
trafilaturaPythonNo, extraction onlyText / metadataNo
Crawl4AIPythonYesMarkdown for LLMsNo
DnD-WebScraperPythonNoCustom markup in CSVYes

Only the last row answers yes to the decisive column. The others are built to collect information. This one is built to hand information forward in a shape the next program already knows how to read.

It is worth being honest about scale. The project has no stars, no forks, one contributor, and no issues or pull requests. It is public because the author's character sheet app depends on it, not because it is trying to attract anyone. The README is the spec, and the spec lives there because there is no library to import it from.

The Cost of Inventing a Language

A custom format is a promise you have to keep everywhere. The delimiters here are ordinary punctuation, and nothing in the visible code escapes them. A feature description containing a single pipe, or a list item containing a closing bracket, can corrupt the parse on the far side.

DecisionBenefitCost
Custom markup over JSONHuman-readable, editable in a cell, nestableNo escaping; punctuation in prose can break parsing
Tree mutation in placeTurns tree-walking into string workSide effects; not idempotent
Offset parsing in spellsPrecise on a known layoutBreaks silently when the wiki changes
Sheets as the databaseFree, queryable, hand-editableNo schema enforcement, no transactions
Two standalone scriptsEach runs on its ownDuplicated logic; spec and implementation drift

There are no tests, no fixtures, and no dependency manifest. The parsing logic is only verifiable by running it against the live wiki, which means a change on dnd5e.wikidot.com would surface as a wrong cell rather than a failing build. That is the price of a personal dependency that nobody else has to install.

And yet the central lesson holds. When the consumer on the other side is a spreadsheet cell, the format is the architecture. The scraper is not really a scraper. It is the compiler for a tiny language, and the spreadsheet is where the output runs.