Skip to content

Extractors that fail loudly

The most common way a scraper fails is not a crash. The site changes its markup, and the scraper keeps running and returns nulls, or the wrong column, for weeks before anyone notices. An extractor is built the other way round: it is learnt once from a few pages, kept in a small file, replayed for nothing, and every replay checks the page against what was learnt. A page that drifted is a failed run, never a quiet one. When it fails, heal says what moved and where.

No model is involved at any step. The same pages always give the same extractor, and the same page always gives the same verdict.

Learn, replay, heal

# Learn from two or three pages built from one template.
sluicer compile https://shop.example/c/brakes?page=1 https://shop.example/c/brakes?page=2 -o brakes.json

# Replay it on any page of that template: rows as JSON, exit 3 if the page drifted.
sluicer run brakes.json https://shop.example/c/brakes?page=7

# After a redesign: see what moved, and write the healed extractor.
sluicer heal brakes.json https://shop.example/c/brakes?page=1 -o brakes.json

From Python:

from sluicer.extractor import compile_extractor, run_extractor, heal

extractor = compile_extractor([(html_1, url_1), (html_2, url_2)])
open("brakes.json", "w").write(extractor.to_json())

run = run_extractor(extractor, html, url)
if not run.ok:
    for check in run.checks:
        if not check.ok:
            print(check.name, check.expected, "->", check.got)
rows = run.rows  # a list of {field name: value}

healed, changes = heal(extractor, [(new_html, new_url)])

An agent gets the same three steps as MCP tools: compile_extractor, run_extractor and heal_extractor. Their answers carry ok, false for a page that drifted (with failed, the checks it broke) and for a heal that lost data (with lost), exactly where the command line exits 3.

What an extractor learns

  • What the pages declare. Every summary question all the pages answered -- title, price, sku, published date -- and every declared record type they all carried. For a structured answer (price, currency, dates, sku, availability) learnt from two pages or more, also the shape of the answer: 41.90 is NP, £51.77 is NPS.
  • The listing the pages repeat, unless a page declares its own subject -- a product page, an article -- or always with --listing: where it sits (html>body>div.page>ol.row), what one row looks like (li.product), how many rows each page had, and for every field its share of empty rows, the one shape its values shared if they did, and a few sample values.

The file is plain JSON, meant to be read and, if you need to, edited.

What a run checks

check fails when
listing the listing is no longer where it was, or two places now match where one did -- a sponsored strip of the same kind inserted before it
rows there are no rows, or on a listing of five members or more, more of them are empty shells than the learnt pages had, plus 20% -- skeletons waiting for a script
field a field every learnt row had is missing from more than 20% of rows, or a field most learnt rows had is missing from every row
shape fewer than half of a field's values keep the characters it was learnt with -- a price slot that now says "Add to basket" -- or a structured summary answer changed shape; 42 still fits a price learnt as 41.90
values on a page of five rows or more, a field that held different values in every row now says the same thing in all of them: a page of placeholders, "Loading"
summary a summary question every learnt page answered goes unanswered
type a declared record type every learnt page carried is gone
extractor the extractor checks nothing at all, so a pass would mean nothing

A row that gains a class -- on-sale -- is still a row, a short page of the same template -- the last of a pagination -- passes, and the second or third tag of a card is a count, not a column, so pages with fewer tags pass too.

sluicer run prints every page's rows and failed checks as JSON and exits 3 when any page failed any check. A run that broke its contract never exits 0.

What healing does

heal learns the new pages from scratch, then matches each old field to its new place:

  • The new field that holds most of the sample values the old one held. Of two that hold as many, the one more rows carry. Two columns that swapped are two moves, not two fields kept in place.
  • Else its own place, when that is still there and holds values of the shape the field was learnt with: a listing's items change between two visits.
  • A numbered slot -- the third tag, the second author -- whose own place is still there never moves to another slot of its group, because one tag turning up in another slot moved nothing.
  • Links are compared by path and parameters. A link that gained a tracking parameter, ?ref_=list_1, is the same link; ?id=2 is another item than ?id=1.

A field found in none of these ways is reported as vanished, even if a new field has the same shape, because a guess would put the wrong column under the old name. A field that moved keeps its old name, so rows read with the healed extractor have the columns downstream code expects.

Every field kept or moved carries its evidence: how many of the values it was learnt with were found in the new place, of how many, and how many the next best place held. It is reported, never used to decide. A move that rests on five values of five, with nothing else close, needs no second look; a move that rests on two of five, with a runner-up at one, is a reason for a person to look before the healed extractor is trusted. heal never heals itself above some confidence, because a wrong move is exactly the silent failure an extractor exists to prevent.

When anything was lost -- a field, a summary answer, a declared type, the listing itself -- sluicer heal exits 3 and does not write the healed extractor unless given --force: the old one keeps failing, which is the honest state until a person looks.

On the shop fixture in the test suite, a redesign that renamed every class and wrapped the listing in a new element moves all four fields to their new places:

container: html>body>div.page>ol.row -> html>body>main.content>div.page>section.grid
member: li.product -> div.card
moved: a.title -> h2.name>a (5 of 5 learnt values found there; the next best place had 0)
moved: a.title@href -> h2.name>a@href (5 of 5 learnt values found there; the next best place had 0)
moved: span.price -> div.cost (5 of 5 learnt values found there; the next best place had 0)
moved: span.stock -> span.availability (2 of 2 learnt values found there; the next best place had 0)

Limits

  • The listing's place is an exact path. Any new wrapper or renamed class above the rows fails the listing check; that is the point, and heal finds the new place.
  • One listing per extractor: the page's most promising repeated group.
  • Healing matches by values seen before. A redesign that changes both the markup and every value at once -- a different page altogether -- is reported as fields vanished and new, not as moves.
  • The thresholds (20% missing for a required field, half the values keeping their shape, five values before a shape is learnt or checked) are fixed.
  • Two text fields of the same shape that swap values on a page the extractor was not learnt from pass the shape checks; the values check catches a swap only when one side becomes the same in every row.
  • How extractors behave across real changes, and how often healing is right, is measured on Wayback Machine captures in drift, losses first.