Roadmap

Ordered by what unblocks the most, not by what is easiest. Each item ships as working software on its own, with tests, before the next one starts.

Shipped in 0.1.0

Reading what the page declares. JSON-LD, microdata, RDFa, Dublin Core, OpenGraph, the Twitter card and HTML's own metadata names, merged across vocabularies and never within one, in a stated order of precedence, with every field carrying the reader that won it. Microformats2 is read too, by mf2py behind sluicer[microformats] and off until a caller asks: it is compatibility with extruct rather than reach, since the twenty pages measured on 2026-09-22 carried it once, on a page OpenGraph already covered.

A summary with provenance, nested values whole. One answer per common question, each naming the reader and key it came from; offers, authors and ingredient lists carried as the JSON the page declared.

A fetch ladder that climbs on a measurement. Plain HTTP, then a browser, and only when the cheap rung brought back a refusal, a challenge page or a script waiting to render. Every climb is reported with the reason that forced it.

We announce ourselves. An identifiable user agent, robots.txt respected by default, and the stealth rung opt-in rather than automatic. A site owner can see us coming and turn us away with one line.

Structure induction. Pages that declare nothing, read by finding the shape the page repeats and aligning field names across the repetitions. Every induced field says so, and induction only runs where the page declared nothing about its own subject.

The first scoreboard. Sluicer's summary measured beside trafilatura, metascraper and newspaper4k on WCXB's 511 annotated test pages, regenerated by one command, losses included: the scoreboard. It is a start, not the ruler: the corpus strips every <script>, so JSON-LD goes unmeasured, and it labels only three fields.

Shipped in 0.2.0

Extractors that fail loudly. sluicer compile learns an extractor from a few pages of one template and writes it as a small, readable JSON file; sluicer run replays it with no induction and checks every page against what was learnt -- the listing's place, the row count, the required fields, the shape of their values, the summary answers and the declared types -- and exits 3 when a page drifted. sluicer heal learns the page again and says which field moved where, keeping the old names. The same three steps are MCP tools. No model anywhere, including the compile step. See extractors.

Shipped in 0.3.0

Limits an agent can rely on. Every page bounded at 16 MiB, every connection of the HTTP rung pinned to the addresses it checked, every request a browser makes judged before it is made, and every MCP answer carrying ok, an error code and an output schema. sluicer inspect for a person debugging a page, and the evidence behind every move heal reports.

Next

The proof, made bigger. A first drift benchmark is in drift: 44 pairs of Wayback Machine captures on 25 sites, judged by an oracle that does not use the extractor's code, with Scrapling's adaptive selectors beside it. It found five defects, now fixed. Next: more pairs, pairs chosen so that A and B share items and heal can be judged, and a compile that picks the listing rather than page furniture, which it did on 6 of the 25 sites.

A better ruler. Pages as they are served, scripts intact, so JSON-LD is measured; more fields, starting with price and currency; a multilingual e-commerce split.

After that

More than one listing per extractor. A page with "featured", "latest" and "recommended" rows is three listings; an extractor learns only the most promising one today.

Provenance down to the fragment. Every field already names its reader and key; naming the element or the script block it was read from would let an agent show the bytes behind an answer.

Crawling a site. Sitemaps, robots.txt as a map rather than only a rule, and a queue that resumes. Fetching one page is solved; fetching a site politely is not, for us.

Considered and declined

Fetching as an arms race. Browsers and anti-bot evasion are full-time work, and better done by projects that do only that. Sluicer delegates fetching and says so.

Extraction with a model. It would be easier and it would end determinism, which is the property everything else here rests on.

Healing itself above a confidence. heal reports what each move rests on and never applies one because a score was high: a wrong move is the silent failure an extractor exists to prevent, and the person who reads the evidence is the check.

A managed service. There is no plan to sell credits. The constraint that no feature may require somebody's key is the point, not a stage.