# Sluicer

> Turn a web page into structured data. No model, no API key, no bill.

Every link below is a page's markdown source, served beside the page;
the site shows the same page as HTML.

## Start

- [Home](https://gi0tto.github.io/sluicer/index.md): Most scrapers break without a sound. The site changes its markup, and the scraper keeps running and returns nulls, or the wrong column, for weeks before anyone notices.
- [Getting started](https://gi0tto.github.io/sluicer/getting-started.md): Ten minutes from nothing to reading your own pages: install Sluicer, read a page, understand what came back, and do the same from Python and from an agent.

## Guides

- [Extractors](https://gi0tto.github.io/sluicer/extractors.md): The most common way a scraper fails is not a crash. The site changes its markup, and the scraper keeps running and returns nulls, or the wrong column, for weeks before anyone notices. An extractor is built the other way round: it is learnt once from a few pages, kept in a small file, replayed for nothing, and every replay checks the page against what was learnt. A page that drifted is a failed run, never a quiet one. When it fails, `heal` says what moved and where.
- [In your agent](https://gi0tto.github.io/sluicer/agents.md): Sluicer's MCP server is one command, and every client that speaks MCP can start it.
- [HTTP API](https://gi0tto.github.io/sluicer/http-api.md): `sluicer serve` answers the MCP server's tools over HTTP, one address per tool, so any language that can send a POST can use Sluicer. They are the twelve tools an agent gets, each in the MCP reference, with the same arguments, the same answers and the same output schemas, because it is built from them: an HTTP call goes through the MCP SDK's own `call_tool`, argument validation and output-schema check included. A tool the MCP server gains is served here with nothing else to change. At `/mcp` it is the MCP server itself, over MCP's streamable HTTP transport, for a client that does not start servers over stdio: MCP over HTTP, with n8n and Dify.
- [In JavaScript](https://gi0tto.github.io/sluicer/javascript.md): `sluicer` on npm is the Python package itself, run in Pyodide, Python compiled to WebAssembly. Nothing is rewritten in JavaScript: the package carries the wheel built from the same commit as the Python release of the same version, installs it into Pyodide when it starts, and hands every call to it. The answers are the Python package's, field for field: its tests hold them to the native package's on 22 pages, a page for each reader and each option.
- [Crawling](https://gi0tto.github.io/sluicer/crawling.md): `extract` reads the page you name. A site is many pages, and there are three ways to read one: from the list its sitemaps give (`map`), by following its links (`crawl`), or from a list you already have (`batch`). Every page goes through the same fetch ladder as one page does, so `robots.txt`, the address guard and the 16 MiB bound apply to each of them, and every site is asked one request at a time with its delay between. Politeness is not an option here; it is what the feature is.
- [Configuration](https://gi0tto.github.io/sluicer/configuration.md): The options you give every command -- a proxy, the headers a site wants, a cache, a crawl's delay, JSON output -- can be given once, in a file. The file is read by the `sluicer` command only: the Python API, the MCP server and the HTTP API take their arguments as they are called and read no file.
- [Web archives](https://gi0tto.github.io/sluicer/warc.md): Two ways in: a WARC file you hold, and a page as the Wayback Machine captured it.
- [Audit](https://gi0tto.github.io/sluicer/audit.md): `sluicer audit` holds what a page declares to what its documentation asks of it. For every record JSON-LD, microdata and RDFa declare, it says which of Google's rich-result features the record's type is documented for, which required and recommended properties it lacks, and which of its values are in a form the documentation or schema.org refuses. Then what the page as a whole lacks, where two vocabularies contradict each other, which AI agents the site's robots.txt admits, and whether its llms.txt keeps to llmstxt.org's format. In a workflow, the GitHub Action runs it on the pages you name and fails the check when one breaks a rule.
- [In GitHub Actions](https://gi0tto.github.io/sluicer/github-action.md): `action.yml` at the repository's root is a GitHub Action that runs `sluicer audit` on the pages you name, so a pull request that breaks a page's structured data fails its checks, with each rule it breaks linked to the documentation that states it.
- [Moving from extruct](https://gi0tto.github.io/sluicer/extruct.md): extruct reads the vocabularies a page declares and returns each as the page wrote it. It has had no release since 8 November 2024. `sluicer.compat.extruct` answers extruct's calls with extruct's shapes, so the move is one line.

## Reference

- [Command line](https://gi0tto.github.io/sluicer/reference/cli.md): Every command's own `--help`, as `sluicer` prints it. Generated from the code by `scripts/reference.py`. Commands and options complete in bash, zsh and fish: see Shell completion; their defaults can come from a file: see Configuration.
- [MCP tools](https://gi0tto.github.io/sluicer/reference/mcp.md): The 12 tools the server lists to an agent, each with its description and parameters exactly as the agent receives them. Generated from the running server by `scripts/reference.py`; how to add the server to a client is in In your agent.
- [Python](https://gi0tto.github.io/sluicer/reference/python.md): The functions and types a program calls, with their signatures and their own docstrings. Generated from the code by `scripts/reference.py`.

## Measured

- [Scoreboard](https://gi0tto.github.io/sluicer/scoreboard.md): How often Sluicer's `summary` gets a page's title, author and publication date right, measured beside the tools people use for the same job, on a public annotated corpus, with the losses in the same table as the wins. Regenerated on 2026-09-25 from commit `10211f3` by `uv run bench/run.py`; the method and every pin are in `bench/`.
- [As served](https://gi0tto.github.io/sluicer/scoreboard-served.md): The scoreboard measures Sluicer on WCXB's copies of its pages, and WCXB removed every `<script>` from them, JSON-LD included. This page measures the same tools, with the same labels and the same scorer, on the same pages as their servers sent them, scripts intact, fetched from web archives. Every page is scored twice, once as served and once as WCXB kept it, so the difference between the two columns is the difference the scripts make, and nothing else. Regenerated on 2026-09-25 from commit `10211f3` by `uv run bench/realweb.py`, from the captures pinned in `bench/realweb-manifest.json`.
- [Products](https://gi0tto.github.io/sluicer/scoreboard-products.md): Sluicer on Zyte's product-extraction benchmark: 140 product pages as they were served in 2021, scripts intact, with price, SKU and availability labelled by hand. The benchmark ships the predictions of Zyte's and Diffbot's paid extraction APIs and an open-source baseline built on extruct and price-parser, and the evaluator that scores them. Sluicer's predictions are added beside theirs and that evaluator runs unchanged, so every rule below is Zyte's: a price matches as a decimal, several values can be right, and a page with no availability counts as in stock.
- [News in many languages](https://gi0tto.github.io/sluicer/scoreboard-news.md): The same questions as the scoreboard -- a page's title, author and publication date -- on news pages from 42 countries' publishers, in 21 declared languages, with their scripts: as fundus fetched them, stored re-encoded as UTF-8. Regenerated on 2026-09-25 from commit `10211f3` by `uv run bench/news.py`, against fundus at `c1b86b675018`; the method is in `bench/`.
- [trafilatura's evaluation set](https://gi0tto.github.io/sluicer/scoreboard-evaldata.md): The same questions as the scoreboard -- a page's title, author and publication date -- on the pages trafilatura evaluates itself on, 990 saved with their scripts, 851 of them annotated for their metadata, and the main text beside them. Regenerated on 2026-09-25 from commit `10211f3` by `uv run bench/evaldata.py`, against trafilatura at `c852cae9708a`; the method is in `bench/`.
- [Extractors on SWDE](https://gi0tto.github.io/sluicer/scoreboard-swde.md): What `sluicer compile --want` promises, measured: point at a value on a few pages of one template, and read it on every other page of that template. The pages are SWDE, the Structured Web Data Extraction dataset (Hao, Cai, Pang and Zhang, SIGIR 2011): 124,291 detail pages from 80 sites in 8 verticals, crawled around 2010, each labelled with the values of three to five attributes. It is the dataset wrapper induction is measured on.
- [RDFa, against the W3C suite](https://gi0tto.github.io/sluicer/scoreboard-rdfa.md): Sluicer's two RDFa readers, and extruct's, on the RDFa test suite the W3C's RDFa working group wrote and rdfa.info runs: every test written for HTML5, each document handed to each reader, each answer read into a graph and asked the test's own SPARQL query.
- [Drift](https://gi0tto.github.io/sluicer/drift.md): What an extractor does when the page it was learnt from changes. Each extractor is learnt on an old Wayback Machine capture of a listing page and replayed on a later one. An oracle that does not use the extractor's code judges the result. Losses come first.
- [Speed and weight](https://gi0tto.github.io/sluicer/speed.md): How long each tool takes on the scoreboards' pages, how much memory it holds, and how much it installs, measured one way for every tool, as `bench/PREREG.md` fixes it under "How a second is measured". Every tool of a table was timed in one run, on one machine, on the same pages; a scoreboard prints no second measured otherwise. Regenerated from commit `10211f3` by `uv run bench/timing.py`.
- [Conformance: JSON-LD in HTML](https://gi0tto.github.io/sluicer/conformance-jsonld.md): The W3C's JSON-LD 1.1 test suite has 50 tests of JSON-LD inside HTML: which of a page's scripts a processor extracts (the first, the one a fragment names, or all of them), which script text is an error, which base IRI the page sets, and what expanding, compacting, flattening or turning the result into RDF gives. Sluicer's JSON-LD reader, its extruct interface and extruct itself read every page; each answer, as the reader gives it, is processed by PyLD, a JSON-LD processor, with the test's options, and compared with the suite's expected result by the suite's own rules. Regenerated on 2026-09-25 from commit `10211f3` by `uv run bench/w3c_jsonld.py`, against the suite at `ffdb326121ea`; the rules are in `bench/PREREG.md`.
- [The field, measured](https://gi0tto.github.io/sluicer/field-survey.md): Claims about being better than the alternatives are worth nothing without a count. This is the count, taken on 2026-09-22, and it is reproducible: the queries are at the bottom.
- [The state of declared data](https://gi0tto.github.io/sluicer/state-of-declared-data.md): What web pages declare about themselves -- JSON-LD, microdata, RDFa, microformats, OpenGraph, the Twitter card, Dublin Core, HTML's own meta names -- counted by Sluicer on 86,029 pages of Common Crawl's `CC-MAIN-2026-39`: every HTML page answered 200 in 4 of the crawl's 100,000 WARC files, chosen by a rule fixed before any was downloaded. Each is read as `sluicer.extract` reads a page, with its provenance, and the conflicts between what one page declares twice are counted with the rest.

## Project

- [Why Sluicer](https://gi0tto.github.io/sluicer/why.md): Most of what a scraper is asked for is already written into the page it scrapes. A product page carries its name, price, currency and SKU in JSON-LD for search engines; an article carries its headline, author and dates in OpenGraph and meta tags. Sluicer reads that, rather than guessing it from the visible text or asking a model to.
- [FAQ](https://gi0tto.github.io/sluicer/faq.md)
- [Known limits](https://gi0tto.github.io/sluicer/known-limits.md): Everything here is deliberate, measured, and open. It is not a bug list: it is the set of places where Sluicer currently stops, written down so nobody has to rediscover them, and so a contributor inherits decided questions instead of surprises.
- [What is stable](https://gi0tto.github.io/sluicer/stability.md): Sluicer is Beta, and 1.0 is not out. This page says which parts you can build on now, which may still change, and how a change reaches you.
- [Design notes](https://gi0tto.github.io/sluicer/design-notes.md): The reasons behind rules the code states tersely. Each note says what the rule is, what it protects against, and where it lives.
- [Roadmap](https://gi0tto.github.io/sluicer/roadmap.md): Ordered by what unblocks the most, not by what is easiest. Each item ships as working software on its own, with tests, before the next one starts.
- [Changelog](https://gi0tto.github.io/sluicer/changelog.md): Dates are the day the work landed. Anything not listed here did not happen.
- [Contributing](https://gi0tto.github.io/sluicer/contributing.md): Thank you for looking. This is a small project with strong opinions, and the opinions are the reason it exists, so it is worth knowing them before you spend an evening on a patch. A question, or an idea you would like to talk through first, goes to Discussions; a page Sluicer read wrong is an issue.
- [Security](https://gi0tto.github.io/sluicer/security.md)