Scrape structured data

Route listing and detail pages, select book fields, and emit typed dataset records.

The frontier now finds pages safely, but not every page has the same job. On Books to Scrape, a category or catalogue page contains pagination controls and links to books. A book detail page contains the title, price, and availability that belong in one record. Plan selectors separately for those two page shapes before writing handler logic.

The millipede_html::selectors! macro defines and caches selectors that are known ahead of time. That crate-direct path requires millipede-html as a direct dependency; enabling the umbrella crate's default html feature alone does not make the transitive crate name available. Inside a handler, ctx.html.select_first(...) applies one of those selectors and maps the first matching element into a value. This keeps document queries close to the fields they populate and avoids repeatedly parsing static selector strings.

Labels connect discovery to dispatch. The listing handler uses ctx.enqueue.options().selector(...).label(...).send() for book links, assigning a label such as DETAIL. A Router then keeps the two page types from becoming one large conditional handler. A labeled route is registered with Router::<HtmlContext>::new().route("DETAIL", detail_handler). The listing fallback is registered with the builder chain Router::<HtmlContext>::new().default(listing_handler).route("DETAIL", detail_handler); it is not a route named default. Unlabeled catalogue requests then reach the listing handler while DETAIL requests reach the detail handler.

The detail handler builds a typed record struct for each book. Deriving serialization for that record lets DatasetExt turn it into a dataset row while the Rust type keeps the expected fields visible in the program.

The finished example combines the selectors, labeled discovery, routing, bounded crawl policy, and file-system dataset output:

It is compiled as a workspace example, where its dependencies are already declared. To place the same source in an external project, also enable the umbrella crate's storage-fs feature and add the crate-direct selector and error dependencies (Tokio and serde_json come from the course prerequisites):

cargo add millipede --features storage-fs
cargo add millipede-html anyhow

millipede/examples/scrape_books.rs

//! Crawls category and detail pages on books.toscrape.com and stores a structured book dataset.
//!
//! Run with:
//! `cargo run -p millipede --example scrape_books --features storage-fs`
//!
//! This example HITS THE REAL NETWORK. books.toscrape.com is an open-source practice site for
//! scraping. CI instead runs the wiremock version in `millipede/tests/scrape_books_mock.rs`, which
//! is why this example is deliberately not in the `.github/workflows/ci.yml` examples job.
//! `MILLIPEDE_BOOKS_BASE_URL` or the first command-line argument overrides the base URL.

use std::sync::Arc;

use millipede::{
    Configuration, CrawlPolicy, Crawler, DatasetExt, EnqueueStrategy, FailedRequestContext,
    FsStorageClient, HtmlContext, HtmlKind, Router,
};
use serde_json::json;

millipede_html::selectors! {
    title_selector = "h1";
    price_selector = "p.price_color";
    availability_selector = "p.instock.availability";
}

fn text(ctx: &HtmlContext, selector: &millipede_html::scraper::Selector) -> String {
    ctx.html
        .select_first(selector, |element| element.text().collect::<String>())
        .unwrap_or_default()
        .trim()
        .to_owned()
}

fn base_url() -> String {
    std::env::args()
        .nth(1)
        .or_else(|| std::env::var("MILLIPEDE_BOOKS_BASE_URL").ok())
        .unwrap_or_else(|| "https://books.toscrape.com/".to_owned())
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let storage_client = Arc::new(FsStorageClient::new("./storage"));
    let configuration = Configuration::builder().purge_on_start(true).build()?;

    let router = Router::<HtmlContext>::new()
        .default(|ctx: HtmlContext| async move {
            let _ = ctx.enqueue.options().selector("ul.pager a").send().await?;
            let _ = ctx
                .enqueue
                .options()
                .selector("article.product_pod h3 a")
                .globs(["**/catalogue/**"])
                .label("detail")
                .send()
                .await?;
            Ok(())
        })
        .route("detail", |ctx: HtmlContext| async move {
            let book = json!({
                "url": ctx.request.url,
                "title": text(&ctx, title_selector()),
                "price": text(&ctx, price_selector()),
                "availability": text(&ctx, availability_selector()),
            });
            ctx.storage.dataset().push(&book).await?;
            Ok(())
        });

    let crawler = Crawler::builder(HtmlKind::new()?)
        .configuration(configuration)
        .storage_client(storage_client)
        .crawl_policy(
            CrawlPolicy::new()
                .strategy(EnqueueStrategy::SameHostname)
                .max_requests_per_crawl(200),
        )
        .max_concurrency(5)
        .request_handler(router)
        .failed_request_handler(|ctx: FailedRequestContext| async move {
            eprintln!("book request failed for {}: {}", ctx.request.url, ctx.error);
            Ok(())
        })
        .build()
        .await?;

    let stats = crawler.run(base_url()).await?;
    println!("crawl complete: {stats:#?}");
    println!(
        "books are in ./storage/datasets/default/ using a Crawlee-compatible layout; see \
         docs/guide/crawlee-storage-migration.md"
    );
    Ok(())
}

Run it from the repository root:

cargo run -p millipede --example scrape_books --features storage-fs

It uses the real practice site by default. Set MILLIPEDE_BOOKS_BASE_URL to a local mock or an approved mirror when testing without the public network. For a larger example that separates multiple page shapes and typed records, see the Hacker News crawler.

The crawler now produces meaningful rows. The final lesson explains where those rows and the rest of the crawl state live, and how to keep persistent state from being cleared on restart.

Next steps

Continue with save and resume data.

Scrape structured data

Next steps

On this page