HTML crawler

HtmlKind builds on HTTP fetching and adds a synchronized parsed document to HtmlContext. It is a strong default for sites whose useful content and links are present in the server response, without running JavaScript.

The selectors! macro validates CSS selectors and caches them for reuse. In a handler, ctx.html.select_first(...) can read from the parsed document without reparsing the response for every query.

The following doc-tested excerpt comes from the millipede-html crate's own README and uses the individual sub-crates directly. To follow it in a new project, add those direct dependencies:

cargo add millipede-core
cargo add millipede-html
cargo add millipede-storage-memory

use std::sync::Arc;

use millipede_core::prelude::Crawler;
use millipede_html::{HtmlContext, HtmlCrawler, HtmlKind};
use millipede_storage_memory::MemoryStorageClient;

millipede_html::selectors! {
    title_selector = "title";
}

let crawler: HtmlCrawler = Crawler::builder(HtmlKind::new()?)
    .storage_client(Arc::new(MemoryStorageClient::new()))
    .request_handler(|ctx: HtmlContext| async move {
        if let Some(title) = ctx
            .html
            .select_first(title_selector(), |element| element.text().collect::<String>())
        {
            println!("{}: {title}", ctx.request.url);
        }
        let _ = ctx.enqueue.options().selector("a[href]").send().await?;
        Ok(())
    })
    .build()
    .await?;

crawler.run(["https://example.com/"]).await?;

Keep discovery intentional

A CrawlPolicy places crawl-wide limits and URL rules alongside the crawler. Pairing it with EnqueueStrategy::SameHostname keeps automatic discovery on the starting hostname. A handler can enqueue matching links directly, while ctx.enqueue.options().selector(...).label(...).send() adds a label when the destination needs different processing.

For multi-page scrapers, Router<HtmlContext> maps those labels to handlers for distinct page types. This keeps listing-page discovery separate from detail-page extraction while both routes share the crawler's queue, storage, retry policy, and statistics.

Next steps

• Follow the introduction course
• Run the book scraper
• Learn enqueue links and crawl policies