Millipede

Millipede is an idiomatic Rust library for crawling and scraping the web. A shared, generic engine handles scheduling, retries, storage, sessions, proxies, and operational limits, while a crawler kind determines how each request is fetched and what typed context reaches your handler. The minimum supported Rust version is 1.85.

Most projects should install the millipede umbrella crate. Its default features are http, html, and storage-memory; storage-fs, browser, browser-chromiumoxide, and fingerprint are opt-in. Enabling browser-chromiumoxide also enables browser, so Chromium users need only select the provider feature.

cargo add millipede

Four ways to crawl

HTTP uses HttpKind when the response body, status, and headers are enough. It avoids DOM parsing and browser startup, making it the leanest path for APIs, feeds, files, and custom byte processing.

HTML uses HtmlKind to add a synchronized parsed document and selector-driven link discovery to the HTTP path. Choose it for server-rendered pages whose useful content is already present in the response.

Browser uses BrowserKind<ChromiumoxideProvider> to drive Chrome or Chromium. It is intended for pages that require JavaScript execution, browser navigation, or access to a live page.

Smart uses SmartKind<ChromiumoxideProvider> to try the efficient HTTP-and-HTML path first, then promote responses identified as browser-only to Chromium. It fits mixed sites where most pages are static but a smaller set needs JavaScript.

A complete starting point

This doc-tested excerpt comes from the umbrella crate's own README. It supplies memory storage explicitly, handles parsed HTML, records a dataset row, discovers same-host links, and reads the final crawl statistics. To run it in a new project after adding millipede, also add the two crates used by the example harness:

cargo add tokio --features full
cargo add serde_json

use std::sync::Arc;

use millipede::{CrawlPolicy, Crawler, DatasetExt, HtmlContext, HtmlCrawler, HtmlKind};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let crawler: HtmlCrawler = Crawler::builder(HtmlKind::new()?)
        .storage_client(Arc::new(millipede::MemoryStorageClient::new()))
        .crawl_policy(CrawlPolicy::new().max_requests_per_crawl(100))
        .request_handler(|ctx: HtmlContext| async move {
            ctx.storage
                .dataset()
                .push(&serde_json::json!({
                    "url": ctx.request.url.as_str(),
                    "status": ctx.response.status.as_u16(),
                }))
                .await?;
            let _ = ctx.enqueue.same_hostname().await?;
            Ok(())
        })
        .build()
        .await?;

    let stats = crawler.run("https://example.com/").await?;
    println!("finished: {}", stats.requests_finished);
    Ok(())
}

Explore the documentation

• Quick start — choose a crawler kind and learn the common build-and-run lifecycle.
• Introduction — follow the course spine from a first crawl into a structured scraper.
• Guides — configure storage, sessions, proxies, retries, autoscaling, browsers, and other controls.
• Examples — find runnable programs organized by use case.
• API reference — jump from Millipede concepts to the relevant crate documentation.

Next steps

• Choose a crawler
• Begin the introduction
• Browse runnable examples