Crawl all the links

Discover same-host links while keeping the crawl frontier bounded and deduplicated.

The first crawler introduced a frontier: the set of discovered requests that have not yet finished. Every handled page can add to it, and every completed request removes work from it. A useful crawler must decide which links may enter that frontier and when to stop accepting more.

For the common case, ctx.enqueue.same_hostname() extracts page links and admits only URLs whose hostname matches the current page. It is backed by the EnqueueStrategy::SameHostname strategy. The convenience method is a good fit when every accepted link receives the same treatment.

Use the full options pipeline when discovery needs more intent: ctx.enqueue.options().selector(...).label(...).send(). The selector restricts extraction to the matching elements, the label marks the requests for a route, and send() submits the candidates. Strategy, glob, and per-call limit options can further narrow that set when the crawler needs them.

Discovery is not the same as scheduling every link occurrence. The request queue deduplicates by a request's unique key. If navigation, category, and pagination elements produce 60 enqueue attempts but several point to the same URLs or otherwise share deduplication keys, the queue might contain only 40 new entries. A URL rediscovered later does not continually recreate completed work. Once no genuinely new unique keys appear and pending work is handled, the frontier can empty instead of cycling forever through repeated links.

Deduplication controls repeats; a CrawlPolicy controls total scope. CrawlPolicy::max_requests_per_crawl(...) is a hard cap for the run. Keep that guard in place while developing selectors, especially before pointing the next version at the book catalogue.

The compiled example below is a gocolly-style link-scraping demonstration. It serves a small local mock site, discovers its links, applies same-hostname filtering, and verifies that an external URL was never fetched.

millipede/examples/basic.rs

//! Demonstrates gocolly-style link scraping and same-hostname filtering with `HtmlCrawler`.
//!
//! Run with: `cargo run -p millipede --example basic`

use std::{collections::BTreeSet, sync::Arc};

use millipede::{
    CrawlPolicy, Crawler, EnqueueStrategy, HtmlContext, HtmlKind, MemoryStorageClient,
};
use wiremock::{
    Mock, MockServer, ResponseTemplate,
    matchers::{method, path},
};

millipede_html::selectors! {
    title_selector = "title";
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let server = MockServer::start().await;
    let server_uri = server.uri();

    Mock::given(method("GET"))
        .and(path("/"))
        .respond_with(ResponseTemplate::new(200).set_body_raw(
            r#"<html><head><title>Home</title></head><body>
                <a href="/a">A</a>
                <a href="/b">B</a>
                <a href="http://external.invalid/x">External</a>
            </body></html>"#,
            "text/html",
        ))
        .mount(&server)
        .await;
    for (page, title) in [("/a", "Page A"), ("/b", "Page B")] {
        Mock::given(method("GET"))
            .and(path(page))
            .respond_with(ResponseTemplate::new(200).set_body_raw(
                format!("<html><head><title>{title}</title></head></html>"),
                "text/html",
            ))
            .mount(&server)
            .await;
    }

    let crawler = Crawler::builder(HtmlKind::new()?)
        .crawl_policy(CrawlPolicy::new().max_requests_per_crawl(10))
        .storage_client(Arc::new(MemoryStorageClient::new()))
        .request_handler(|ctx: HtmlContext| async move {
            let title = ctx
                .html
                .select_first(title_selector(), |element| {
                    element.text().collect::<String>()
                })
                .unwrap_or_else(|| "<untitled>".to_owned());
            println!("{} -> {title}", ctx.request.url);
            let _ = ctx
                .enqueue
                .options()
                .strategy(EnqueueStrategy::SameHostname)
                .send()
                .await?;
            Ok(())
        })
        .build()
        .await?;

    let stats = crawler.run(format!("{server_uri}/")).await?;
    let requests = server
        .received_requests()
        .await
        .ok_or_else(|| anyhow::anyhow!("wiremock request recording is unavailable"))?;
    let paths = requests
        .iter()
        .map(|request| request.url.path().to_owned())
        .collect::<BTreeSet<_>>();

    anyhow::ensure!(
        stats.requests_finished == 3 && stats.requests_failed == 0,
        "domain filtering failed: expected 3 local successes and no failures, got {stats:#?}"
    );
    anyhow::ensure!(
        paths == BTreeSet::from(["/".to_owned(), "/a".to_owned(), "/b".to_owned()]),
        "expected only the three local paths, got {paths:?}"
    );
    anyhow::ensure!(
        requests.len() == 3,
        "external link may have escaped filtering: local server saw {} requests",
        requests.len()
    );
    println!(
        "summary: crawled {} local pages; external.invalid was filtered before fetching",
        stats.requests_finished
    );
    Ok(())
}

Run it from the repository root:

cargo run -p millipede --example basic

With a safe frontier in place, the crawler is ready to distinguish catalogue navigation from the book pages that contain the records we want.

Next steps

Continue with scrape structured data.

Crawl all the links

Next steps

On this page