Error handling

Classify crawl failures with typed errors, control retries and session rotation, and inspect terminal failure artifacts.

Request handlers return errors through CrawlError. The variant communicates retry intent to the engine, while an optional custom retry strategy can refine the decision for non-critical failures.

Typed crawl errors

CrawlError is non-exhaustive, so future releases may add variants. The current set, cross-checked against the guide and the runnable example, is:

Variant	Default engine behavior
`Retry(error)`	Retry and count the attempt against `max_request_retries`.
`Session(error)`	Retry with a new session and count against `max_session_rotations`, not the ordinary retry limit.
`ForceRetry(error)`	Retry without observing `max_request_retries`.
`NonRetryable(error)`	Stop retrying and enter the permanent-failure path.
`Critical(error)`	Abort the crawler.
`MissingRoute { label, method }`	Report that no router entry matched both label and HTTP method.
`AntiBotDetected { tech, source }`	Report a recognized anti-bot response and rotate the session on retry.

Constructor helpers named retry, session, force_retry, non_retryable, and critical accept values convertible to an anyhow::Error. Classification methods such as is_retryable, rotates_session, ignores_max_retries, is_critical, and counts_against_retries expose the same policy to custom code.

millipede/examples/error_handling.rs

//! Demonstrates retryable HTTP statuses, anti-bot detection, and typed `CrawlError` dispatch.
//!
//! Run with: `cargo run -p millipede --example error_handling`

use std::sync::{
    Arc,
    atomic::{AtomicUsize, Ordering},
};

use millipede::{
    CrawlError, Crawler, FailedRequestContext, HttpContext, HttpKind, MemoryStorageClient,
};
use wiremock::{
    Mock, MockServer, ResponseTemplate,
    matchers::{method, path},
};

fn variant_name(error: &CrawlError) -> &'static str {
    match error {
        CrawlError::Retry(_) => "Retry",
        CrawlError::Session(_) => "Session",
        CrawlError::ForceRetry(_) => "ForceRetry",
        CrawlError::NonRetryable(_) => "NonRetryable",
        CrawlError::Critical(_) => "Critical",
        CrawlError::MissingRoute { label, method } => {
            println!("missing route details: label={label:?}, method={method}");
            "MissingRoute"
        }
        CrawlError::AntiBotDetected { tech, source } => {
            println!("anti-bot details: tech={tech:?}, source={source}");
            "AntiBotDetected"
        }
        _ => "unknown future CrawlError variant",
    }
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let server = MockServer::start().await;
    Mock::given(method("GET"))
        .and(path("/ok"))
        .respond_with(ResponseTemplate::new(200).set_body_string("ok"))
        .mount(&server)
        .await;
    Mock::given(method("GET"))
        .and(path("/flaky"))
        .respond_with(ResponseTemplate::new(500).set_body_string("try again"))
        .up_to_n_times(2)
        .mount(&server)
        .await;
    Mock::given(method("GET"))
        .and(path("/flaky"))
        .respond_with(ResponseTemplate::new(200).set_body_string("recovered"))
        .mount(&server)
        .await;
    Mock::given(method("GET"))
        .and(path("/blocked"))
        .respond_with(ResponseTemplate::new(403).set_body_raw(
            "<html><head><title>Just a moment...</title></head></html>",
            "text/html",
        ))
        .mount(&server)
        .await;

    let failures = Arc::new(AtomicUsize::new(0));
    let anti_bot_failures = Arc::new(AtomicUsize::new(0));
    let kind = HttpKind::builder()
        .retry_status_codes([500, 502, 503])
        .detect_anti_bot_default()
        .build()?;
    let crawler = Crawler::builder(kind)
        .max_request_retries(2)
        // AntiBotDetected rotates sessions, so this is its corresponding terminal retry cap.
        .max_session_rotations(2)
        .storage_client(Arc::new(MemoryStorageClient::new()))
        .request_handler(|ctx: HttpContext| async move {
            println!("success: {} ({})", ctx.request.url, ctx.response.status);
            Ok(())
        })
        .failed_request_handler({
            let failures = Arc::clone(&failures);
            let anti_bot_failures = Arc::clone(&anti_bot_failures);
            move |ctx: FailedRequestContext| {
                let failures = Arc::clone(&failures);
                let anti_bot_failures = Arc::clone(&anti_bot_failures);
                async move {
                    failures.fetch_add(1, Ordering::SeqCst);
                    let variant = variant_name(ctx.error.as_ref());
                    if matches!(ctx.error.as_ref(), CrawlError::AntiBotDetected { .. }) {
                        anti_bot_failures.fetch_add(1, Ordering::SeqCst);
                    }
                    println!(
                        "failed: {} variant={variant} ignores_max_retries={} retry_count={}",
                        ctx.request.url,
                        ctx.error.ignores_max_retries(),
                        ctx.retry_count
                    );
                    Ok(())
                }
            }
        })
        .build()
        .await?;

    let base = server.uri();
    let stats = crawler
        .run([
            format!("{base}/ok"),
            format!("{base}/flaky"),
            format!("{base}/blocked"),
        ])
        .await?;

    anyhow::ensure!(
        stats.requests_finished == 2,
        "expected /ok and /flaky to succeed, got {stats:#?}"
    );
    anyhow::ensure!(
        stats.requests_failed == 1,
        "expected only /blocked to fail, got {stats:#?}"
    );
    anyhow::ensure!(
        stats.requests_retries == 4,
        "expected two /flaky retries and two /blocked rotations, got {stats:#?}"
    );
    anyhow::ensure!(
        failures.load(Ordering::SeqCst) == 1,
        "the failed-request handler did not fire exactly once"
    );
    anyhow::ensure!(
        anti_bot_failures.load(Ordering::SeqCst) == 1,
        "the /blocked terminal error was not AntiBotDetected"
    );
    println!(
        "summary: /ok succeeded, /flaky succeeded after two retries, /blocked reached the typed failure handler"
    );
    Ok(())
}

Run the example with cargo run -p millipede --example error_handling.

Retry strategy and directives

Without a custom RetryStrategy, the error variant supplies the default retry and session policy. A configured strategy receives an AttemptOutcome and returns a RetryDirective. The directive's should_retry value can stop a normally retryable error or retry a normally non-retryable error, and the strategy's max_retries() supplies the ordinary retry limit.

Two cases bypass that authority: a Critical error aborts, and a request marked no_retry is not offered to the strategy. ForceRetry remains special because it ignores the normal retry maximum.

The engine appends error messages to the request as attempts fail. A request enters the permanent-failure path when its applicable budget is exhausted or when the active default or custom policy decides not to retry.

HTTP status failures

HttpKindBuilder has two independent status controls:

retry_status_codes(...) replaces the explicit set of retryable response codes.
retry_server_errors(bool) enables or disables retry treatment for server-error responses.

Session-status codes are different. They classify a response as an identity failure, retire or rotate the current session, and consume max_session_rotations. That budget remains separate from max_request_retries so a blocked identity does not spend the transient request-failure allowance.

Failed-request handler timing

Register failed_request_handler on the crawler builder for terminal, isolated request failures. It receives a FailedRequestContext containing the terminal request and error. Retryable failures reach this callback only after the relevant retry or session-rotation policy stops; permanent errors can reach it immediately.

Use the callback for terminal logging, dead-letter storage, or alerts. A Critical error aborts the run instead of being dispatched as an isolated failed request. If the failed-request handler itself returns an error, the engine logs that error.

Anti-bot detection

An AntiBotDetector inspects response signals and may identify an anti-bot technology. The default detector uses bounded static response markers for Cloudflare, DataDome, PerimeterX, Kasada, Imperva, and Akamai, and can also report a custom or unknown result.

Enable the built-in detector with detect_anti_bot_default(), or install an Arc<dyn AntiBotDetector> with detect_anti_bot(...). Before installation, the default detector can be adjusted with an inspection limit and custom markers. A detected challenge becomes AntiBotDetected { tech, source }; by default it is retryable and rotates the session.

Error snapshots

ErrorSnapshotter stores failure artifacts in a key-value store. For HTTP crawling, snapshot_errors_on_failure(true) captures the response body when the request handler fails. An execute-time failure before a handler context exists has no response body to capture.

Snapshot keys are deterministic. The base is ERROR_SNAPSHOT_ followed by a 16-digit lowercase hexadecimal hash of the request's unique key. Capture appends a suffix; the HTTP handler-failure path uses .body.

See fingerprint crawl for anti-bot recovery, body snapshots, and snapshot loading in action.

Normalized error statistics and skipped URLs

StatisticsSnapshot and FinalStatistics expose errors for terminal failures and retry_errors for retry attempts. Both group counts in BTreeMap<String, u64> values. Millipede normalizes the first error line by masking URLs and UUIDs, collapsing digit runs, and limiting key length, which prevents request-specific values from fragmenting the groups.

Admission failures are observable before navigation. EnqueueResult returns skipped URLs with reasons for depth and request limits, strategy, glob, or regex exclusion, transform rejection, duplicate unique keys, and invalid URLs. A crawl policy may also observe each reported rejection through on_skipped.

Next steps

Configure identity recovery in sessions and proxies.
Walk through failure artifacts in fingerprint crawl.

Error handling

On this page