Sessions and proxies

Sessions give requests a reusable crawling identity. A Session carries cookies, user data, an error score, a usage count, and a stable session ID. Proxy configuration is independent: it decides how a request reaches its target. The HTTP crawler connects the two systems and exposes the checked-out identity as ctx.session and the resolved proxy metadata as ctx.proxy_info.

Session lifecycle

A SessionPool checks out a usable session for each attempt. Asking the pool for a session without an ID selects or creates one. Supplying an existing ID requests sticky reuse; if that session is missing or no longer usable, the pool selects or creates a replacement. A successful checkout increments the session's usage count.

You can retire an identity through the pool or directly through the session. A session also becomes unusable when it expires, reaches its maximum usage count, reaches its error-score threshold, or has already been retired. Checkout removes unusable entries from consideration.

The health methods make that lifecycle explicit:

• mark_bad() adds one error point.
• mark_good() subtracts the configured scaled decrement.
• is_blocked() reports whether the error threshold has been reached.
• retire() prevents further use of that identity.

Configure pool size, maximum usage, error thresholds, and the persistence key with SessionPoolOptions and SessionConfig.

Persistence

Session pools can exist before crawler storage is open, so storage attachment is a separate operation. Attach a KeyValueStore, call restore().await to replace in-memory state from its saved representation, and call persist().await at an application durability point.

Persisted state includes session IDs, cookies, scores, usage counts, retirement state, and original expiry. If no store is attached, restore and persist succeed without changing storage. Unless overridden, the pool uses SESSION_POOL_PERSIST_KEY.

HTTP integration and separate budgets

Sessions are enabled by default on HttpKindBuilder. The builder can own a configured pool, share an Arc<SessionPool> with another crawler, disable sessions, or replace the response-status set treated as session failures. The usual session-status set can be replaced with session_status_codes(...) when a site uses different blocking responses.

Session rotation is not an ordinary request retry. These are two separate budgets:

• max_session_rotations limits attempts that replace a blocked or otherwise bad identity.
• max_request_retries limits ordinary retryable request failures.

A response classified as a session error rotates or retires the current identity and consumes the session-rotation budget. It does not consume the ordinary request-retry budget. Keeping the budgets separate lets a crawler replace a blocked cookie or IP identity without spending the allowance intended for transient transport and server failures.

Cookies

Every session owns a shared, authoritative CookieJar. A cookie records its name, value, domain, path, expiry, secure and HTTP-only flags, host-only state, and optional SameSite value. Supported SameSite variants are Strict, Lax, and None.

The jar can construct the cookie header for a URL, store response cookies, import and export cookies, report its count, serialize to JSON, restore from JSON, and clear itself. Session::set_cookies_from_response stores response cookies in the same jar. This is the state reused when the next request checks out that session.

Proxy configuration modes

ProxyConfiguration supports four selection shapes:

• Static (round_robin): repeatedly cycles through a fixed URL list.
• Rotating: rotating(urls, strategy) selects from a URL list with either round-robin or random rotation.
• Tiered: tiered(tiers) begins at a lower tier and escalates a target domain after blocking.
• Custom: custom(resolver) delegates selection to an asynchronous ProxyResolver.

new_url(context).await returns a selected proxy URL or None for a direct request. new_proxy_info(context).await returns parsed ProxyInfo, including tier and session metadata. Build the resolution context with ProxyResolveContext and optionally add the request, session ID, and attempt number.

For a tiered configuration, report_blocked(target_url) escalates that domain unless the failed request was a recovery probe. report_success(target_url) accepts a successful lower-tier recovery probe. Both calls are harmless no-ops for non-tiered configurations.

Buckets and proxy retry routing

ProxyBuckets associates proxy kinds with configurations. It can hold a default fallback, a media route, and named custom routes. Install a single configuration with HttpKindBuilder::proxy, install buckets with proxy_buckets, and install a ProxyStrategy with proxy_strategy to select a proxy kind from each route context. When a proxy is selected, its metadata reaches the handler through ctx.proxy_info.

The runnable example demonstrates round-robin proxy resolution both directly and across an offline crawl:

//! Demonstrates round-robin proxy resolution directly and across an offline HTTP crawl.
//!
//! Run with: `cargo run -p millipede --example proxy_switcher`

use std::sync::Arc;

use millipede::{
    Crawler, HttpContext, HttpKind, MemoryStorageClient, ProxyConfiguration, ProxyResolveContext,
};
use url::Url;
use wiremock::{Mock, MockServer, ResponseTemplate, matchers::any};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    resolver_level_demo().await?;
    crawl_level_demo().await?;
    Ok(())
}

async fn resolver_level_demo() -> anyhow::Result<()> {
    let urls = [
        Url::parse("http://proxy-a.invalid:8001")?,
        Url::parse("http://proxy-b.invalid:8002")?,
        Url::parse("http://proxy-c.invalid:8003")?,
    ];
    let config = ProxyConfiguration::round_robin(urls.clone());
    let mut observed = Vec::new();

    for index in 0..6 {
        let context = ProxyResolveContext::new().attempt(index);
        let selected = if index % 2 == 0 {
            config
                .new_url(context)
                .await?
                .ok_or_else(|| anyhow::anyhow!("round-robin resolver returned direct mode"))?
        } else {
            config
                .new_proxy_info(context)
                .await?
                .ok_or_else(|| anyhow::anyhow!("round-robin resolver returned no proxy info"))?
                .url
        };
        println!("resolver selection {} -> {selected}", index + 1);
        observed.push(selected);
    }

    let expected = urls.iter().cycle().take(6).cloned().collect::<Vec<_>>();
    anyhow::ensure!(
        observed == expected,
        "expected A/B/C/A/B/C, got {observed:?}"
    );
    let target = Url::parse("http://target.invalid/")?;
    config.report_success(&target);
    config.report_blocked(&target);
    println!("reported one success and one blocked result to the resolver");
    Ok(())
}

async fn crawl_level_demo() -> anyhow::Result<()> {
    let proxy_a = MockServer::start().await;
    let proxy_b = MockServer::start().await;
    let proxy_c = MockServer::start().await;
    for proxy in [&proxy_a, &proxy_b, &proxy_c] {
        Mock::given(any())
            .respond_with(ResponseTemplate::new(200).set_body_string("stand-in proxy"))
            .mount(proxy)
            .await;
    }

    let proxies = [&proxy_a, &proxy_b, &proxy_c]
        .into_iter()
        .map(|proxy| Url::parse(&proxy.uri()))
        .collect::<Result<Vec<_>, _>>()?;
    let kind = HttpKind::builder()
        .proxy(ProxyConfiguration::round_robin(proxies))
        .build()?;
    let crawler = Crawler::builder(kind)
        .max_concurrency(3)
        .storage_client(Arc::new(MemoryStorageClient::new()))
        .request_handler(|ctx: HttpContext| async move {
            println!(
                "{} reached {:?}",
                ctx.request.url,
                ctx.proxy_info.as_ref().map(|info| &info.url)
            );
            Ok(())
        })
        .build()
        .await?;
    let targets = (0..9)
        .map(|index| format!("http://target-{index}.invalid/"))
        .collect::<Vec<_>>();
    let stats = crawler.run(targets).await?;

    anyhow::ensure!(
        stats.requests_finished == 9 && stats.requests_failed == 0,
        "expected all nine requests to be intercepted by proxies, got {stats:#?}"
    );
    for (name, proxy) in [("A", &proxy_a), ("B", &proxy_b), ("C", &proxy_c)] {
        let count = proxy
            .received_requests()
            .await
            .ok_or_else(|| anyhow::anyhow!("proxy {name} request recording is unavailable"))?
            .len();
        println!("proxy {name} received {count} requests");
        anyhow::ensure!(count == 3, "proxy {name} expected 3 requests, got {count}");
    }
    Ok(())
}

Run it with cargo run -p millipede --example proxy_switcher.

Next steps

• Continue with fingerprinting to keep headers coherent with a stable session seed.
• See error handling for session failures, anti-bot detection, and rotation limits.