Browser crawler
Drive a local Chrome or Chromium browser for pages that require JavaScript execution.
Browser crawler
Choose BrowserKind<ChromiumoxideProvider> when the target must execute JavaScript or your handler needs a live browser page. The Chromiumoxide integration connects Millipede's provider-neutral browser crawler to Chrome or Chromium through the Chrome DevTools Protocol.
Install the opt-in provider feature; it also enables the browser feature:
cargo add millipede --features browser-chromiumoxideFind the browser executable
A local Chrome or Chromium installation is required. find_browser() checks MILLIPEDE_CHROME first, then CHROME, then conventional installation paths for the current platform. An explicit path can be applied with ChromiumLaunchOptions::default().with_executable(...) before the browser kind is built.
The following doc-tested excerpt comes from the millipede-browser-chromiumoxide crate's own README and uses the individual sub-crates directly. To follow it in a new project, add those direct dependencies and Tokio:
cargo add millipede-browser
cargo add millipede-browser-chromiumoxide
cargo add millipede-core
cargo add millipede-storage-memory
cargo add tokio --features fulluse std::sync::Arc;
use millipede_browser::{BrowserContext, BrowserCrawler, BrowserKind};
use millipede_browser_chromiumoxide::{
ChromiumLaunchOptions, ChromiumoxideProvider, find_browser,
};
use millipede_core::prelude::Crawler;
use millipede_storage_memory::MemoryStorageClient;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let executable = find_browser().expect("Chrome or Chromium is required");
let kind = BrowserKind::builder(ChromiumoxideProvider)
.launch_options(ChromiumLaunchOptions::default().with_executable(executable))
.build()?;
let crawler: BrowserCrawler<ChromiumoxideProvider> = Crawler::builder(kind)
.storage_client(Arc::new(MemoryStorageClient::new()))
.request_handler(|ctx: BrowserContext| async move {
let title = ctx.page.evaluate_js("document.title").await.ok();
println!("{}: {title:?}", ctx.request.url);
Ok(())
})
.build()
.await?;
crawler.run(["https://example.com/"]).await?;
Ok(())
}Run the complete workspace example with:
cargo run -p millipede --features browser-chromiumoxide,storage-memory --example browser_crawl