Millipede

Rust web crawling, Crawlee-shaped. An idiomatic library for building reliable HTTP, HTML, and browser crawlers.

cargo add millipede --git https://github.com/satvik007/millipede

Once 0.1.0 lands on crates.io, this becomes plain cargo add millipede.

Quick start

Pick a crawler — HTTP, HTML, or browser — and get a crawl running in minutes.

Introduction

A step-by-step course: build your first crawler, follow links, scrape and save data.

Examples

Complete, compiling programs straight from the repository, one page per example.