Rust-based web crawler expands to 14 languages, cuts memory use
Kreuzcrawl, the Rust-powered web crawler, has reached version 0.3.0, expanding its language coverage to fourteen with the addition of Dart, Kotlin for Android, Swift, and Zig while cutting peak streaming memory usage from about 2.5 GB to roughly 20 MB.
A growing polyglot ecosystem
The new release adds four language bindings—Dart, Kotlin for Android, Swift, and Zig—bringing the total to fourteen. All bindings are generated automatically from the Rust core using the alef polyglot binding generator, eliminating manual per-language glue code. Dart and Kotlin bind via the C FFI layer, Swift through clang, and Zig through its C interoperability. The generation pipeline has also been hardened, with native architecture builds replacing QEMU emulation and automated artifact checksums for Swift packages.
Memory efficiency redefined
Previously, streaming operations kept every page result in memory before delivery, leading to peak working sets around 2.5 GB during large crawls. Version 0.3.0 changes ownership semantics so each page is emitted immediately as a CrawlEvent::Page, allowing callers to process and drop it. The engine now only retains in-flight pages, reducing memory usage to roughly 20 MB regardless of crawl size.
Security and stability at the core
The update introduces a tiered, signal-driven dispatch engine that integrates WAF fingerprints from 35 rules across eight vendors, with lock-free hot-reload via ArcSwap and 500 ms debounce. Server-side request forgery defenses are now enabled by default across every outbound call path. With these changes, the project declares its first API-stable release, marking a milestone in reliability and security for web crawling operations.
Source: DEV Community. AI-assisted editorial synthesis — TechnoExpress.