WikiClaw turns any MediaWiki site into a clean, structured corpus
WikiClaw turns any MediaWiki site into a clean, structured corpus. Paste a URL, pick a mode, run. Output is JSON Lines (Markdown, infoboxes, categories, and heading-aware chunks) ready for any AI pipeline, vector store, or archive workflow. Built for people who want a wiki scraped but don't want to learn shell scripting, Docker, or a Python toolchain. WHO IT'S FOR • Hobbyist AI builders making a chatbot from a wiki they care about • Researchers and archivists preserving community wikis • Wiki maintainers wanting offline backups with full attribution • Anyone migrating content out of a MediaWiki install WHAT YOU GET Every crawl writes a clean folder of JSON Lines files you can reveal in Finder: • pages.jsonl: one record per page, with HTML, Markdown, infobox, content hash, and token estimate • chunks.jsonl: RAG-ready chunks with section breadcrumbs • categories.json: full category-to-members map • assets_manifest.json: image and file index, with optional binary download • site_manifest.json: run metadata, timestamps, and license attribution • errors.jsonl: per-page failure records TWO MODES, ONE SWITCH Pick once, the right defaults follow: • For AI / RAG: chunks ready for embedding, no raw HTML, no image binaries. The default. • Full archive: saves raw HTML and downloads every image and file. Bigger, slower, complete. KNOWS WHAT'S A WIKI BEFORE YOU DO Paste a URL and WikiClaw verifies it's actually a MediaWiki install, then shows you the wiki's name and page count before you start. No more letting a crawl run for an hour to find out it was the wrong URL. SEE EVERY STEP Plain-English progress. Looking through pages. Saving each page. Downloading images. Wrapping up. No cryptic phases, no surprises three hours into a long crawl. PREVIEW BEFORE YOU COMMIT One-page preview: paste a single title, see exactly what the JSONL output looks like. Catch parsing issues in two seconds, not three hours. MORE • Resume from checkpoint after cancellation • Per-host rate limiting so you stay polite to small wikis • Sync mode auto-appears when you've crawled before, re-fetching only changed pages • Filter mode for title pattern, prefix list, or explicit allow-list • Custom User-Agent string • Light or dark mode, follows your system PRIVACY WikiClaw runs entirely on your Mac. No account, no telemetry, no cloud sync. Your crawled data lives in the app's sandbox folder and never leaves your machine. You are responsible for following each source wiki's license and rate-limit policies.
| 序号 | 名称 | 价格 | 货币 | 地区 |
|---|---|---|---|---|
| 1 | 项目 | ¥ 0 | CNY | 中国 |
