site-mirror

A CLI tool to mirror websites for offline browsing using Playwright.

Installation

# Install globally
npm install -g site-mirror

# Or use directly via npx
npx site-mirror --help

Quick Start

# Download a single page with all its assets (no config needed!)
site-mirror run --start https://www.apple.com/iphone/ --singlePage

# Crawl an entire site
site-mirror run --start https://example.com/

# Or use interactive config-based workflow:
site-mirror init          # Interactive prompts to create site-mirror.config.json
site-mirror run           # Runs the mirror using config
site-mirror serve         # Serve locally on port 8080

Commands

Command	Description
site-mirror init	Interactive setup - creates site-mirror.config.json
site-mirror run	Run the mirror (reads config + CLI overrides)
site-mirror serve	Serve the ./offline folder locally
site-mirror serve 3000	Serve on a custom port

CLI Options (for run)

Option	Description	Default
--start <url>	Start URL (required if not in config)	-
--out <dir>	Output directory	./offline
--maxPages <n>	Max pages to crawl (0 = unlimited)	0
--maxDepth <n>	Max link depth (0 = unlimited)	0
--sameOriginOnly	Only crawl same-origin pages	true
--seedSitemaps	Seed URLs from sitemap.xml/robots.txt	false
--singlePage	Download only this page + all its assets	false

Config File (site-mirror.config.json)

Created via site-mirror init (interactive) or manually:

{
  "start": "https://example.com/",
  "out": "./offline",
  "singlePage": false,
  "maxPages": 200,
  "maxDepth": 6,
  "sameOriginOnly": true,
  "seedSitemaps": false
}

CLI options override config file settings.

Output Structure

./offline/
├── index.html              # Homepage
├── about/
│   └── index.html          # /about/ page
├── _next/                   # Same-origin assets
│   └── static/
├── _external/               # Cross-origin assets
│   └── cdn.example.com/
│       └── script.js

How It Works

1. Launches headless Chromium via Playwright
2. Navigates to each page, waits for network idle
3. Captures all static assets (CSS, JS, images, fonts, videos)
4. Rewrites absolute same-origin URLs to relative paths
5. Injects a script to handle SPA-style navigation offline
6. Discovers new pages via <a href> links
7. Saves everything to the output directory

Notes

• XHR/fetch API responses are not saved (only rendered HTML + static assets)
• Some interactive features requiring live APIs won't work offline
• Be mindful of target site's Terms of Service and robots.txt

License

MIT