Skip to content
indice

Importing from Browsertrix

If your WACZs live in a Browsertrix account (Webrecorder’s hosted crawler), indice import browsertrix downloads them into <home>/archive and indexes them as durable local sources.

At the moment an indice server can only be associated with one Browsertrix account. Credentials come from the environment, never the command line, so they don’t show up in the process list:

Terminal window
export BROWSERTRIX_USER='you@example.org'
read -rs BROWSERTRIX_PASSWORD; export BROWSERTRIX_PASSWORD   # prompts, no echo
# or, instead of user/password:  export BROWSERTRIX_TOKEN='<a JWT>'

Then import, or preview first with --dry-run, and then pull for real:

Terminal window
# everything you've QA'd in your (only) org, into ~/webarchive
indice import browsertrix --home ~/webarchive --dry-run
indice import browsertrix --home ~/webarchive


# just one collection (by id, slug, or name) → a matching indice collection
indice import browsertrix --collection us-govarchive --home ~/webarchive


# a single crawl
indice import browsertrix --crawl <item-id> --home ~/webarchive

Notes

Section titled “Notes”
  • QA’d crawls only, by default. Browsertrix lets a reviewer rate a crawl (reviewStatus); indice imports only reviewed crawls so you publish vetted content. Add --include-unreviewed to import everything, or --min-review <1-5> for a rating threshold. A single named --crawl is always imported. When crawls are skipped for this reason, indice says so.
  • Selection. --collection <ID|SLUG|NAME> limits to one Browsertrix collection; --crawl <ID> to a single archived item; neither imports the whole org. --org <SLUG> picks the org when your account has more than one.
  • Incremental. Re-running skips crawls already imported (matched by content hash), so syncing an account is cheap; --force re-imports anyway.
  • Durable by default. WACZs are downloaded into <home>/archive/<item-id>/ (a subfolder per Browsertrix item, so items can’t clash on a shared filename), because Browsertrix’s presigned URLs expire after ~48h, so a downloaded copy keeps replay working long-term. --host <URL> targets a self-hosted Browsertrix (default is https://app.browsertrix.com).
  • --stream (index-only footprint). Instead of downloading, index the WACZ in place from Browsertrix and store only its stable identity, not a copy. Since presigned URLs expire, indice re-resolves a fresh one on demand. So serve needs the same BROWSERTRIX_* credentials to replay these crawls (they show a 503 otherwise). Good for a self-hoster who wants search over their own crawls without keeping the bytes; download (the default) is better for a durable, offline, or shared library.
  • Grouping. Importing a --collection groups its crawls into an indice collection of the same name. --into <NAME> overrides that name (and is the way to group an org-wide or single---crawl import, which otherwise land as individual collections). --limit <N> caps how many are imported; --dry-run lists them without downloading.

Both importers are also available in management mode as browse-and-import wizards on the accession desk, using the server’s configured credentials.