Scale up
indice is built to run the same way on a laptop with a handful of WACZs and on a server holding an institution’s collections: one binary, an embedded index, no Solr/Elasticsearch cluster to operate. Two things make that hold up as a corpus grows: a frugal, predictable on-disk footprint and query latency that grows sub-linearly.
Footprint
Section titled “Footprint”Only a capped prefix of each page’s body is stored (for snippets); the full text is indexed for search but not stored, so the doc store stays bounded. On a real corpus that works out to ~1.7 KB per document. Two commands make the footprint legible:
indice stats # size by index file type, bytes/doc, projections to 1M/100M docsindice stats --fields # what fills the doc store, by stored fieldAt ~1.7 KB/doc the index projects to roughly 1.6 GB at 1M docs and ~160 GB at 100M. If that’s still too much, the stored_body_cap_kb knob (see Operator configuration) trades snippet depth for a smaller store: a laptop keeps generous snippets, an institution dials it down.
Query latency
Section titled “Query latency”Search stays interactive as the corpus grows because the result-grouping and snippet work is bounded (a fixed candidate window), not proportional to the corpus. Measured on a laptop with the scale_bench example (single binary, one process):
| corpus | full-text search | facet overview | collection page lookup |
|---|---|---|---|
| 100k docs | ~20 ms | ~2.5 ms | ~1.5 ms |
| 1M docs | ~40 ms | ~16 ms | ~1.5 ms |
10× the data costs roughly 2× the search latency; everything stays well under 100 ms at 1M docs. Indexed lookups (opening a collection, resolving a URL for replay) are corpus-size-independent. You can reproduce with:
cargo run --release --example scale_bench -- 1000000 # docs, [iters], [segments]Index health
Section titled “Index health”A search fans out across every segment, so a fragmented index is a slow one. indice keeps the index compact automatically: index and the management-mode import auto-compact when they detect fragmentation, serve warns at startup if the index it opens is fragmented, and indice optimize merges segments on demand, which reclaims disk from deleted crawls and sweeps orphaned segment files left behind by an interrupted (Ctrl-C’d) run. So an interrupted ingest self-heals on the next index.