# ANNex benchmarks

All claims are governed by the benchmark policy (https://github.com/rosharma719/ANNex/blob/master/BENCHMARK_POLICY.md). No result here establishes production superiority or launch readiness.

## Retrieval quality: BEIR nDCG@10

BEIR nDCG@10: ANNex vs LanceDB and Qdrant on five held-out corpora. Shared pinned MiniLM vectors, exact cosine, 100 candidates per channel. Every per-corpus 95% bootstrap interval between hybrid systems includes zero: parity, not a statistically established advantage.
Implementation f68f970, Apple M2 / 16 GiB, 2026-09-29.

| System | NFCorpus (162q) | SciFact (150q) | ArguAna (703q) | FiQA (324q) | SCIDOCS (500q) | Macro |
|---|---|---|---|---|---|---|
| ANNex hybrid | 0.3574 | 0.7264 | 0.5238 | 0.3798 | 0.2133 | 0.4401 |
| LanceDB native hybrid | 0.3543 | 0.7276 | 0.5265 | 0.3784 | 0.2132 | 0.4400 |
| Qdrant Server hybrid | 0.3580 | 0.7257 | 0.5207 | 0.3798 | 0.2136 | 0.4396 |
| ANNex dense | 0.3149 | 0.6642 | 0.4830 | 0.3773 | 0.2243 | 0.4127 |
| ANNex BM25 | 0.3285 | 0.6741 | 0.4668 | 0.2435 | 0.1625 | 0.3751 |

## Head-to-head: NYT256

ANNex vs FAISS, hnswlib and USearch at three recall tiers (rows grouped by recall: ~0.86, ~0.92, ~0.96).
NYT256, 290K docs, 256-D angular, Apple M2, offset=4000.
Versions: ANNex 0.1.0, FAISS 1.15.1, hnswlib 0.8.0, USearch 2.26.2. Interleaved run: all engines share OS memory state, so per-engine timing is realistic but not fully isolated.

| System | ef | Recall | p50 ms | p99 ms |
|---|---|---|---|---|
| annex_screen | 32 | 0.864 | 0.252 | 1.042 |
| faiss_hnsw16 | 128 | 0.868 | 0.356 | 0.598 |
| faiss_hnsw32 | 32 | 0.834 | 0.208 | 0.373 |
| usearch_f16 | 128 | 0.864 | 0.540 | 0.941 |
| hnswlib_m16 | 128 | 0.861 | 0.684 | 1.127 |
| annex_screen | 128 | 0.918 | 0.580 | 0.972 |
| faiss_hnsw16 | 512 | 0.925 | 1.457 | 2.758 |
| faiss_hnsw32 | 128 | 0.904 | 0.627 | 1.009 |
| usearch_f16 | 512 | 0.922 | 2.059 | 4.246 |
| hnswlib_m16 | 512 | 0.918 | 2.562 | 7.010 |
| annex_screen | 512 | 0.961 | 2.205 | 5.990 |
| faiss_hnsw32 | 512 | 0.957 | 2.489 | 4.344 |
| faiss_ivf1024 | 512 | 0.964 | 3.753 | 9.103 |
| usearch_f32 | 1024 | 0.948 | 6.261 | 9.592 |
| hnswlib_m16 | 1024 | 0.946 | 5.043 | 10.500 |

## NYT256 recall–latency Pareto

Fastest ANNex config at each recall threshold vs the hnswlib baseline (hnswlib 0.8.0). Offset 1000 excludes query vectors from the corpus. The 0.97 row is where hnswlib takes the Pareto front.
Apple M2 / 16 GiB, 3 rounds, commit 4b11157.

| Recall target | Config | ef | Recall | p50 ms | p99 ms | Note |
|---|---|---|---|---|---|---|
| 0.80 | combined_stop0.01 | 32 | 0.827 | 0.225 | 0.517 |  |
| 0.85 | pq_screen0.1 | 128 | 0.871 | 0.300 | 0.564 |  |
| 0.87 | pq_screen0.1 | 128 | 0.871 | 0.300 | 0.564 |  |
| 0.90 | pq_screen0.15 | 128 | 0.903 | 0.466 | 1.122 |  |
| 0.92 | pq_screen0.1 | 256 | 0.925 | 0.629 | 1.108 |  |
| 0.94 | pq_screen0.1 | 512 | 0.956 | 1.412 | 2.219 |  |
| 0.96 | pq_screen0.15 | 512 | 0.961 | 2.699 | 4.239 |  |
| 0.97 | hnswlib | 2048 | 0.971 | 9.162 | 11.065 | hnswlib wins Pareto here |

## EC2 throughput, top-k=10

sq8 = SQ8 quantization; rcm = residual compression map. Serial requests, no warmup. Timing excludes embedding generation.
NYT256, 290K docs, 256-D angular, EC2, 2026-10-01.

| Config | ef | Recall | p50 ms | p99 ms | QPS |
|---|---|---|---|---|---|
| m16 | 32 | 0.836 | 0.203 | 0.331 | 4706 |
| m16+sq8 | 32 | 0.836 | 0.182 | 0.282 | 5324 |
| m16+rcm | 32 | 0.836 | 0.202 | 0.326 | 4763 |
| m16+rcm+sq8 | 32 | 0.836 | 0.179 | 0.282 | 5424 |
| m16 | 128 | 0.902 | 0.559 | 0.894 | 1769 |
| m16+sq8 | 128 | 0.902 | 0.500 | 0.792 | 1971 |
| m16+rcm+sq8 | 128 | 0.902 | 0.484 | 0.765 | 2037 |
| m16+sq8 | 256 | 0.925 | 0.914 | 1.313 | 1104 |
| m16+rcm+sq8 | 256 | 0.925 | 0.897 | 1.308 | 1127 |

## Methodology

- Recall is measured against exact nearest neighbors computed independently. All ANNex runs use 3 rounds; reported latencies are medians.
- Timing measures end-to-end search latency only. Embedding generation, index build, training and recovery are excluded.
- nDCG@10 uses linear relevance gains matching BEIR's evaluator and trec_eval, verified against pytrec_eval within 1e-12.
- Interleaved runs share OS memory state between engines: realistic, but not fully isolated per-engine timing.

Corrections history: https://github.com/rosharma719/ANNex/blob/master/crates/annex-multivector/benchmark/RESULTS.md

## Download the data

- https://annexsearch.dev/data/benchmarks.json: All benchmark tables with full provenance (JSON)
- https://annexsearch.dev/data/nyt256-pareto.csv: NYT256 recall–latency Pareto (CSV)
- https://annexsearch.dev/data/nyt256-head-to-head.csv: NYT256 head-to-head vs FAISS, hnswlib, USearch (CSV)
- https://annexsearch.dev/data/nyt256-ec2-topk10.csv: NYT256 throughput on EC2, top-k=10 (CSV)
- https://annexsearch.dev/data/beir-ndcg10.csv: BEIR nDCG@10 vs LanceDB and Qdrant (CSV)
