SIGN IN SIGN UP

fix(search): pure-JS model2vec embedder — semantic tier works in the compiled binary

The bun-compiled binary could load neither onnxruntime-node's napi binding
nor sharp (transformers.js's eager import), so getEmbedder() returned null
and every compiled install silently lost the semantic tier at setup.

Replace the transformers.js+ONNX stack on the default path with a pure-JS
model2vec backend over the same pinned artifacts: WordPiece/BertNormalizer
tokenizer from tokenizer.json, embedding matrix parsed straight out of the
model.onnx protobuf, mean-pool + L2-normalize. Verified token-identical and
numerically bit-close (max diff 6e-8, zero sign flips) against the
transformers.js reference. Adds an int8 M2VQ8 matrix codec (roundtrip:
mean cosine 0.999968, 100% top-10 ranking parity on 3000 real docs) and a
direct pin-verified HF fetcher for missing model files. transformers.js
remains for the gated feature-extraction backends, with a sharp stub
fallback. index-embeddings now reports the real failure reason and gained
updated_at-based incrementality (embed_indexed_at stamp, scan-start
semantics like the body index).
G
Gigi committed
f7b642cef59f2e255eb67a8bb3c4bc48ebbc3821
Parent: 0759d8d