Your vector search is silently blind to emoji, CJK, SKU codes, and drug names.
Tokenizers map unknown-vocab entities to [UNK], and [UNK]-heavy strings collapse to near-identical embeddings. Result: a reaction log of 🔥😂👍 or a catalog of SKU-XJ2091 becomes semantically indistinguishable garbage to your retriever.
Receipt: default ChromaDB + all-MiniLM-L6-v2 loses 3/3 on a trivial emoji reaction-log retrieval test. Not cherry-picked — 60-second repro included.
Paper + code, pre-registered falsification tests (t2-t35), standalone repro script, killshot demo:
https://github.com/Jaybell31/dreamwalk/tree/master/tokenizer-collapse-paper
"Silent [UNK] Collapse: Tokenizer Vocabulary Gaps Break Text Embedding Retrieval"
Tokenizers map unknown-vocab entities to [UNK], and [UNK]-heavy strings collapse to near-identical embeddings. Result: a reaction log of 🔥😂👍 or a catalog of SKU-XJ2091 becomes semantically indistinguishable garbage to your retriever.
Receipt: default ChromaDB + all-MiniLM-L6-v2 loses 3/3 on a trivial emoji reaction-log retrieval test. Not cherry-picked — 60-second repro included.
Paper + code, pre-registered falsification tests (t2-t35), standalone repro script, killshot demo:
https://github.com/Jaybell31/dreamwalk/tree/master/tokenizer-collapse-paper
"Silent [UNK] Collapse: Tokenizer Vocabulary Gaps Break Text Embedding Retrieval"
1🍆1
