From the source
Mapping text to large taxonomies of 100,000+ labels, whether that's biomedical entity linking, vendor normalization, or company deduplication, is a common production problem where regex, trained classifiers, and direct LLM calls all struggle on cost, maintenance, and context limits.
Our solution pairs vector search with the Databricks AI Classify function: retrieve a shortlist of candidate labels per document, then let the AI Classify function pick from that shortlist instead of the full taxonomy.
Across three benchmarks spanning these use cases, SQL-native vector search plus AI Classify beat the best cost-efficient frontier model by five points of accuracy at roughly a hundredth of the token cost.
Across Databricks, thousands of customers build production workloads that map freeform text to normalized taxonomies of 100k+ labels.
A few common use cases include: Biomedical entity linking.
Clinical notes and research papers mention diseases, drugs, and procedures that must be matched to a concept in the Unified Medical Language System , a vocabulary with thousands of biomedical concept identifiers.
Vendor normalization.
…
