Indexing Data

Optimize search performance with LanceDB using vector indexes, full-text search, scalar indexes, and more.

An index is a data structure that facilitates efficient scans and lookups on the embeddings of a given dataset. LanceDB provides a comprehensive suite of indexes to optimize query performance across diverse workloads:

• Vector Index: Efficiently searches for similar vectors across high-dimensional data (e.g. images, audio, or text embeddings)
• Full-Text Search Index: Enables fast keyword-based searches by indexing words and phrases
• Scalar Index: Accelerates filtering and sorting of structured numeric or categorical data (e.g. timestamps, prices)

Supported Indexes

LanceDB provides a comprehensive suite of indexes for different use cases and data types:

Index	Use Case	Description
IVF (Vector)	Large-scale vector search with configurable accuracy/speed trade-offs. Supports binary vectors with hamming distance.	Inverted File Index—a partition-based approximate nearest neighbor algorithm that groups similar vectors into partitions for efficient search. Distance metrics: $\ell_2$ cosine dot hamming Quantizations: None/Flat PQ SQ RQ
IVF_HNSW (Vector)	Large-scale vector search requiring both high recall and efficient partitioning. Combines the scalability of IVF with the search quality of HNSW.	Hybrid index combining IVF partitioning with HNSW graphs built within each partition. Provides improved search quality over pure IVF while maintaining scalability. Distance metrics: $\ell_2$ cosine dot Quantizations: None/Flat SQ PQ
FTS (Full-text search)	String columns (e.g., title, description, content) requiring keyword-based search with BM25 ranking.	Full-text search index using BM25 ranking algorithm. Tokenizes text with configurable tokenization, stemming, stop word removal, and language-specific processing.
BTree (Scalar)	Numeric, temporal, and string columns with mostly distinct values. Best for selective equality, inequality, and range predicates.	Sorted index storing sorted copies of scalar columns with block headers in a btree cache. Header entries map to blocks of rows (4096 rows per block) for efficient disk reads.
Bitmap (Scalar)	Low-cardinality columns with few thousand or fewer distinct values. Accelerates equality and range filters.	Stores a bitmap for each distinct value in the column, with one bit per row indicating presence. Memory-efficient for low-cardinality data.
LabelList (Scalar)	List columns (e.g., tags, categories, keywords) requiring array_contains_all or array_contains_any filters.	Scalar index for List<T> and LargeList<T> columns of primitive values, using an underlying bitmap index structure to enable fast array membership lookups.
FM (Scalar)	String or binary columns that need raw substring search.	FM-Index over Utf8, LargeUtf8, Binary, or LargeBinary data for filters such as contains(path, 'needle'). Use FTS instead for tokenized word search and BM25 ranking.

TypeScript currently doesn’t support IvfSq (IVF with Scalar Quantization).

Quantization

LanceDB also supports several different quantization methods, used by vector indexes to compress vectors and improve search performance:

Quantization	Use Case	Description
PQ (Product Quantization)	Default choice for most vector search scenarios. Use when you need to balance index size and recall.	Divides vectors into subvectors and quantizes each subvector independently. Provides a good balance between compression ratio and search accuracy.
SQ (Scalar Quantization)	Use when you need faster indexing or when vector dimensions have consistent value ranges.	Quantizes each dimension independently. Simpler than PQ but typically provides less compression.
RQ (RabitQ Quantization)	Use when you need maximum compression or have specific per-dimension requirements.	Per-dimension quantization using a RabitQ codebook. Provides fine-grained control over compression per dimension. For IVF_RQ, vector dimensions must be divisible by 8.
None/Flat	Use for binary vectors (with hamming distance) or when you need maximum recall and have sufficient storage.	No quantization—stores raw vectors. Provides the highest accuracy but requires more storage and memory.