reindexer

Creation

Reindexer supports three types of vector indexes: hnsw (based on HNSWlib), ivf (based on FAISS IVF FLAT) and brute force (vec_bf), and three types of metrics for calculating the measure of similarity of vectors: inner_product, l2 and cosine.

For all types of vector indexes, the metric and the dimension should be explicitly specified. Vectors of only the specified dimension can be inserted and searched in the index.

The initial size start_size can optionally be specified for brute force и hnsw indexes, which helps to avoid reallocation and reindexing. The optimal value is equal to the size of the fully filled index. A much larger start_size value will result in memory overuse, while a much smaller start_size value will slow down inserts. Minimum and default values are 1000.

Automatic embedding of vector indexes is also supported. It is expected that the vector generation service is configured. Its URL is needed, and the basic index fields are specified. Contents of the base fields are passed to the service, and the service returns the calculated vector value.

HNSW options

Hnsw index is also configured with the following parameters:

IVF options

For ivf index, the number of vectors (called centroids) should be specified, that will be chosen to partition the entire set of vectors into clusters. Each vector belongs to the cluster of the centroid closest to itself. The higher the centroids_count, the fewer vectors each cluster contains, this will speed up the search but will slow down the index building.
Required, range of values is [1, 65536]. It is recommended to take values of the order between $4 * \sqrt{number: of: vectors: in: the: index}$ and $16 * \sqrt{number: of: vectors: in: the: index}$.

Examples

// For a vector field, the data type must be array or slice of `float32`.
type Item struct {
	Id      int           `reindex:"id,,pk"`
	// In case of a slice, `dimension` should be explicitly specified in the field tag.
	VecBF   []float32     `reindex:"vec_bf,vec_bf,start_size=1000,metric=l2,dimension=1000"`
	// In case of an array, `dimension` is taken to be equal to the size of the array.
	VecHnsw [2048]float32 `reindex:"vec_hnsw,hnsw,m=16,ef_construction=200,start_size=1000,metric=inner_product,multithreading=1"`
	VecIvf  [1024]float32 `reindex:"vec_ivf,ivf,centroids_count=80,metric=cosine"`
}

When adding a vector index to an existing namespace, the field on which the index will be built must be empty or contain an array of numbers of length equal to the dimension of the index.

ivfOpts := reindexer.FloatVectorIndexOpts {
	Metric:         "l2",
	Dimension:      1024,
	CentroidsCount: 32,
}
indexDef := reindexer.IndexDef {
	Name:       "vec",
	JSONPaths:  []string{"vec"},
	IndexType:  "ivf",
	FieldType:  "float_vector",
	Config:     ivfOpts,
}
err := DB.AddIndex("ns_name", indexDef)
if err != nil {
	panic(err)
}

Embedding configuration

Reindexer is able to perform automatic remote HTTP API calls to receive embedding for documents’ fields or strings in KNN queries conditions. Currently, reindexer’s core simply sends fields/conditions content to external user’s service and expects to receive embedding results.

Embedding service has to implement this openapi spec.

Notice: current embedding callback API is in beta and may be changed in the next releases

To configure automatic embedding you should set config field in the target vector index:

"config": {
  "embedding": {
    "upsert_embedder": {
      "name": <Embedder name>
      "URL": <URL service>,
      "cache_tag": <name, used to access the cache>,
      "fields": [ "idx1", "idx2" ]
      "embedding_strategy": <"always"|"empty_only"|"strict">
      "pool": {
        "connections": 10,
        "connect_timeout_ms": 300,
        "read_timeout_ms": 5000,
        "write_timeout_ms": 5000
      }
    },
    "query_embedder": {
      "name": <Embedder name>
      "URL": <URL service>,
      "cache_tag": <name, used to access the cache>,
      "pool": {
        "connections": 10,
        "connect_timeout_ms": 300,
        "read_timeout_ms": 5000,
        "write_timeout_ms": 5000
      }
    }
  }
}

It is also optionally possible to configure a connection pool:

Upsert embedder used in Insert/Update/Upsert operations. Query embedder starts with WhereKNN, sending a string as the search value. The embedding process: sends JSON values for all fields involved in the embedding to the specified URL. For one requested vector:

{"data":[["field1Val", "field2Val", "field3Val"]]}

Or a batch for several:

{"data":[["fieldVal00", "field01Val", "field02Val"], ..., ["fieldValN0", "fieldN1Val", "fieldN2Val"]]}

embeddingPoolOpts := reindexer.EmbedderConnectionPool{
	Connections:    20,
	ConnectTimeout: 100,
}
upsertEmbedderOpts := reindexer.UpsertEmbedder{
	Name:                   "UpsertEmbedder",
	URL:                    "http://127.0.0.1:7777/embedder",
	Fields:                 []string{"strIdxField1", "strIdxField2"},
	EmbeddingStrategy:      "empty_only",
	EmbedderConnectionPool: embeddingPoolOpts,
}
queryEmbedderOpts := DefaultQueryEmbedderConfig("http://127.0.0.1:7777/embedder")
embeddingOpts := reindexer.Embedding{
	UpsertEmbedder: upsertEmbedderOpts,
	QueryEmbedder: queryEmbedderOpts,
}
hnswSTOpts := reindexer.FloatVectorIndexOpts{
	Metric:             "inner_product",
	Dimension:          1024,
	M:                  32,
	EfConstruction:     100,
	StartSize:          200,
	MultithreadingMode: 0,
	Embedding:          embeddingOpts,
}
indexDef = reindexer.IndexDef{
	Name:      "vec",
	JSONPaths: []string{"vec"},
	IndexType: "hnsw",
	FieldType: "float_vector",
	Config:    hnswSTOpts,
}
err := DB.AddIndex("ns_name", indexDef)
if err != nil {
	panic(err)
}

Embedding cache configuration

When do embedding, it makes sense to use result caching, which can improve performance. To do this, you need to configure it. Setting up embedding caches is done in two places. Firstly, cache_tag parameter that was described above. cache_tag is part of config field in the target vector index description. cache_tag it’s simple name\identifier, used to access the cache. Optional, if not specified, caching is not used. The name may not be unique. In this case, different embedders can put the result in the same cache. But keep in mind that this only works well if the source data for the embedder does not overlap. Or if the embedders return exactly the same values for the same request. Secondly, special item in system #config namespace, with type embedders. Optional, if not specified, caching is not used for all embedders.

{
  "type":"embedders",
  "caches":[
    {
      "cache_tag":"*",
      "max_cache_items":1000000,
      "hit_to_cache":1
    },
    {
      "cache_tag":"the jungle book",
      "max_cache_items":2025,
      "hit_to_cache":3
    }
  ]
}

Float vector fields in selection results

By default, float vector fields are excluded from the results of all queries to namespaces containing vector indexes. If you need to get float vector fields, you should specify this explicitly in the query. Either by listing the required fields, or by requesting the output of all vectors().

Supported filtering operations on floating-point vector fields are KNN, Empty, and Any. It is not possible to use multiple KNN filters in a query, and it is impossible to combine filtering by KNN and fulltext.

Parameters set for KNN-query depends on the specific index type. The only required parameter for KNN is k - the maximum number of documents returned from the index for subsequent filtering.

When searching by hnsw index, you can additionally specify the ef parameter. Increasing this parameter allows you to get a higher quality result (recall rate), but at the same time slows down the search. See description here. Optional, minimum and default values are k.

When searching by ivf index, you can additionally specify the nprobe parameter - the number of clusters to be looked at during the search. Increasing this parameter allows you to get a higher quality result (recall rate), but at the same time slows down the search. Optional, should be greater than 0, default value is 1.

KNN search with auto-embedding

KNN search with automatic embedding works much like simple KNN search. With one exception, it expects a string instead of a vector. It then sends that string to the embedding service, gets back a calculated vector. And that vector is then used as the actual value for filtering the database. Before this need to configure “query_embedder”.

Rank

By default, the results of queries with KNN are sorted by rank, that is equal to requested metric values. For indexes with the l2 metric, from a lower value to a higher value, and with the inner_product and cosine metrics, from a higher value to a lower value. This is consistent with the best match for the specified metrics.

When it is necessary to get the rank-value of each document in the query result, it must be requested explicitly via the RANK() function in SQL or WithRank() in GO:

SELECT *, RANK() FROM test_ns WHERE KNN(vec_bf, [2.4, 3.5, ...], k=200)

knnBaseSearchParams, err := reindexer.NewBaseKnnSearchParam(200)
if err != nil {
	panic(err)
}
db.Query("test_ns").WithRank().WhereKnn("vec_bf", []float32{2.4, 3.5, ...}, knnBaseSearchParams)

Result:

{"id": 0, "rank()": 1.245}

rank can also be used to sort by expression. Described in detail here.

Query examples

Environment variables affecting vector indexes

Additional action commands

These commands can be used by inserting them via upsert into the #config namespace.

Rebuilding clusters for IVF index

{"type":"action","action":{"command":"rebuild_ivf_index", "namespace":"*", "index":"*", "data_part": 0.5}}

The command can be useful for cases where the composition of vectors in the index has changed significantly and the current centroids do not provide sufficiently high-quality output.

Removing disk cache for ANN indexes

{"type":"action","action":{"command":"drop_ann_storage_cache", "namespace":"*", "index":"*"}}

The command can be useful for cases when you need to force the re-creation of the disk cache for ANN indexes, or disable it completely (using it together with the RX_DISABLE_ANN_CACHE environment variable).

Removing disk cache for embedders

The command can be useful in cases where it is necessary to reset cached content for all or a specific embedder

This site is open source. Improve this page.