Skip to main content
Configure how Pinecone indexes and matches text in your full-text-search fields. These options are set per field at index creation.

Tokens and analyzers

The word ā€œtokenā€ appears in every scoring method, but it means different things in each. Knowing what counts as a token in your chosen method is essential to writing queries that match what you expect.

FTS tokens (type: "text", type: "query_string", and $match_* filters)

When you declare a field with full_text_search: { ... }, Pinecone runs the field’s text through an analyzer pipeline at index time and at query time. Both type: "text" and type: "query_string" use the same pipeline, and the text-match filter operators ($match_phrase, $match_all, $match_any) reuse it as well, so a token that scores in BM25 will match in a filter on the same field. The pipeline (in order):
  1. Split the text on whitespace and punctuation. Hyphenated words become multiple tokens (state-of-the-art → state, of, the, art).
  2. Lowercase every token. Lowercasing is server-applied and can’t be overridden.
  3. Stem each token to its root form, if stemming is enabled on the field. The stemmer is selected by the field’s language setting (models → model, running → run).
  4. Drop stop words (common words like the, and), if stop_words: true is set on the field. Not all languages have built-in stop word lists; see the Language table for details.
For example, with the english analyzer, stemming: true, and stop_words: false, the input "State-of-the-Art Models" becomes the tokens state, of, the, art, model. Those are the tokens BM25 scores against, and the tokens a $match_phrase: "art models" filter will look for. A single token can be up to 1,024 bytes, measured after splitting and before lowercasing or stemming. Tokens aren’t truncated: an upsert, text query, or query_string query that contains a longer token fails with a 400 error, and a $match_* filter ignores a longer token without an error. This limit is server-applied and can’t be overridden. Fields configured for substring search replace this whole-token pipeline with character n-gram tokenization, so a token is further split into overlapping character sequences. These fields accept tokens over 1,024 bytes at upsert, but a query token over 1,024 bytes still fails with a 400 error.

Dense-vector tokens (type: "dense_vector")

Dense embedding models have their own internal tokenizer, usually a subword scheme like BPE, WordPiece, or SentencePiece, that breaks text into pieces the model was trained on. Those tokens are private to the model. You never query them directly: a dense search compares the full embedding of a query against the full embedding of a document. The same string can therefore behave very differently in type: "text" (which sees the FTS analyzer tokens above) and type: "dense_vector" (which sees a single high-dimensional vector). The $match_* filter operators don’t apply to dense-vector fields.

Sparse-vector tokens (type: "sparse_vector")

Sparse encoders also tokenize internally, and the tokenization depends on the encoder. Pinecone’s hosted pinecone-sparse-english-v0 produces learned per-token weights and expands to related terms that don’t appear in the source text. Encoder tokens aren’t interchangeable with FTS analyzer tokens, and $match_* filters don’t apply to sparse-vector fields.

Practical implication

If your application stores the same source text in an FTS-enabled string field and also encodes it into a dense_vector or sparse_vector field, the three representations are tokenized independently: the FTS analyzer for the string field, and each model’s internal tokenizer for the vector fields. Identical query strings will therefore retrieve different documents under different score_by types, and $match_* filters can only narrow on the FTS-analyzer tokens of FTS-enabled string fields.

Stemming

Stemming reduces words to their root form so that morphological variants match each other. For example, with stemming enabled, a query for ā€œrunā€ also matches documents containing ā€œrunningā€ or ā€œrunsā€. Stemming is opt-in and disabled by default. To enable it, set stemming: true on a text-searchable field when creating the index. The stemming algorithm is determined by the field’s language setting. Stemming applies to both type: "text" and type: "query_string" queries on the field.
Stemming is set at index creation and can’t be changed afterward.

Enable stemming with French

Language

The language parameter controls tokenization and stemming behavior for a text-searchable field. It determines how text is analyzed during indexing and search: how words are split into tokens and, when stemming is enabled, which language-specific rules are used to reduce words to their root forms. The default language is "en" (English). You can specify a language using either its short code or full name (e.g., "fr" or "french").
Language is set at index creation and can’t be changed afterward.

Supported languages

Substring search with n-grams

By default, full-text search matches whole tokens: a query for comp doesn’t match a document containing computer. To match substrings (for example, to find computer from comp, mput, or uter), configure a text field for character n-gram tokenization. With n-gram tokenization, each token is broken into overlapping character sequences (n-grams) at index time, and query text is broken the same way at search time, so a substring of an indexed word matches. This is useful for partial-word matching, autocomplete, and searching identifiers or codes where users type only a fragment. To enable it, set an ngram object on a text field’s full_text_search config at index creation:
ngram parameters: For example, with min_gram: 3, max_gram: 4, and prefix_only: false, the token search is indexed as sea, ear, arc, rch, sear, earc, arch. A shorter or longer window changes the tradeoff: smaller n-grams match more loosely and grow the index more; larger n-grams are more precise but require longer matching substrings. Querying an n-gram field needs no special syntax. Once a field is configured for n-grams, ordinary type: "text" and query_string queries against it match on substrings automatically, because the query text is tokenized into the same n-grams as the indexed text:
Python
N-gram tokenization can’t be combined with stemming or stop_words on the same field — an index-creation request that sets ngram alongside either is rejected with a 400 error. Tokens are always lowercased. Because every position emits a token for each gram length, an n-gram field is larger on disk than a plain text field; keep min_gram/max_gram as narrow as your matching needs allow.
N-gram configuration is set at index creation and can’t be changed afterward.