{
  "id": "vectorizers",
  "title": "Vectorizers",
  "url": "https://redis.io/docs/latest/develop/ai/redisvl/0.9.1/user_guide/vectorizers/",
  "summary": "",
  "content": "\n\nIn this notebook, we will show how to use RedisVL to create embeddings using the built-in text embedding vectorizers. Today RedisVL supports:\n1. OpenAI\n2. HuggingFace\n3. Vertex AI\n4. Cohere\n5. Mistral AI\n6. Amazon Bedrock\n7. Bringing your own vectorizer\n8. VoyageAI\n\nBefore running this notebook, be sure to\n1. Have installed ``redisvl`` and have that environment active for this notebook.\n2. Have a running Redis Stack instance with RediSearch \u003e 2.4 active.\n\nFor example, you can run Redis Stack locally with Docker:\n\n```bash\ndocker run -d -p 6379:6379 -p 8001:8001 redis/redis-stack:latest\n```\n\nThis will run Redis on port 6379 and RedisInsight at http://localhost:8001.\n\n\n```python\n# import necessary modules\nimport os\n```\n\n## Creating Text Embeddings\n\nThis example will show how to create an embedding from 3 simple sentences with a number of different text vectorizers in RedisVL.\n\n- \"That is a happy dog\"\n- \"That is a happy person\"\n- \"Today is a nice day\"\n\n\n### OpenAI\n\nThe ``OpenAITextVectorizer`` makes it simple to use RedisVL with the embeddings models at OpenAI. For this you will need to install ``openai``. \n\n```bash\npip install openai\n```\n\n\n\n```python\nimport getpass\n\n# setup the API Key\napi_key = os.environ.get(\"OPENAI_API_KEY\") or getpass.getpass(\"Enter your OpenAI API key: \")\n```\n\n\n```python\nfrom redisvl.utils.vectorize import OpenAITextVectorizer\n\n# create a vectorizer\noai = OpenAITextVectorizer(\n    model=\"text-embedding-ada-002\",\n    api_config={\"api_key\": api_key},\n)\n\ntest = oai.embed(\"This is a test sentence.\")\nprint(\"Vector dimensions: \", len(test))\ntest[:10]\n```\n\n    Vector dimensions:  1536\n\n\n\n\n\n    [-0.0011391325388103724,\n     -0.003206387162208557,\n     0.002380132209509611,\n     -0.004501554183661938,\n     -0.010328996926546097,\n     0.012922565452754498,\n     -0.005491119809448719,\n     -0.0029864837415516376,\n     -0.007327961269766092,\n     -0.03365817293524742]\n\n\n\n\n```python\n# Create many embeddings at once\nsentences = [\n    \"That is a happy dog\",\n    \"That is a happy person\",\n    \"Today is a sunny day\"\n]\n\nembeddings = oai.embed_many(sentences)\nembeddings[0][:10]\n```\n\n\n\n\n    [-0.017466850578784943,\n     1.8471690054866485e-05,\n     0.00129731057677418,\n     -0.02555876597762108,\n     -0.019842341542243958,\n     0.01603139191865921,\n     -0.0037347301840782166,\n     0.0009670283179730177,\n     0.006618348415941,\n     -0.02497442066669464]\n\n\n\n\n```python\n# openai also supports asynchronous requests, which we can use to speed up the vectorization process.\nembeddings = await oai.aembed_many(sentences)\nprint(\"Number of Embeddings:\", len(embeddings))\n\n```\n\n    Number of Embeddings: 3\n\n\n### Azure OpenAI\n\nThe ``AzureOpenAITextVectorizer`` is a variation of the OpenAI vectorizer that calls OpenAI models within Azure. If you've already installed ``openai``, then you're ready to use Azure OpenAI.\n\nThe only practical difference between OpenAI and Azure OpenAI is the variables required to call the API.\n\n\n```python\n# additionally to the API Key, setup the API endpoint and version\napi_key = os.environ.get(\"AZURE_OPENAI_API_KEY\") or getpass.getpass(\"Enter your AzureOpenAI API key: \")\napi_version = os.environ.get(\"OPENAI_API_VERSION\") or getpass.getpass(\"Enter your AzureOpenAI API version: \")\nazure_endpoint = os.environ.get(\"AZURE_OPENAI_ENDPOINT\") or getpass.getpass(\"Enter your AzureOpenAI API endpoint: \")\ndeployment_name = os.environ.get(\"AZURE_OPENAI_DEPLOYMENT_NAME\", \"text-embedding-ada-002\")\n\n```\n\n\n```python\nfrom redisvl.utils.vectorize import AzureOpenAITextVectorizer\n\n# create a vectorizer\naz_oai = AzureOpenAITextVectorizer(\n    model=deployment_name, # Must be your CUSTOM deployment name\n    api_config={\n        \"api_key\": api_key,\n        \"api_version\": api_version,\n        \"azure_endpoint\": azure_endpoint\n    },\n)\n\ntest = az_oai.embed(\"This is a test sentence.\")\nprint(\"Vector dimensions: \", len(test))\ntest[:10]\n```\n\n\n    ---------------------------------------------------------------------------\n\n    ValueError                                Traceback (most recent call last)\n\n    Cell In[7], line 4\n          1 from redisvl.utils.vectorize import AzureOpenAITextVectorizer\n          3 # create a vectorizer\n    ----\u003e 4 az_oai = AzureOpenAITextVectorizer(\n          5     model=deployment_name, # Must be your CUSTOM deployment name\n          6     api_config={\n          7         \"api_key\": api_key,\n          8         \"api_version\": api_version,\n          9         \"azure_endpoint\": azure_endpoint\n         10     },\n         11 )\n         13 test = az_oai.embed(\"This is a test sentence.\")\n         14 print(\"Vector dimensions: \", len(test))\n\n\n    File ~/src/redis-vl-python/redisvl/utils/vectorize/text/azureopenai.py:78, in AzureOpenAITextVectorizer.__init__(self, model, api_config, dtype)\n         54 def __init__(\n         55     self,\n         56     model: str = \"text-embedding-ada-002\",\n         57     api_config: Optional[Dict] = None,\n         58     dtype: str = \"float32\",\n         59 ):\n         60     \"\"\"Initialize the AzureOpenAI vectorizer.\n         61 \n         62     Args:\n       (...)\n         76         ValueError: If an invalid dtype is provided.\n         77     \"\"\"\n    ---\u003e 78     self._initialize_clients(api_config)\n         79     super().__init__(model=model, dims=self._set_model_dims(model), dtype=dtype)\n\n\n    File ~/src/redis-vl-python/redisvl/utils/vectorize/text/azureopenai.py:106, in AzureOpenAITextVectorizer._initialize_clients(self, api_config)\n         99 azure_endpoint = (\n        100     api_config.pop(\"azure_endpoint\")\n        101     if api_config\n        102     else os.getenv(\"AZURE_OPENAI_ENDPOINT\")\n        103 )\n        105 if not azure_endpoint:\n    --\u003e 106     raise ValueError(\n        107         \"AzureOpenAI API endpoint is required. \"\n        108         \"Provide it in api_config or set the AZURE_OPENAI_ENDPOINT\\\n        109             environment variable.\"\n        110     )\n        112 api_version = (\n        113     api_config.pop(\"api_version\")\n        114     if api_config\n        115     else os.getenv(\"OPENAI_API_VERSION\")\n        116 )\n        118 if not api_version:\n\n\n    ValueError: AzureOpenAI API endpoint is required. Provide it in api_config or set the AZURE_OPENAI_ENDPOINT                    environment variable.\n\n\n\n```python\n# Just like OpenAI, AzureOpenAI supports batching embeddings and asynchronous requests.\nsentences = [\n    \"That is a happy dog\",\n    \"That is a happy person\",\n    \"Today is a sunny day\"\n]\n\nembeddings = await az_oai.aembed_many(sentences)\nembeddings[0][:10]\n```\n\n### Huggingface\n\n[Huggingface](https://huggingface.co/models) is a popular NLP platform that has a number of pre-trained models you can use off the shelf. RedisVL supports using Huggingface \"Sentence Transformers\" to create embeddings from text. To use Huggingface, you will need to install the ``sentence-transformers`` library.\n\n```bash\npip install sentence-transformers\n```\n\n\n```python\nos.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\nfrom redisvl.utils.vectorize import HFTextVectorizer\n\n\n# create a vectorizer\n# choose your model from the huggingface website\nhf = HFTextVectorizer(model=\"sentence-transformers/all-mpnet-base-v2\")\n\n# embed a sentence\ntest = hf.embed(\"This is a test sentence.\")\ntest[:10]\n```\n\n\n```python\n# You can also create many embeddings at once\nembeddings = hf.embed_many(sentences, as_buffer=True)\n\n```\n\n### VertexAI\n\n[VertexAI](https://cloud.google.com/vertex-ai/docs/generative-ai/embeddings/get-text-embeddings) is GCP's fully-featured AI platform including a number of pretrained LLMs. RedisVL supports using VertexAI to create embeddings from these models. To use VertexAI, you will first need to install the ``google-cloud-aiplatform`` library.\n\n```bash\npip install google-cloud-aiplatform\u003e=1.26\n```\n\n1. Then you need to gain access to a [Google Cloud Project](https://cloud.google.com/gcp?hl=en) and provide [access to credentials](https://cloud.google.com/docs/authentication/application-default-credentials). This is accomplished by setting the `GOOGLE_APPLICATION_CREDENTIALS` environment variable pointing to the path of a JSON key file downloaded from your service account on GCP.\n2. Lastly, you need to find your [project ID](https://support.google.com/googleapi/answer/7014113?hl=en) and [geographic region for VertexAI](https://cloud.google.com/vertex-ai/docs/general/locations).\n\n\n**Make sure the following env vars are set:**\n\n```\nGOOGLE_APPLICATION_CREDENTIALS=\u003cpath to your gcp JSON creds\u003e\nGCP_PROJECT_ID=\u003cyour gcp project id\u003e\nGCP_LOCATION=\u003cyour gcp geo region for vertex ai\u003e\n```\n\n\n```python\nfrom redisvl.utils.vectorize import VertexAITextVectorizer\n\n\n# create a vectorizer\nvtx = VertexAITextVectorizer(api_config={\n    \"project_id\": os.environ.get(\"GCP_PROJECT_ID\") or getpass.getpass(\"Enter your GCP Project ID: \"),\n    \"location\": os.environ.get(\"GCP_LOCATION\") or getpass.getpass(\"Enter your GCP Location: \"),\n    \"google_application_credentials\": os.environ.get(\"GOOGLE_APPLICATION_CREDENTIALS\") or getpass.getpass(\"Enter your Google App Credentials path: \")\n})\n\n# embed a sentence\ntest = vtx.embed(\"This is a test sentence.\")\ntest[:10]\n```\n\n### Cohere\n\n[Cohere](https://dashboard.cohere.ai/) allows you to implement language AI into your product. The `CohereTextVectorizer` makes it simple to use RedisVL with the embeddings models at Cohere. For this you will need to install `cohere`.\n\n```bash\npip install cohere\n```\n\n\n```python\nimport getpass\n# setup the API Key\napi_key = os.environ.get(\"COHERE_API_KEY\") or getpass.getpass(\"Enter your Cohere API key: \")\n```\n\n\nSpecial attention needs to be paid to the `input_type` parameter for each `embed` call. For example, for embedding \nqueries, you should set `input_type='search_query'`; for embedding documents, set `input_type='search_document'`. See\nmore information [here](https://docs.cohere.com/reference/embed)\n\n\n```python\nfrom redisvl.utils.vectorize import CohereTextVectorizer\n\n# create a vectorizer\nco = CohereTextVectorizer(\n    model=\"embed-english-v3.0\",\n    api_config={\"api_key\": api_key},\n)\n\n# embed a search query\ntest = co.embed(\"This is a test sentence.\", input_type='search_query')\nprint(\"Vector dimensions: \", len(test))\nprint(test[:10])\n\n# embed a document\ntest = co.embed(\"This is a test sentence.\", input_type='search_document')\nprint(\"Vector dimensions: \", len(test))\nprint(test[:10])\n```\n\nLearn more about using RedisVL and Cohere together through [this dedicated user guide](https://docs.cohere.com/docs/redis-and-cohere).\n\n### VoyageAI\n\n[VoyageAI](https://dash.voyageai.com/) allows you to implement language AI into your product. The `VoyageAITextVectorizer` makes it simple to use RedisVL with the embeddings models at VoyageAI. For this you will need to install `voyageai`.\n\n```bash\npip install voyageai\n```\n\n\n```python\nimport getpass\n# setup the API Key\napi_key = os.environ.get(\"VOYAGE_API_KEY\") or getpass.getpass(\"Enter your VoyageAI API key: \")\n```\n\n\nSpecial attention needs to be paid to the `input_type` parameter for each `embed` call. For example, for embedding \nqueries, you should set `input_type='query'`; for embedding documents, set `input_type='document'`. See\nmore information [here](https://docs.voyageai.com/docs/embeddings)\n\n\n```python\nfrom redisvl.utils.vectorize import VoyageAITextVectorizer\n\n# create a vectorizer\nvo = VoyageAITextVectorizer(\n    model=\"voyage-law-2\",  # Please check the available models at https://docs.voyageai.com/docs/embeddings\n    api_config={\"api_key\": api_key},\n)\n\n# embed a search query\ntest = vo.embed(\"This is a test sentence.\", input_type='query')\nprint(\"Vector dimensions: \", len(test))\nprint(test[:10])\n\n# embed a document\ntest = vo.embed(\"This is a test sentence.\", input_type='document')\nprint(\"Vector dimensions: \", len(test))\nprint(test[:10])\n```\n\n### Mistral AI\n\n[Mistral](https://console.mistral.ai/) offers LLM and embedding APIs for you to implement into your product. The `MistralAITextVectorizer` makes it simple to use RedisVL with their embeddings model.\nYou will need to install `mistralai`.\n\n```bash\npip install mistralai\n```\n\n\n```python\nfrom redisvl.utils.vectorize import MistralAITextVectorizer\n\nmistral = MistralAITextVectorizer()\n\n# embed a sentence using their asynchronous method\ntest = await mistral.aembed(\"This is a test sentence.\")\nprint(\"Vector dimensions: \", len(test))\nprint(test[:10])\n```\n\n### Amazon Bedrock\n\nAmazon Bedrock provides fully managed foundation models for text embeddings. Install the required dependencies:\n\n```bash\npip install 'redisvl[bedrock]'  # Installs boto3\n```\n\n#### Configure AWS credentials:\n\n\n```python\nimport os\nimport getpass\n\nif \"AWS_ACCESS_KEY_ID\" not in os.environ:\n    os.environ[\"AWS_ACCESS_KEY_ID\"] = getpass.getpass(\"Enter AWS Access Key ID: \")\nif \"AWS_SECRET_ACCESS_KEY\" not in os.environ:\n    os.environ[\"AWS_SECRET_ACCESS_KEY\"] = getpass.getpass(\"Enter AWS Secret Key: \")\n\nos.environ[\"AWS_REGION\"] = \"us-east-1\"  # Change as needed\n```\n\n#### Create embeddings:\n\n\n```python\nfrom redisvl.utils.vectorize import BedrockTextVectorizer\n\nbedrock = BedrockTextVectorizer(\n    model=\"amazon.titan-embed-text-v2:0\"\n)\n\n# Single embedding\ntext = \"This is a test sentence.\"\nembedding = bedrock.embed(text)\nprint(f\"Vector dimensions: {len(embedding)}\")\n\n# Multiple embeddings\nsentences = [\n    \"That is a happy dog\",\n    \"That is a happy person\",\n    \"Today is a sunny day\"\n]\nembeddings = bedrock.embed_many(sentences)\n```\n\n### Custom Vectorizers\n\nRedisVL supports the use of other vectorizers and provides a class to enable compatibility with any function that generates a vector or vectors from string data\n\n\n```python\nfrom redisvl.utils.vectorize import CustomTextVectorizer\n\ndef generate_embeddings(text_input, **kwargs):\n    return [0.101] * 768\n\ncustom_vectorizer = CustomTextVectorizer(generate_embeddings)\n\ncustom_vectorizer.embed(\"This is a test sentence.\")[:10]\n```\n\nThis enables the use of custom vectorizers with other RedisVL components\n\n\n```python\nfrom redisvl.extensions.cache.llm import SemanticCache\n\ncache = SemanticCache(name=\"custom_cache\", vectorizer=custom_vectorizer)\n\ncache.store(\"this is a test prompt\", \"this is a test response\")\ncache.check(\"this is also a test prompt\")\n```\n\n## Search with Provider Embeddings\n\nNow that we've created our embeddings, we can use them to search for similar sentences. We will use the same 3 sentences from above and search for similar sentences.\n\nFirst, we need to create the schema for our index.\n\nHere's what the schema for the example looks like in yaml for the HuggingFace vectorizer:\n\n```yaml\nversion: '0.1.0'\n\nindex:\n    name: vectorizers\n    prefix: doc\n    storage_type: hash\n\nfields:\n    - name: sentence\n      type: text\n    - name: embedding\n      type: vector\n      attrs:\n        dims: 768\n        algorithm: flat\n        distance_metric: cosine\n```\n\n\n```python\nfrom redisvl.index import SearchIndex\n\n# construct a search index from the schema\nindex = SearchIndex.from_yaml(\"./schema.yaml\", redis_url=\"redis://localhost:6379\")\n\n# create the index (no data yet)\nindex.create(overwrite=True)\n```\n\n\n```python\n# use the CLI to see the created index\n!rvl index listall\n```\n\nLoading data to RedisVL is easy. It expects a list of dictionaries. The vector is stored as bytes.\n\n\n```python\nfrom redisvl.redis.utils import array_to_buffer\n\nembeddings = hf.embed_many(sentences)\n\ndata = [{\"text\": t,\n         \"embedding\": array_to_buffer(v, dtype=\"float32\")}\n        for t, v in zip(sentences, embeddings)]\n\nindex.load(data)\n```\n\n\n```python\nfrom redisvl.query import VectorQuery\n\n# use the HuggingFace vectorizer again to create a query embedding\nquery_embedding = hf.embed(\"That is a happy cat\")\n\nquery = VectorQuery(\n    vector=query_embedding,\n    vector_field_name=\"embedding\",\n    return_fields=[\"text\"],\n    num_results=3\n)\n\nresults = index.query(query)\nfor doc in results:\n    print(doc[\"text\"], doc[\"vector_distance\"])\n```\n\n## Selecting your float data type\nWhen embedding text as byte arrays RedisVL supports 4 different floating point data types, `float16`, `float32`, `float64` and `bfloat16`, and 2 integer types, `int8` and `uint8`.\nYour dtype set for your vectorizer must match what is defined in your search index. If one is not explicitly set the default is `float32`.\n\n\n```python\nvectorizer = HFTextVectorizer(dtype=\"float16\")\n\n# subsequent calls to embed('', as_buffer=True) and embed_many('', as_buffer=True) will now encode as float16\nfloat16_bytes = vectorizer.embed('test sentence', as_buffer=True)\n\n# to generate embeddings with different dtype instantiate a new vectorizer\nvectorizer_64 = HFTextVectorizer(dtype='float64')\nfloat64_bytes = vectorizer_64.embed('test sentence', as_buffer=True)\n\nfloat16_bytes != float64_bytes\n```\n\n\n```python\n# cleanup\nindex.delete()\n```\n",
  "tags": [],
  "last_updated": "2026-04-08T12:21:52-07:00"
}