Vector Semantic Text Search Using LangChain (OpenAI) and Redis

TL;DR:

Semantic text search uses AI embeddings to match queries by meaning instead of keywords. In this tutorial, you generate OpenAI embeddings for product descriptions, store them in Redis using LangChain's RedisVectorStore, and query them with a similarity search API—so a search for "pure cotton blue shirts" returns relevant results even if those exact words don't appear in the product listing.

#What you'll learn

• How semantic search differs from traditional full-text search
• How to generate vector embeddings from product descriptions using OpenAI
• How to store and index embeddings in Redis with LangChain
• How to build a search API that finds products by meaning using vector similarity
• How to integrate semantic search into a frontend application

#How is semantic search different from full-text search?

#Terminology

• LangChain: A versatile library for developing language model applications, combining language models, storage systems, and custom logic.
• OpenAI: A provider of cutting-edge language models like GPT-3, essential for applications in semantic search and conversational AI.

#What does the e-commerce application architecture look like?

GITHUB CODE

Below is a command to the clone the source code for the application used in this tutorial

git clone --branch v9.2.0 https://github.com/redis-developer/redis-microservices-ecommerce-solutions

1. products service: handles querying products from the database and returning them to the frontend
2. orders service: handles validating and creating orders
3. order history service: handles querying a customer's order history
4. payments service: handles processing orders for payment
5. api gateway: unifies the services under a single endpoint
6. mongodb/ postgresql: serves as the write-optimized database for storing orders, order history, products, etc.

INFO

You don't need to use MongoDB/ Postgresql as your write-optimized database in the demo application; you can use other prisma supported databases as well. This is just an example.

#What does the frontend look like?

• Dashboard: Displays a list of products with different search functionalities, configurable in the settings page.

• Settings: Accessible by clicking the gear icon at the top right of the dashboard. Control the search bar, chatbot visibility, and other features here.

• Dashboard (Semantic Text Search): Configured for semantic text search, the search bar enables natural language queries. Example: "pure cotton blue shirts."

• Dashboard (Semantic Image-Based Queries): Configured for semantic image summary search, the search bar allows for image-based queries. Example: "Left chest nike logo."

• Chat Bot: Located at the bottom right corner of the page, assisting in product searches and detailed views.

• Shopping Cart: Add products to the cart and check out using the "Buy Now" button.

• Order History: Post-purchase, the 'Orders' link in the top navigation bar shows the order status and history.

• Admin Panel: Accessible via the 'admin' link in the top navigation. Displays purchase statistics and trending products.

#How do you set up the database for semantic search?

INFO

Sign up for an OpenAI account to get your API key to be used in the demo (add OPEN_AI_API_KEY variable in .env file). You can also refer to the OpenAI API documentation for more information.

GITHUB CODE

Below is a command to the clone the source code for the application used in this tutorial

git clone --branch v9.2.0 https://github.com/redis-developer/redis-microservices-ecommerce-solutions

#What does the sample data look like?

#How do you generate and store embeddings in Redis?

1. Generating Vector Documents: Utilizing the convertToVectorDocuments function, we transform product details into vector documents. This transformation is crucial as it converts product details into a format suitable for Redis storage.
2. Seeding Embeddings into Redis: The seedOpenAIEmbeddings function is then employed to store these vector documents into Redis. This step is essential for enabling efficient retrieval and search capabilities within the Redis database.

TIP

Download Redis Insight to visually explore your Redis data or to engage with raw Redis commands in the workbench.

#How do you build the semantic search API?

#What does the search API request and response look like?

#Search API request format

#Search API response structure

#How does the search API implementation work?

1. getProductsByVSSText function handles the API Request.
2. getSimilarProductsScoreByVSS function performs semantic search on product details. It integrates with OpenAI's semantic analysis capabilities to interpret the searchText and identify relevant products from Redis vector store.

#How do you configure the frontend for semantic search?

• Settings configuration: Enable Semantic text search in the settings page

• Performing a search: Use textual queries on the dashboard.

• Note: Users can click on the product description within the product card to view the complete details.

#Next steps

• Get started with vector similarity search: Learn the fundamentals of vector search in Redis with the vector similarity search getting started tutorial.
• Build a RAG chatbot: Combine semantic search with generative AI to create a conversational chatbot powered by Redis and LangChain.
• Try image-based semantic search: Extend your search beyond text with semantic image-based queries using LangChain and Redis.

#Further reading

• Perform vector similarity search using Redis
• Build a RAG GenAI chatbot with Redis and LangChain
• Semantic image-based queries using LangChain (OpenAI) and Redis
• LangChain JS
• Learn LangChain
• LangChain Redis integration