Semantic Text Search Using LangChain (OpenAI) and Redis

For developersSemantic Text Search Using LangChain (OpenAI) and Redis

Prasan Kumar

Will Johnston

What you will learn in this tutorial

This tutorial explores the implementation of semantic text search in product descriptions using LangChain (OpenAI) and Redis. The focus areas include:

Contextualizing E-Commerce: Dive into an e-commerce scenario where semantic text search empowers users to find products through detailed textual queries.
Database Implementation: Learn to create and store semantic embeddings from product descriptions in Redis for efficient search capabilities.
Search API Development: Understand how to build an API that leverages OpenAI for semantic analysis of text and Redis for data management.

Terminology

LangChain: A versatile library for developing language model applications, combining language models, storage systems, and custom logic.
OpenAI: A provider of cutting-edge language models like GPT-3, essential for applications in semantic search and conversational AI.

Microservices architecture for an e-commerce application

GITHUB CODE

Below is a command to the clone the source code for the application used in this tutorial

git clone --branch v9.2.0 https://github.com/redis-developer/redis-microservices-ecommerce-solutions

Lets take a look at the architecture of the demo application:

products service: handles querying products from the database and returning them to the frontend
orders service: handles validating and creating orders
order history service: handles querying a customer's order history
payments service: handles processing orders for payment
api gateway: unifies the services under a single endpoint
mongodb/ postgresql: serves as the write-optimized database for storing orders, order history, products, etc.

INFO

You don't need to use MongoDB/ Postgresql as your write-optimized database in the demo application; you can use other prisma supported databases as well. This is just an example.

E-commerce application frontend using Next.js and Tailwind

The e-commerce microservices application consists of a frontend, built using Next.js with TailwindCSS. The application backend uses Node.js. The data is stored in Redis and either MongoDB or PostgreSQL, using Prisma. Below are screenshots showcasing the frontend of the e-commerce app.

Dashboard: Displays a list of products with different search functionalities, configurable in the settings page.

Settings: Accessible by clicking the gear icon at the top right of the dashboard. Control the search bar, chatbot visibility, and other features here.

Dashboard (Semantic Text Search): Configured for semantic text search, the search bar enables natural language queries. Example: "pure cotton blue shirts."

Dashboard (Semantic Image-Based Queries): Configured for semantic image summary search, the search bar allows for image-based queries. Example: "Left chest nike logo."

Chat Bot: Located at the bottom right corner of the page, assisting in product searches and detailed views.

Selecting a product in the chat displays its details on the dashboard.

Shopping Cart: Add products to the cart and check out using the "Buy Now" button.

Order History: Post-purchase, the 'Orders' link in the top navigation bar shows the order status and history.

Admin Panel: Accessible via the 'admin' link in the top navigation. Displays purchase statistics and trending products.

Database setup

INFO

Sign up for an OpenAI account to get your API key to be used in the demo (add OPEN_AI_API_KEY variable in .env file). You can also refer to the OpenAI API documentation for more information.

GITHUB CODE

Below is a command to the clone the source code for the application used in this tutorial

git clone --branch v9.2.0 https://github.com/redis-developer/redis-microservices-ecommerce-solutions

Sample data

Consider a simplified e-commerce dataset featuring product details for semantic search.

Seeding product details embeddings

Implement the addEmbeddingsToRedis function to integrate AI-generated product description embeddings with Redis.

This process involves two main steps:

Generating Vector Documents: Utilizing the convertToVectorDocuments function, we transform product details into vector documents. This transformation is crucial as it converts product details into a format suitable for Redis storage.
Seeding Embeddings into Redis: The seedOpenAIEmbeddings function is then employed to store these vector documents into Redis. This step is essential for enabling efficient retrieval and search capabilities within the Redis database.

Examine the structured openAI product details within Redis using RedisInsight.

TIP

Download RedisInsight to visually explore your Redis data or to engage with raw Redis commands in the workbench.

Setting up the search API

API end point

This section covers the API request and response structure for getProductsByVSSText, which is essential for retrieving products based on semantic text search.

Request Format

The example request format for the API is as follows:

Response Structure

The response from the API is a JSON object containing an array of product details that match the semantic search criteria:

API implementation

The backend implementation of this API involves following steps:

getProductsByVSSText function handles the API Request.
getSimilarProductsScoreByVSS function performs semantic search on product details. It integrates with OpenAI's semantic analysis capabilities to interpret the searchText and identify relevant products from Redis vector store.

Frontend UI

Settings configuration: Enable Semantic text search in the settings page

Performing a search: Use textual queries on the dashboard.

Note: Users can click on the product description within the product card to view the complete details.

Ready to use Redis for semantic text search?

Discover the power of semantic text search for enhancing the e-commerce experience. This tutorial guides you through integrating OpenAI's semantic capabilities with Redis for a dynamic product search engine.