Vector Semantic Image Based Queries Using LangChain (OpenAI) and Redis

TL;DR:

Use OpenAI to generate text descriptions of product images, store those descriptions as vector embeddings in Redis, and then search images by meaning using natural language queries. Redis handles the vector similarity search, returning products whose images best match the search text.

#What you'll learn

• How to generate AI-powered text summaries from product images using OpenAI's vision model
• How to create vector embeddings from image summaries and store them in Redis
• How to build a semantic search API that finds products by image content
• How to integrate LangChain, OpenAI, and Redis into an e-commerce application

#Terminology

#What does the e-commerce application architecture look like?

GITHUB CODE

Below is a command to the clone the source code for the application used in this tutorial

git clone --branch v9.2.0 https://github.com/redis-developer/redis-microservices-ecommerce-solutions

1. products service: handles querying products from the database and returning them to the frontend
2. orders service: handles validating and creating orders
3. order history service: handles querying a customer's order history
4. payments service: handles processing orders for payment
5. api gateway: unifies the services under a single endpoint
6. mongodb/ postgresql: serves as the write-optimized database for storing orders, order history, products, etc.

NOTE

You don't need to use MongoDB/ Postgresql as your write-optimized database in the demo application; you can use other prisma supported databases as well. This is just an example.

#How is the e-commerce frontend built?

• Dashboard: Displays a list of products with different search functionalities, configurable in the settings page.

• Settings: Accessible by clicking the gear icon at the top right of the dashboard. Control the search bar, chatbot visibility, and other features here.

• Dashboard (Semantic Text Search): Configured for semantic text search, the search bar enables natural language queries. Example: "pure cotton blue shirts."

• Dashboard (Semantic Image-Based Queries): Configured for semantic image summary search, the search bar allows for image-based queries. Example: "Left chest nike logo."

• Chat Bot: Located at the bottom right corner of the page, assisting in product searches and detailed views.

• Shopping Cart: Add products to the cart and check out using the "Buy Now" button.

• Order History: Post-purchase, the 'Orders' link in the top navigation bar shows the order status and history.

• Admin Panel: Accessible via the 'admin' link in the top navigation. Displays purchase statistics and trending products.

#How do you set up the database for image search?

NOTE

Sign up for an OpenAI account to get your API key to be used in the demo (add OPEN_AI_API_KEY variable in .env file). You can also refer to the OpenAI API docs for more information.

GITHUB CODE

Below is a command to the clone the source code for the application used in this tutorial

git clone --branch v9.2.0 https://github.com/redis-developer/redis-microservices-ecommerce-solutions

#What does the sample data look like?

#How do you generate image summaries with OpenAI?

#What does a generated image summary look like?

#How do you store image summary embeddings in Redis?

1. Generating Vector Documents: Utilizing the getImageSummaryVectorDocuments function, we transform image summaries into vector documents. This transformation is crucial as it converts textual summaries into a format suitable for Redis storage.
2. Seeding Embeddings into Redis: The seedImageSummaryEmbeddings function is then employed to store these vector documents into Redis. This step is essential for enabling efficient retrieval and search capabilities within the Redis database.

TIP

Download Redis Insight to visually explore your Redis data or to engage with raw Redis commands in the workbench.

#How do you build the image search API?

#What does the search API endpoint look like?

#Search API request format

#Search API response structure

#How does the vector similarity search work?

1. getProductsByVSSImageSummary function handles the API Request.
2. getSimilarProductsScoreByVSSImageSummary function performs semantic search on image summaries. It integrates with OpenAI's semantic analysis capabilities to interpret the searchText and identify relevant products from Redis vector store.

#How do you use the image search in the frontend?

• Settings configuration: Initially, ensure that the Semantic image summary search option is enabled in the settings page.

• Performing a search: On the dashboard page, users can conduct searches using image-based queries. For example, if the query is Left chest nike logo, the search results will display products like a Nike jacket, characterized by a logo on its left chest, reflecting the query.

• Viewing image summaries: Users can click on any product image to view the corresponding image summary generated by OpenAI. This feature offers an insightful glimpse into how AI interprets and summarizes visual content.

#Next steps

• How to perform vector similarity search using Redis in NodeJS -- learn the fundamentals of vector search with Redis
• Semantic text search using LangChain (OpenAI) and Redis -- apply similar techniques to text-based semantic search
• LangChain JS
• Learn LangChain
• LangChain Redis integration