Semantic Image Based Queries Using LangChain (OpenAI) and Redis

For developersSemantic Image Based Queries Using LangChain (OpenAI) and Redis

Prasan Kumar

Will Johnston

What you will learn in this tutorial

This tutorial demonstrates how to perform semantic search on product images using LangChain (OpenAI) and Redis. Specifically, we'll cover the following topics:

E-Commerce Application Context : Consider a sample e-commerce application scenario where customers can utilize image-based queries for product searches, add items to their shopping cart, and complete purchases, thereby highlighting a real-world application of semantic search.
Database setup : This involves generating descriptive summaries for product images, creating semantic embeddings for generated summaries and efficiently storing them in Redis.
Setting up the search API : This API is designed to process user queries in the context of image content. It integrates the capabilities of OpenAI for semantic analysis with Redis for efficient data retrieval and storage.

Terminology

LangChain is an innovative library for building language model applications. It offers a structured way to combine different components like language models (e.g., OpenAI's models), storage solutions (like Redis), and custom logic. This modular approach facilitates the creation of sophisticated AI applications.

OpenAI provides advanced language models like GPT-3, which have revolutionized the field with their ability to understand and generate human-like text. These models form the backbone of many modern AI applications including semantic text/ image search and chatbots.

Microservices architecture for an e-commerce application

GITHUB CODE

Below is a command to the clone the source code for the application used in this tutorial

git clone --branch v9.2.0 https://github.com/redis-developer/redis-microservices-ecommerce-solutions

Lets take a look at the architecture of the demo application:

products service: handles querying products from the database and returning them to the frontend
orders service: handles validating and creating orders
order history service: handles querying a customer's order history
payments service: handles processing orders for payment
api gateway: unifies the services under a single endpoint
mongodb/ postgresql: serves as the write-optimized database for storing orders, order history, products, etc.

NOTE

You don't need to use MongoDB/ Postgresql as your write-optimized database in the demo application; you can use other prisma supported databases as well. This is just an example.

E-commerce application frontend using Next.js and Tailwind

The e-commerce microservices application consists of a frontend, built using Next.js with TailwindCSS. The application backend uses Node.js. The data is stored in Redis and either MongoDB or PostgreSQL, using Prisma. Below are screenshots showcasing the frontend of the e-commerce app.

Dashboard: Displays a list of products with different search functionalities, configurable in the settings page.

Settings: Accessible by clicking the gear icon at the top right of the dashboard. Control the search bar, chatbot visibility, and other features here.

Dashboard (Semantic Text Search): Configured for semantic text search, the search bar enables natural language queries. Example: "pure cotton blue shirts."

Dashboard (Semantic Image-Based Queries): Configured for semantic image summary search, the search bar allows for image-based queries. Example: "Left chest nike logo."

Chat Bot: Located at the bottom right corner of the page, assisting in product searches and detailed views.

Selecting a product in the chat displays its details on the dashboard.

Shopping Cart: Add products to the cart and check out using the "Buy Now" button.

Order History: Post-purchase, the 'Orders' link in the top navigation bar shows the order status and history.

Admin Panel: Accessible via the 'admin' link in the top navigation. Displays purchase statistics and trending products.

Database setup

NOTE

Sign up for an OpenAI account to get your API key to be used in the demo (add OPEN_AI_API_KEY variable in .env file). You can also refer to the OpenAI API documentation for more information.

GITHUB CODE

Below is a command to the clone the source code for the application used in this tutorial

git clone --branch v9.2.0 https://github.com/redis-developer/redis-microservices-ecommerce-solutions

Sample data

In this tutorial, we'll use a simplified e-commerce dataset. Specifically, our JSON structure includes product details and a key named styleImages_default_imageURL, which links to an image of the product. This image will be the focus of our AI-driven semantic search.

Generating OpenAI image summary

The following code segment outlines the process of generating a text summary for a product image using OpenAI's capabilities. We'll first convert the image URL to a base64 string using fetchImageAndConvertToBase64 function and then utilize OpenAI to generate a summary of the image using getOpenAIImageSummary function.

Sample image & OpenAI summary

The following section demonstrates the result of the above process. We'll use the image of a Puma T-shirt and generate a summary using OpenAI's capabilities.

Comprehensive summary generated by the OpenAI model is as follows:

Seeding Image summary embeddings

The addImageSummaryEmbeddingsToRedis function plays a critical role in integrating AI-generated image summaries with Redis. This process involves two main steps:

Generating Vector Documents: Utilizing the getImageSummaryVectorDocuments function, we transform image summaries into vector documents. This transformation is crucial as it converts textual summaries into a format suitable for Redis storage.
Seeding Embeddings into Redis: The seedImageSummaryEmbeddings function is then employed to store these vector documents into Redis. This step is essential for enabling efficient retrieval and search capabilities within the Redis database.

The image below shows the JSON structure of openAI image summary within RedisInsight.

TIP

Download RedisInsight to visually explore your Redis data or to engage with raw Redis commands in the workbench.

Setting up the search API

API end point

This section covers the API request and response structure for getProductsByVSSImageSummary, which is essential for retrieving products based on semantic search using image summaries.

Request Format

The example request format for the API is as follows:

Response Structure

The response from the API is a JSON object containing an array of product details that match the semantic search criteria:

API implementation

The backend implementation of this API involves following steps:

getProductsByVSSImageSummary function handles the API Request.
getSimilarProductsScoreByVSSImageSummary function performs semantic search on image summaries. It integrates with OpenAI's semantic analysis capabilities to interpret the searchText and identify relevant products from Redis vector store.

Frontend UI

Settings configuration: Initially, ensure that the Semantic image summary search option is enabled in the settings page.

Performing a search: On the dashboard page, users can conduct searches using image-based queries. For example, if the query is Left chest nike logo, the search results will display products like a Nike jacket, characterized by a logo on its left chest, reflecting the query.

Viewing image summaries: Users can click on any product image to view the corresponding image summary generated by OpenAI. This feature offers an insightful glimpse into how AI interprets and summarizes visual content.

Ready to use Redis for semantic image based queries?

Performing semantic search on image summaries is a powerful tool for e-commerce applications. It allows users to search for products based on their descriptions or images, enabling a more intuitive and efficient shopping experience. This tutorial has demonstrated how to integrate OpenAI's semantic analysis capabilities with Redis to create a robust search engine for e-commerce applications.