Embeddings and vector databases

Embeddings and vector databases

⚠️
In this article we're going to explore how to create a Vector Database by creating embeddings using the OpenAI API and then storing them in a Qdrant cluster.
The Goal is to create a long term memory for an AI model or having an advanced and high-performance vector similarity search technology based on a huge database of PDF or any other files connected to an AI model.

Embeddings and Vector Database

What are Embeddings?

Embeddings are object entities like images, texts, audios etc, that have been converted into an array of numbers known as vectors, that contains patterns of relationships and similarities between these entities in a way that's meaningful for a computational model. For example, in word embeddings, words with similar meanings are often close together in this vector space.

What is a Vector Database?

Well once an embedding is created, it can be stored in a database, a database full of embeddings is called a vector database. It can be used in several ways including:
  1. Searching where results are ranked by relevance to a query string.
  2. Clusterings where text strings are grouped by similarity.
  3. Classifications where text strings are classified by their most similar label.
  4. Recommendations where items with related text strings are recommended.
In this article we're going to focus on the first use case, searching because it's the most common use case. There are many practical ways to do this, but OpenAI has provided a great AI Model to specifically create embeddings, however, they doesn't provide a way to store them, in which we'll use Qdrant cloud service to store them.

Getting Started

Setting up OpenAI account

  1. Create an account if you do not have one OpenAI, follow this Guide if you have trouble accessing your account.
  2. Create an API key here API Keys
    • click on the + Create new secret key button. Give it a name and click the green create secret key button.
    • Once you have created your API key, copy it and save it somewhere safe, you'll need it later.
      Follow this Guide if you have trouble creating an API key.

Setting up Qdrant Cloud

  1. Create an account if you do not have one Qdrant Cloud.
  2. To use Qdrant Cloud, you will need to create at least one cluster. There are two ways to start:
    • Create a Free Tier cluster with 1 node and a default configuration (1GB RAM, 0.5 CPU and 4GB Disk). This option is perfect for prototyping and you don’t need a credit card to join.
    • Configure a custom cluster with additional nodes and more resources. For this option, you will have to provide billing information.
    In this article I'll be using a Free Tier cluster for testing purposes. The capacity should be enough to serve up to 1M vectors of 768dim. To calculate your needs, refer to capacity planning
  3. Once you have created your cluster, copy the API key and the cluster url save them somewhere safe, you'll need them later.
  4. Check the status of your cluster, it should be HEALTHY before you can use it.

Creating our first Embedding

we're going to use OpenAI's embeddings API endpoint to create our first embedding using text-embedding-ada-002 model, let's use a curl command to send a request to the API endpoint.

Curl request

The response should look like this:

Curl response

Great, now we have our first embedding, we can now store it in a Qdrant cluster.
for the sake of this article I asume you already have python installed on your machine, if not, you can follow this Guide to install it.

Building a Chatbot with OpenAI and Qdrant

In this section we're going to start building our Chatbot, we start by setting up a python environment, installing required packages and then interact with Qdrant cluster and OpenAI API with the help of LangChain. I'll be working on a Lunix machine, PyCharm IDE and python 3.11, you can use any IDE of your choice. I'll be working in a jupyter notebook also you can use a python file.

Creating a python environment

To start, create a new project directory, navigate in the project's directory, open your terminal and run the following command:

Create env

This will create a new folder called venv in the project directory, this folder will contain all packages. activate the venv by running the following command:

Activate env

Installing required packages

create a requirements.txt file in your project's root directory and add the following packages:

Requirements

then run the following command to install the packages:

Install requirements

setting up the project structure

Create three files in our project's root directory:
  • main.ipynb this is where we'll write our code.
  • .env this is where we'll store our API keys and cluster url.
  • settings.py this is where we'll read our API keys and cluster url.
the project structure should look like this:

File structure

API keys and cluster url

Add API keys and cluster url in the .env file, it should look like this:

Environment variables

Reading API keys and cluster url from the .env file

In the settings.py file we're going to read our API keys and cluster url from the .env file with the help of python-decouple.

Setting API keys

Now that we are done with the setup, we can now start writing our code inside main.ipynb.

Getting env variables

Getting variables

Story file for the Chatbot

Feel free to use your own story, but for this article's story file you can download it here, add it to your project directory under data/.

Load story

Creating a qdrant client

Creating a cluster

Creating a collection

A collection is a named set of vectors points among which you can search. The vector of each point within the same collection must have the same dimensionality and be compared by a single metric. more about collections here

Creating a collection

Checking the collection info

Checking the collection info

For the first time, the collection will be empty with vectors_count=0 and the CollectionStatus should be green if it is ready, so the response should look like this:

Collection info

Creating a vector store

Creating a vector store

Creating text chunks

To add the text story to the vector store, we need to split it into chunks, for a free cluster API limit, we use CharacterTextSplitter from langchain to achieve this.

Creating text chunks

Adding chunks to the vector store

Adding chunks to the vector store

The response should look like this:

Collection info

Creating a QA retrieval chain on the vector store

Creating a QA retrieval chain

Ask a question

Ask a question

That's it, you can now ask any question about the story and get the answer.

Conclusion and next steps

In this article we have seen how to create embeddings using OpenAI API and store them in a Qdrant cluster, we have also seen how to use the vector store to create a QA retrieval chain and and create a Chatbot. this is just a simple example of what you can do with OpenAI and Qdrant, you can create chatbots, search engines, recommendation systems etc. There also a lot off many other tools and services that you can use to achieve the same goals, you can check out Jina, Gemini, Pinecone, Faiss and many others.
For the next steps, I have created a SaaS platform on top of OpenAI and Qdrant, the platform allows you to add pdf files and ask questions about them, you can check it out here.

References && Resources