Similarity search for motif mining#

In this tutorial, we demonstrate how to utilize the similarity functionality to discover images with similar motifs from a collection of Reddit images. We employ the classic SIFT feature to identify images with a strikingly similar appearance (image-level pipeline).

Additionally, we extend the pipeline by incorporating an object detection model, YOLO, in combination with the SIFT feature. This enables us to identify objects within the images that exhibit a similar appearance (object-level similarity).

To illustrate the seamless integration of different vector stores, we leverage the power of multiple vector stores, namely FAISS and QDRANT, within evadb. This demonstrates the ease with which you can utilize diverse vector stores to construct indexes, enhancing your similarity search experience.

Run on Google Colab View source on GitHub Download notebook


Connect to EvaDB#

%pip install --quiet "evadb[vision,document,notebook]"
%pip install --quiet kornia
import evadb
cursor = evadb.connect().cursor()

[notice] A new release of pip is available: 23.0.1 -> 23.1.2
[notice] To update, run: pip install --upgrade pip

Note: you may need to restart the kernel to use updated packages.

Download reddit dataset#

!wget -nc https://www.dropbox.com/scl/fo/fcj6ojmii0gw92zg3jb2s/h\?dl\=1\&rlkey\=j3kj1ox4yn5fhonw06v0pn7r9 -O reddit-images.zip
!unzip -o reddit-images.zip -d reddit-images

--2023-06-08 00:31:01--  https://www.dropbox.com/scl/fo/fcj6ojmii0gw92zg3jb2s/h?dl=1&rlkey=j3kj1ox4yn5fhonw06v0pn7r9
Resolving www.dropbox.com (www.dropbox.com)... 162.125.9.18, 2620:100:601f:18::a27d:912
Connecting to www.dropbox.com (www.dropbox.com)|162.125.9.18|:443... connected.
HTTP request sent, awaiting response... 

302 Found
Location: https://uc912506ea27523ad54fa19d99cb.dl.dropboxusercontent.com/zip_download_get/Bg_n1TIe-Rpj8XQrqPf2tWuG5IPRQFoTZox_tvDaJGIY0xql83b2V5WvE2lD_TX-Oe-oeAQxGuanYugiLTTWKR4ph5tXDnqxjzRDBXIVIb0bbw# [following]
--2023-06-08 00:31:03--  https://uc912506ea27523ad54fa19d99cb.dl.dropboxusercontent.com/zip_download_get/Bg_n1TIe-Rpj8XQrqPf2tWuG5IPRQFoTZox_tvDaJGIY0xql83b2V5WvE2lD_TX-Oe-oeAQxGuanYugiLTTWKR4ph5tXDnqxjzRDBXIVIb0bbw
Resolving uc912506ea27523ad54fa19d99cb.dl.dropboxusercontent.com (uc912506ea27523ad54fa19d99cb.dl.dropboxusercontent.com)... 

162.125.9.15, 2620:100:601f:15::a27d:90f
Connecting to uc912506ea27523ad54fa19d99cb.dl.dropboxusercontent.com (uc912506ea27523ad54fa19d99cb.dl.dropboxusercontent.com)|162.125.9.15|:443... connected.
HTTP request sent, awaiting response... 

200 OK
Length: 4397651 (4.2M) [application/zip]
Saving to: ‘reddit-images.zip’


reddit-images.zip     0%[                    ]       0  --.-KB/s               

reddit-images.zip   100%[===================>]   4.19M  23.5MB/s    in 0.2s    

2023-06-08 00:31:03 (23.5 MB/s) - ‘reddit-images.zip’ saved [4397651/4397651]

Archive:  reddit-images.zip
warning:  stripped absolute path spec from /
mapname:  conversion of  failed
 extracting: reddit-images/g348_d7jgzgf.jpg  
 extracting: reddit-images/g348_d7jphyc.jpg  
 extracting: reddit-images/g348_d7ju7dq.jpg  
 extracting: reddit-images/g348_d7jhhs3.jpg  
 extracting: reddit-images/g1074_d4n1lmn.jpg  
 extracting: reddit-images/g1074_d4mxztt.jpg  
 extracting: reddit-images/g1074_d4n60oy.jpg  
 extracting: reddit-images/g1074_d4n6fgs.jpg  
 extracting: reddit-images/g1190_cln9xzr.jpg  
 extracting: reddit-images/g1190_cln97xm.jpg  
 extracting: reddit-images/g1190_clna260.jpg  
 extracting: reddit-images/g1190_clna2x2.jpg  
 extracting: reddit-images/g1190_clna91w.jpg  
 extracting: reddit-images/g1190_clnad42.jpg  
 extracting: reddit-images/g1190_clnajd7.jpg  
 extracting: reddit-images/g1190_clnapoy.jpg  
 extracting: reddit-images/g1190_clnarjl.jpg  
 extracting: reddit-images/g1190_clnavnu.jpg  
 extracting: reddit-images/g1190_clnbalu.jpg  
 extracting: reddit-images/g1190_clnbf07.jpg  
 extracting: reddit-images/g1190_clnc4uy.jpg  
 extracting: reddit-images/g1190_clncot0.jpg  
 extracting: reddit-images/g1190_clndsnu.jpg  
 extracting: reddit-images/g1190_clnce4b.jpg  
 extracting: reddit-images/g1209_ct65pvl.jpg  
 extracting: reddit-images/g1209_ct66erw.jpg  
 extracting: reddit-images/g1209_ct67oqk.jpg  
 extracting: reddit-images/g1209_ct6a0g5.jpg  
 extracting: reddit-images/g1209_ct6bf1n.jpg  
 extracting: reddit-images/g1418_cj3o1h6.jpg  
 extracting: reddit-images/g1418_cj3om3h.jpg  
 extracting: reddit-images/g1418_cj3qysz.jpg  
 extracting: reddit-images/g1418_cj3r4gw.jpg  
 extracting: reddit-images/g1418_cj3z7jw.jpg

Load all images into evadb#

response = cursor.query("DROP TABLE IF EXISTS reddit_dataset;").df()
cursor.query(
    "LOAD IMAGE 'reddit-images/*.jpg' INTO reddit_dataset;"
).df()

06-08-2023 09:50:33 WARNING[drop_object_executor:drop_object_executor.py:_handle_drop_table:0050] Table: reddit_dataset does not exist
0
0 Number of loaded IMAGE: 34

Register a SIFT FeatureExtractor#

It uses kornia library to extract sift features for each image

cursor.query("DROP UDF IF EXISTS SiftFeatureExtractor;").df()
cursor.query("""CREATE UDF IF NOT EXISTS SiftFeatureExtractor
                    IMPL  '../evadb/udfs/sift_feature_extractor.py'""").df()

06-08-2023 09:50:36 WARNING[drop_object_executor:drop_object_executor.py:_handle_drop_udf:0082] UDF SiftFeatureExtractor does not exist, therefore cannot be dropped.
0
0 UDF SiftFeatureExtractor successfully added to... 
# Keep track of which image gets the most votes
from collections import Counter
vote = Counter()

Image-level similarity search pipeline.#

This pipeline creates one vector per image. Next, we should breakdown steps how we build the index and search similar vectors using the index.

#1. Create index for the entire image
cursor.query("""DROP INDEX IF EXISTS reddit_sift_image_index""").df()
cursor.query("""CREATE INDEX reddit_sift_image_index 
                    ON reddit_dataset (SiftFeatureExtractor(data)) 
                    USING FAISS""").df()

06-08-2023 09:50:37 WARNING[drop_object_executor:drop_object_executor.py:_handle_drop_index:0107] Index reddit_sift_image_index does not exist, therefore cannot be dropped.
0
0 Index reddit_sift_image_index successfully add... 
#2. Search similar vectors
response = cursor.query("""SELECT name FROM reddit_dataset ORDER BY
                    Similarity(
                      SiftFeatureExtractor(Open('reddit-images/g1190_clna260.jpg')),
                      SiftFeatureExtractor(data)
                    )
                    LIMIT 5""").df()

#3. Update votes
for i in range(len(response)):
    vote[response["reddit_dataset.name"][i]] += 1
print(vote)

Counter({'reddit-images/g1190_clna260.jpg': 1, 'reddit-images/g1190_clndsnu.jpg': 1, 'reddit-images/g1190_clna91w.jpg': 1, 'reddit-images/g1190_clnc4uy.jpg': 1, 'reddit-images/g1190_cln97xm.jpg': 1})

Object-level similarity search pipeline.#

This pipeline detects objects within images and generates vectors exclusively from the cropped objects. The index is then constructed using these vectors. To showcase the versatility of evadb, we leverage Qdrant vector store specifically for building this index. This demonstrates how seamlessly you can leverage different vector stores within evadb.

1. Extract all the object using Yolo from the images#

create_index_query = """
    CREATE MATERIALIZED VIEW IF NOT EXISTS 
      reddit_object_table (name, data, bboxes,labels)
      AS SELECT name, data, bboxes, labels FROM reddit_dataset
      JOIN LATERAL UNNEST(Yolo(data)) AS Obj(labels, bboxes, scores)"""
cursor.query(create_index_query).df()

2023-06-08 09:50:42,010	INFO worker.py:1625 -- Started a local Ray instance.

2. Build an index on the feature vectors of the extracted objects#

cursor.query("""CREATE INDEX reddit_sift_object_index
                    ON reddit_object_table (SiftFeatureExtractor(Crop(data, bboxes)))
                    USING QDRANT""").df()
0
0 Index reddit_sift_object_index successfully ad... 
# Create a cropped images (We are actively working on features to allow
# us to not do this outside SQL)
response = (
    cursor.query(
        "LOAD IMAGE 'reddit-images/g1190_clna260.jpg' INTO reddit_search_image_dataset"
    )
    .df()
)
print(response)
response = (
    cursor.query("SELECT Yolo(data).bboxes FROM reddit_search_image_dataset;")
    .df()
)
print(response)


import cv2
import pathlib

bboxes = response["yolo.bboxes"][0]

img = cv2.imread("reddit-images/g1190_clna260.jpg")
pathlib.Path("reddit-images/search-object/").mkdir(parents=True, exist_ok=True)
for i, bbox in enumerate(bboxes):
    xmin, ymin, xmax, ymax = bbox
    xmin, ymin, xmax, ymax = int(xmin), int(ymin), int(xmax), int(ymax)
    cropped_img = img[ymin:ymax, xmin:xmax]
    cv2.imwrite(f"reddit-images/search-object/search-{i}.jpg", cropped_img)

                           0
0  Number of loaded IMAGE: 1

                                         yolo.bboxes
0  [[257.2467956542969, 256.8749084472656, 457.67...

Combine the scores from image level and object level similarity to show similar images#

# !pip install matplotlib
import matplotlib.pyplot as plt

# Display top images
vote_list = list(reversed(sorted([(path, count) for path, count in vote.items()], key=lambda x: x[1])))
img_list = [path for path, _ in vote_list]

fig, ax = plt.subplots(nrows=1, ncols=6, figsize=[18,10])
ax[0].imshow(cv2.imread("reddit-images/g1190_clna260.jpg"))
ax[0].set_title("Search")

for i in range(5):
    axi = ax[i + 1]
    img = cv2.imread(img_list[i])
    axi.imshow(img)
    axi.set_title(f"Top-{i + 1}")

plt.show()
../../_images/addc06ca5c863bad857489d107d3c66beb323d37263ca68f0022fbb2a53cda11.png