Snap-It Find-It: Your Shopping Companion Bot

Image Retrieval Application Using Novel Computer Vision Techniques

Searching for furniture is a surprisingly time-consuming and mundane process. There are multiple different furniture retailers such as Ikea and HipVan which have a high volume of furniture in their catalogs. Typically, people would spend hours browsing through the catalogs trying to find something they would like, or spend hours visiting the physical stores in an attempt to find something that fits into the theme of their new apartment. Often, people may draw inspiration when they unexpectedly come across a nice piece of furniture, and decide they want to own a similar piece. The problem statement surfaces: Where can I find something similar to this piece of furniture? Does Ikea or HipVan have something similar?

This presents the problem statement the team would be trying to tackle. Given an input RGB image of certain household objects or furniture, develop a computer vision system to return images of similar objects or furniture from Ikea and HipVan catalog. The computer vision system would include an object detector module and an image similarity module, neatly packaged into a telegram bot as the user interface. This system can help users narrow down quickly which major furniture vendor the user should visit, and help make the entire furniture hunting process more efficient.

Our team has the answer: we build a telegram bot that help you search for matching product from Ikea and Hipvan!

We created a Telegram bot to help you search for furniture and household products @ https://t.me/SnapitFindit_bot

This is a series of Medium posts made by 4 NUS SCALE master students (MSc. in Industry 4.0) who are taking ISY5004 Intelligent Sensing Systems. Here’s a snapshot of what we’ll be sharing:

Web scrapping, feature extraction using CNN, and image searching.(this post)
Object detection using YOLOv3.
Object detection and background removal using Detectron2.
Further improvement using various CNN models.
Telegram bot and hosting on cloud.

So, to start, how are we going to get the data? Let’s scrape the data from Ikea and Hipvan!

Let’s start with understanding the HTML page of Ikea website, we need to get all the product category link. To do so, I use BeautifulSoup to extract the link that I needed:

Now, if you are familiar with using Google Chrome developer tools, you can expose the API to query the product catalog which comprises product id, price, product name, image URL (which is what we looking for)! Here’s a sneak peek of the code…

And, the all_data dataframe consist of following information:

Now, we can download all the product image easily:

And, here is a snapshot of the images we scraped:

Now, let’s prepare the dataset and split them into train and test set for feature extraction and performance metrics measurement:

To extract the features for all downloaded images, we can utilize the transfer learning method which uses the weights from imagenet. We get the output from FC2 layer using VGG16 CNN model and save the train and test features into an .npy file.

To measure how good our model is, we then define functions to measure the accuracy and precision as well as the vector distance of 2 features set from 2 images by using Euclidean distance:

Now we can measure the accuracy and precision using the following code:

And here’s the result, you can see the precision and accuracy are not that good, however, our team worked a lot further to improve this straight out-of-the-box benchmark, so, please stay tuned until the end of this series of posts!

Nevertheless, let’s visualize the result by searching for a lounge sofa…

It’s not that bad, right? This sofa is out-of-sample data, meaning that it does not exist in our database or extract features, the algorithm gives us the top 5 closest matching furniture!

However, in real life, a picture took by any camera consists of background and could be more than 1 object, in the following post, we’ll be sharing how to use different computer vision object detection methods to identify the subject and remove background before conducting searching… stay tuned!

In part 2 (link below), we’ll be sharing how to use YOLOv3 as the object detection model first before searching for a similar product!