Google AI has opened a new ML tool for conceptual and subjective image queries

Google AI open source mood board searcha new ML-powered tool for subjective or conceptual image queries.

Mood board search, helps users define conceptual and subjective queries like peacefuland beautiful, on the pictures. Advancements in the use of deep learning in computer vision have enabled engineers and researchers to provide different functionalities such as similar image search, object detection, tagging, etc. One of the main challenges in this field is how to define and interrogate images with conceptual intent. Mood board research helps people train and personalize the deep learning model in the way they see the world. The following snapshot shows how an artist sees the world by categorizing images into different artistic concepts.

personal classification of images into abstract and subjective concepts

In the search for mood boards, the researchers used pre-trained computer vision models like GoogLeNet and MobileNetand a machine learning approach called Concept Activation Vectors (CAC).

CAV is a technique for measuring the sensitivity of a trained model to the concept presented by the user. The following image shows how the CAV or CAV under test works.

Obtain the tested CAV score which quantifies the sensitivity of the classifier to the concept

For example, a deep learning model trained to classify images as zebra or non-zebra. We want to quantify the importance of the band concept for the classifier. By simply running TCAV and getting the score, we can answer the question. CAV is used as one of the general techniques for the explainability of deep learning models. As mentioned in the blog post :

In Mood Board Search, we use CAVs to find a model’s sensitivity to a user-created mood board. In other words, each mood board creates a CAV – a direction in the integration space – and the tool searches a dataset of images, bringing up images that best match the CAV. However, the tool goes one step further, segmenting each image in the dataset in 15 different ways, to uncover as many relevant compositions as possible.

Working with the Mood Board Search GUI is simple. As explained in the blog post :

To get started, simply drag and drop a small number of images that represent the idea you want to convey. Mood Board Search returns best results when images share a consistent visual quality, so results are more likely to be relevant with mood boards that share visual similarities in color, pattern, texture, or composition .

Some of the design studios like morrama enthusiastic about this tool and tweeted :

Great job, can’t wait to play with it.

Google AI open-source code for researchers and developers for more contributions in this area. Also, there is a experimental application by design invention studio Nordic projectswhich uses mood board search.

Comments are closed.