Content-based image retrieval

Content-based image retrieval (CBIR), also known as query by image content (QBIC) and content-based visual information retrieval (CBVIR) is the application of computer vision to the image retrieval problem, that is, the problem of searching for digital images in large databases. "Content-based" means that the search makes use of the contents of the images themselves, rather than relying on human-inputted metadata such as captions or keywords. A content-based image retrieval system (CBIRS) is a piece of software that implements CBIR.

There is growing interest in CBIR because of the limitations inherent in metadata-based systems. Textual information about images can be easily searched using existing technology, but requires humans to personally describe every image in the database.This is impractical for very large databases, or for images that are generated automatically, e.g. from surveillance cameras. It is also possible to miss images that use different synonyms in their descriptions. Systems based on categorizing images in semantic classes like "cat" as a subclass of "animal" avoid this problem but still face the same scaling issues.

The ideal CBIR system from a user perspective would involve what is referred to as semantic retrieval, where the user makes a request like "find pictures of dogs" or even "find pictures of Abraham Lincoln". This type of open-ended task is very difficult for computers to perform - pictures of chihuahuas and Great Danes look very different, and Lincoln may not always be facing the camera or in the same pose. Current CBIR systems therefore generally make use of lower-level features like texture, color, and shape, although some systems take advantage of very common higher-level features like faces (see facial recognition system). Not every CBIR system is generic. Some systems are designed for a specific domain, e.g. shape matching can be used for finding parts inside a CAD-CAM database.

Different implementations of CBIR make use of different types of user queries.

• With query by example, the user searches with a query image (supplied by the user or chosen from a random set), and the software finds images similar to it based on various low-level criteria.
• With query by sketch, the user draws a rough approximation of the image they are looking for, for example with blobs of color, and the software locates images whose layout matches the sketch.
• Other methods include specifying the proportions of colors desired (e.g. "80% red, 20% blue") and searching for images that contain an object given in a query image (as at [1] (http://seco.asa.cs.uni-frankfurt.de/Seco/mascot/)).

CBIR systems can also make use of relevance feedback, where the user progressively refines the search results by marking images in the results as "relevant", "not relevant", or "neutral" to the search query, then repeating the search with the new information.

One application of CBIR is to identify images with skin-tones and shapes that could indicate the presence of nudity, for filtering and for searching by law enforcement.

External links

• SIMPLIcity and ALIP online Demos (http://wang.ist.psu.edu/IMAGE/) developed by Stanford and Penn State Universities
• GIFT - The GNU Image Finding Tool (http://www.gnu.org/software/gift/) - an open source query by example CBIRS
• SIMBA  (http://simba.informatik.uni-freiburg.de/) - demo of the Search IMages By Appearance system by the Albert-Ludwigs-Universität Freiburg (Germany) - Inst. for Pattern Recognition and Image Processing
• FIRE online demo (http://www-i6.informatik.rwth-aachen.de/~deselaers/cgi_bin/fire.cgi), FIRE homepage (http://www-i6.informatik.rwth-aachen.de/~deselaers/fire.html) FIRE (Flexible Image Retrieval Engine) is another open source query by example CBIRS
• Viper Demo (http://viper.unige.ch/demo/) - an online demonstration of the GIFT
• LCPD: Leiden 19th-Century Portrait Database (http://nies.liacs.nl:1860/) - an online database of 19th century studio portraits searchable via CBIR and commonly referenced in the literature
• imgSeek (http://imgseek.python-hosting.com/wiki) - opensource photo collection manager and viewer with content-based search and many other features
• Video Google demo (http://www.robots.ox.ac.uk/~vgg/research/vgoogle/) - search movies for specific objects

Relevant research papers

• SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture Libraries (http://www-db.stanford.edu/~wangz/project/imsearch/SIMPLIcity/TPAMI/) (Wang, Li, and Wiederhold, 2001]

• Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach (http://www-db.stanford.edu/~wangz/project/imsearch/ALIP/PAMI03/) (Li and Wang, 2003]

• System for Screening Objectionable Images (http://www-db.stanford.edu/~wangz/project/imscreen/JCC98/) (Wang et al., 1998)

• Video google: A text retrieval approach to object matching in videos (http://ieeexplore.ieee.org/iel5/8769/27772/01238663.pdf) (Sivic & Zisserman, 2003)

• Names and Faces in the News (http://www1.cs.columbia.edu/CAVE/NSF-ITR/research/publications/Berkeley/miller_cvpr_04.pdf) (Berg et al, 2004)

• FACERET: An Interactive Face Retrieval System Based on Self-Organizing Maps (http://www.springerlink.com/index/10FL41074LE5699P.pdf) (Ruiz-del-Solar et al, 2002)

• Costume: A New Feature for Automatic Video Content Indexing (http://www.irit.fr/~Gael.Jaffre/RECHERCHE/thesis.html) (Jaffre 2005)

• Finding Naked People (http://www.cs.hmc.edu/~fleck/naked-people.ps) (Fleck et al, 1996)

• Automatic Face Recognition for Film Character Retrieval in Feature-Length Films (http://mi.eng.cam.ac.uk/~oa214/academic/publications/2005_CVPR_paper2.pdf) (Arandjelovic & Zisserman, 2005)