Recognizing Textured Planar Objects with OpenCV

Here is the problem to begin with: We have a few planar textured objects that we want to recognize in images. We are permitted to create models of these objects in a training phase. Then the task is to recognize and locate these objects in still 2D gray-scale images.

The approach presented in this article is available as open source (tpofinder) and has been developed for robotic machine vision (mappotino). The technique is very similar to those used in some panorama stitching applications. More precisely, it is a bag-of-features approach, where the test images and the models are described by local features that are somewhat robust towards scale, rotation and change in illumination. An approximative nearest-neighbor algorithm matches the local features between the test image and the models of the objects. This is done in order to find correspondences between the each model and the test image. Finally, a robust model fitting algorithm estimates homographies between the models and test images based on the matched keypoints. Here is a real-time video of the detector in action:

This approach is not really new. There are already a few implementations of object detectors available on the web. Just in order to mention a few I am aware of, there are both pure object detectors or frameworks such as: BLORT, MOPED, RoboEarth, Object Recognition Kitchen. However, in the robot challenge, we required software that is quickly installed, compiled, and integrated. This was a few months ago, and after a few attempts on the one or other third-party detector, we turned towards rolling out our own detector on reinvented wheels, tpofinder.

The models are obtained from multiple views. The collection process currently requires manual annotation of the region of interest in an image that shows the object of interest. Also, if there are multiple views of the object, the homographies between the views need to be specified through pairs of annotated keypoints on the respective images. Apart from that, tpofinder can be tested out-of-the-box with a webcam. If no object from the sample model database is at hand, then a picture of it will do as well. Here the simplification of planarity pays out in an unexpected way. During testing, the detector is able to locate more than one object in the image. However, in the implementation, the time required for detection increases with the number of models: tpofinder was tested with five objects on a 2-years old notebook and obtained a detection rate of 2 fps.

At the end, tpofinder is a small and, up to some Boost libraries, purely OpenCV-based object detector. In fact, most of the credits belong to OpenCV, which provides algorithms for feature extraction and matching, as well as functions for robust model fitting. The OpenCV project has advanced in big steps in the last year in several aspects, such as: incorporation of recent vision algorithms, cleaning up of mess from earlier times, an appropriate website, improved documentation, migration to git. Progress, which hopefully continues.