Fundamental Guide for Stereo Vision Cameras in Robotics – Tutorials and Resources

Advertisement

Machine vision is based on information from digital images and depending on the application, the vision system can be designed for inspection, guidance, detecting, tracking, etc. Human visual system is the most sophisticated and powerful vision solution to observe the environment and extract information. A similar system with the biological vision was built for robotic applications and is called stereo vision.

A stereo vision system is designed to extract 3D information from digital images and use these for examining the position of objects in two images, to build an advanced object recognition system that recognizes objects in different arrangements (for example when objects are placed one in front of the other), tracking different objects, etc.

Because a stereo vision is similar to the human biological system, some of the features are identical. For example, a human has two eyes to see slightly different views of the same environment. A stereo vision system has two cameras located at a known distance and take pictures of the scene at the same time. Using the geometry of the cameras, we can apply algorithms and create the geometry of the environment.

Among the advantages of a stereo vision system can be included its reliability and effectiveness in extracting various information (like color, or dimension), it can be used for different vision routines like tracking or detecting objects, and it’s a passive sensor which cannot be influenced by environment.

A series of most popular stereo vision sensors as well as tutorials about how these can be used are the subject of this article.

Advertisement




Stereo Camera Sensors

A wide variety of 3D stereo vision sensors for simple to complex applications.
A large variety of camera sensors make more difficult the choice and this is the case when before purchasing any stereo vision system has to be calculated a series of features. Some cameras are more sensitive while others have the ability to let the user to specify the bit-rate, image quality, set the shutter speed or average illumination in the image.

As an example, for a mobile outdoor robot is preferable to be used a wide field of view to capture a large number of objects that may be moving and at a time, these will get in range of the robot.

How many frames per second is required, if the focal length is fixed or variable, how interface the sensor with electronic boards, and a minimum camera resolution are four features which must be taken into account before buying a camera.

Below is available a collection of most popular stereo camera sensors with different specifications and designed for different applications.

Bumblebee XB3 and Bumblebee 2
Two stereo vision cameras with complete hardware and software packages. Bumblebee 2 has a resolution of 640×480 at 48FPS or 1024×768 at 20 FPS while XB3 provide a higher resolution of 1280×960 at 15 FPS.

Bumblebee XB3

Bumblebee XB3

Kinect 3D
Coming from a virtual gaming world, the sensor Kinect is appreciated among robotics enthusiasts and is used in a wide range of projects. Since it is a 3D sensor with a friendly interface, Kinect is a powerful tool for 3D vision and provide 640×480 pixels resolution at 30 FPS.

Kinect 3D

Kinect 3D

Surveyor Stereo Vision System (“SVS”)
Open-source stereo vision system designed especially for education, research, or hobbyist applications. The vision sensor comes with Omnivision OV9655 at 1.3 megapixel sensor and provides clear images in different light conditions.

Surveyor Stereo Vision System ("SVS")

Surveyor Stereo Vision System (“SVS”)

MEGA-DCS
Compact stereo camera vision compatible with a wide range of operating systems and provide a maximum resolution of 1280 x 960 pixels.

MEGA-DCS

MEGA-DCS

PCI nDepth vision system
With a baseline of 6 centimeters, PCI nDepth is a friendly vision system with two wide-VGA CMOS digital sensors and a resolution of 752×480 pixels at 60 FPS.

PCI nDepth vision system

PCI nDepth vision system

Ensenso N10
Stereo camera with compact size, USB 2.0 interface, and a 3D image sensor with 752×480 pixels resolution at 30 FPS.

Ensenso N10

Ensenso N10

Capella
Complete vision system with OpenCV library support. At 30 FPS the camera can record images at 736×480 pixels resolution.

Capella

Capella

Minoru 3D Webcam
Perhaps the cheapest stereo vision camera that working as a normal webcam and with 3D vision capabilities.

Minoru 3D Webcam

Minoru 3D Webcam

Scorpion 3D Stinger
Vision system designed especially for industrial applications including here picking parts robots, its features include capture images, identifies and located objects.

Scorpion 3D Stinger

Scorpion 3D Stinger

Camera Calibration

To achieve a high accuracy in detecting, tracking or measuring different objects with a stereo vision system the calibration is required.
Interface a stereo vision system with a robot and then start detecting, measuring or tracking different objects in the real world is not a simple process. Increasing the accuracy in detecting and measuring objects is part of camera calibration aiming to establish a relation between image pixels and real world dimensions.

Having two different views of the same scene is the case when the distance between an object part of a scene and stereo camera can be estimated by finding the object in both left and right images and applying trigonometry formula.

In the following are available a series of tutorials and guides for stereo camera calibrations.

  • Stereo Calibration – tutorial for implementation of pinhole camera model in camera calibration;
  • How do I get started with Point Grey Bumblebee 2/ XB3 Stereo Cameras? – guide how to connect and calibrate XB3 or Bumblebee 2 stereo vision cameras;
  • OpenCV: stereo camera calibration – tutorial how to calibrate stereo vision camera and start calculating 3D stuff by using OpenCV;
  • Stereo-vision (image rectification) Python OpenCV tutorial – comprehensive tutorial for stereo vision system calibration using Python programming language and OpenCV open-source vision software;
  • Camera Calibration Toolbox for Matlab – different tools and methods are available for stereo camera calibration and in this tutorial can be found all information to start using Toolbox for Matlab in stereo image rectification and 3D stereo triangulation;
  • How to Calibrate a Stereo Camera – step-by-step guide for beginners to start calibrating with ROS and cameracalibrator.py node a stereo vision camera;

Stereo Vision Sensors Tutorials and Guides

OpenCV or Matlab are two powerful software tools used in a wide range of applications including distance estimation between objects and stereo system.
A stereo vision system can be used in different applications like distance estimation between object relative to the stereo vision system, as well as the use of stereo vision camera with different methods for image processing like cvFindStereoCorrespondenceBM from OpenCV, or with Matlab and Computer Vision System Toolbox to calculate the stereo disparity seen by left and right camera.

A series of examples how to use different vision systems with OpenCV or Matlab are available below.

  • Stereo Vision with OpenCV and QT – guide how to use two webcams for stereo image capture and calculate 3D depth based on stereo correspondence;
  • OpenCV Stereo Matching – guide how to calculate the stereo disparity relative to the stereo camera of any object. This guide also describes two methods available in OpenCV. One method is cvFindStereoCorrespondenceBM for processing images very fast, while the second method uses the cvFindStereoCorrespondenceGC node for image processing which is very low in processing images, but has high accuracy in calculations;
  • OpenCV: Stereo Webcam – OpenCV source code for left and right image visualization with cheap webcams;
  • Minoru 3D webcam for real-time stereo imaging – tutorial start using Minoru 3D Webcam in real time applications with OpenCV, PointCloud and libv4l2cam libraries;
  • Receiving images from the stereo camera – OpenCV source code to receive images from Minoru 3D Webcam;
  • Simple triangulation with OpenCV from Harley & Zisserman – guide how to use cvTriangulatePoints method for triangulation with OpenCV;
  • Computing a disparity map in OpenCV – example how to compute and use a disparity map for a stereo vision system;
  • Stereo Vision – example how to use Matlab together with Computer Vision System Toolbox for calculates the depth map from stereo images;
  • Structure from Motion and 3D reconstruction on the easy in OpenCV 2.3+ – step-by-step tutorial using OpenCV and stereo vision system to estimation motion between two images;
  • Making a step to stereo vision – comprehensive tutorial to build a stereo vision system using two webcams and start detecting or tracking objects;
  • How to synchronize two CMOS Camera Modules for Stereo Vision – guide how to build a cheap stereo vision system using two CMOS cameras;
Robotics community only grows by sharingShare on FacebookShare on Google+Tweet about this on TwitterShare on RedditShare on StumbleUponDigg this

1 comment » Write a comment

  1. Another stereo vision camera from e-con Systems: Tara is a UVC-Complaint USB 3.0 Stereo vision camera based on MT9V024 stereo sensor from OnSemi imaging which supports WVGA(1504×480) at 60fps over USB 3.0 in uncompressed format.

Leave a Reply

Required fields are marked *.