Face Detection and RC Servos

Yesterday I gave a guest lecture at the Korea National University of the Arts  in an Arduino class taught by my friend Uram Choe, an amazing mechanized sculpture artist.  As seen above, if you like metal working, gears and motors, you will love his sculptures!

The topic of my lecture was using computer vision with the Arduino.  Now I am not a computer vision expert or scientist, I have however, dabbled with OpenCV (the most popular computer vision library) enough over the years to know my way around the library fairly well, and get some pretty decent results.  The emphasis here is how to use OpenCV with an Arduino to combine things like motors with video tracking.  It’s surprisingly easy!

Usually I work with openframeworks when using OpenCv but for this lecture I used Processing for simplicity.  After all this lecture was for an Arduino class not a computer graphics class.  Processing also has a nice OpenCV library that implements most of the basic functions of OpenCV.  It’s a great place to get started with computer vision.

Demonstrated here is one part of my lecture, which is a simple example of using the Haarfinder face detection algorithm from OpenCV in combination with the Arduino, the Firmata firmware and an RC servo motor.  The first step is to setup your servo motor and Arduino board, this is quite painless but if you are new to RC servos here is a breadboard diagram I whipped up in Fritzing.

The next step is to go and install all the correct libraries for both Processing and Arduino. For Processing you will need:

1. The OpenCV library and the OpenCV Processing Library. Follow this link and instructions to download and install both.  Download

2. The Firmata library for Processing (also called the Arduino Library for Processing).      Download. Place this Library into your Processing libraries folder, which is located inside of your Processing sketchbook.

For the Arduino you will need:

1. The Firmata library firmware for Arduino. Download. Place this library inside your Arduino libraries folder which is located inside of your Arduino Sketchbook.

So now you have all the proper files and libraries installed, the next step is to put the Firmata firmware on to the Arduino.  All you need to do here is in your Arduino IDE select the Menu: Files -> Examples -> Arduino -> firmware -> Standard_Firmata and open and program this sketch onto your Arduino.  So now you have the Firmata firmware on your Arduino so you can now easily interface with Processing.

So go ahead and startup your Processing IDE and lets start some coding.  The full example code is here: Download.  The code is documented with lots of comments so most of it I won’t cover here.  However, I will go over a few of the more important or difficult aspects of the code.

In your Processing setup function there is one line of code that sets the input resolution of your camera.  In this example the input resolution is being set to width, height, which is the size of your window.  In many applications you will want to have a window that is bigger then your camera input resolution.  If your windows is larger then 320×240 go ahead and change the opencv.capture parameters to your actual camera resolution.

void setup() {

    size( 320, 240 ); //set window size

    opencv = new OpenCV( this ); //setup openCV
    opencv.capture( width, height );   // open video stream

The next critical part of code in the setup function is to set the opencv cascade type.

 opencv.cascade( OpenCV.CASCADE_FRONTALFACE_ALT );

In this line the cascade type is set to: CASCADE_FRONTALFACE_ALT which means the face detection algorithm will look for front facing faces.  There are however, other types of cascades available to you.  Here is a fairly self explainable list of alternatives from the OpenCV processing library docs:

• CASCADE_FRONTALFACE_DEFAULT
• CASCADE_PROFILEFACE
• CASCADE_FULLBODY
• CASCADE_LOWERBODY
• CASCADE_UPPERBODY

Now there is only one important part of the code left to explain and that is the code contained in the Processing draw() function.  The first part of this code is a call to the openCV library to look at the current video frame, detect all the faces in that video and populate those face location and sizes as Rectangles into an array.

// detect faces and put them in an array
Rectangle[] faces = opencv.detect( 1.2, 2, OpenCV.HAAR_DO_CANNY_PRUNING, 40, 40 );

The next few lines of code is what interfaces your Arduino with processing which is the core element of this whole tutorial.  Thanks to the Firmata library its surprisingly simple. First we check if there are faces in the scene.  We don’t want to try to send values if there are no faces present.  The next important thing is to map the values of the face X location to a value between 0 and 255.  Since 0-255 is the range of values the analogWrite command can send to the RC servo.  So now when the face is at X coordinate 320, the RC servo will be sent the value 255.  In the next line we actually call the Arduino analogWrite command and send the mapped value.  The cool feature of Firmata is that you can make actual Arduino function calls from inside processing and the interfacing happens seamlessly.  You don’t need to worry about sending serial data and parsing the results on the Arduino side.  By calling the analogWrite command in Processing the Arduino calls that command and PWMs the correct value to the RC servo.

 if(faces.length > 0)
 {
    int mappedValue = (int) map(faces[0].x,0,width, 0, 255);
    arduino.analogWrite(3, mappedValue);
 }

Now we have face tracking controlling a motor!  Very nice!  Obviously this could be taken much further with multiple faces, multiple motors, and multiple axises. This however, is a great start and its super easy to extend from here.  Goodluck!

UPDATE

It has come to my attention that people are having some issues with this example using windows 7. Something to do with the Processing window showing a blank screen and no video image. I imagine it has something to do with the OpenCV lib having problems grabbing the video from the camera. So I found a work around for the issue. It involves using the normal processing video library to grab the video, then to copy each frame of video into openCV to perform the face detection. This avoids depending on OpenCV to grab the video from the camera. Here is the code:

import processing.video.*;
import hypermedia.video.*;
import java.awt.Rectangle;

Capture myCapture;
OpenCV opencv;

int contrast_value    = 0;
int brightness_value  = 0;

void setup() 
{
  size( 320,240 );    

  myCapture = myCapture = new Capture(this, width, height, 30);
 
  opencv = new OpenCV( this );	
  
  opencv.allocate(width,height);  

  opencv.cascade( OpenCV.CASCADE_FRONTALFACE_ALT ); 
}

void draw() 
{
 //read the output image 
  myCapture.read();
  opencv.copy(myCapture); //copy the video frame into openCV
  
  //put the image on the frame
  image( opencv.image(), 0, 0 );

   opencv.convert( GRAY ); //convert frame to gray scale
   opencv.contrast( contrast_value ); //apply contrast filter
   opencv.brightness( brightness_value ); //apply brightness filter

    // detect faces and put them in an array
    Rectangle[] faces = opencv.detect( 1.2, 2, OpenCV.HAAR_DO_CANNY_PRUNING, 40, 40 );

 //etc, etc, the same as before

}