Finger Detection and Tracking using OpenCV and Python

TL;DR. Code is here.

Tracking the movement of a finger is an important feature of many computer vision applications. In this application, A histogram-based approach is used to separate out the hand from the background frame. Thresholding and Filtering techniques are used for background cancellation to obtain optimum results.

One of the challenges that I faced in detecting fingers is differentiating a hand from the background and identifying the tip of a finger. I’ll show you my technique for tracking a finger, which I used in this project. To see finger detection and tracking in action check out this video.

Finger Detection and Tracking using OpenCV and Python

In an application where you want to track a user’s hand movement, skin color histogram will be very useful. This histogram is then used to subtracts the background from an image, only leaving parts of the image that contain skin tone.

A much simpler method to detect skin would be to find pixels that are in a certain RGB or HSV range. If you want to know more about this approach follow here.

The problem with the above approach is that changing light conditions and skin colors can really mess with skin detection. While on the other hand, Histogram tends to be more accurate and takes into account the current light conditions.

Image for post

Image for post

Hand covering rectangle completely to generate Histogram

Green rectangles are drawn on the frame and the user places their hand inside these rectangles. Application is taking skin color samples from the user’s hand and then creates a histogram.

The rectangles are drawn with the following function:

Function to draw rectangles in Frame

There’s nothing too complicated going on here. I have created four arrays **hand_rect_one_x**, **hand_rect_one_y**, **hand_rect_two_x**, **hand_rect_two_y** to hold the coordinates of each rectangle. The code then iterates over these arrays and draws them on the frame using **cv2.rectangle**. Here **total_rectangle** is just the length of the array i.e. **9**.

Now that the user understands where to place his or her palm, the succeeding step is to extract pixels from these rectangles and use them to generate an HSV histogram.

Generate HSV Histogram

Here function transforms the input frame to HSV. Using Numpy, we create an image of size **[90 * 10]** with **3** color channels and we name it as ROI (Region of Intrest). It then takes the 900-pixel values from the green rectangles and puts them in the ROI matrix.

The **cv2.calcHist** creates a histogram using the ROI matrix for the skin color and **cv2.normalize** normalizes this matrix using the norm Type **cv2.NORM_MINMAX**. Now we have a histogram to detect skin regions in the frames.

Now that we hold a skin color histogram we can use it to find the components of the frame that contains skin. OpenCV provides us with a convenient method, **cv2.calcBackProject**, that uses a histogram to separate features in an image. I used this function to apply the skin color histogram to a frame. If you want to read more about back projection, you can read from here and here.

Back Project using Skin color Histogram

In the first two lines, I changed the input frame to HSV and then applied **cv2.calcBackProject** with the skin color histogram **hist**. Following that, I have used the Filtering and Thresholding function to smoothen the image. Lastly, I masked the input frame using the **cv2.bitwise_and** function. This final frame should just contain skin color regions of the frame.

Image for post

Image for post

Hand seperated from background (1)Image for post

Image for post

Hand separated from the background (2)

Now we have a frame with skin color regions only, but what we really want is to find the location of a fingertip. Using OpenCV you can find contours in a frame if you don’t know what contour is you can read here. Using contours you can find convexity defects, which will be a potential fingertip location.

In my application, I needed to find the tip of a finger with which a user is aiming. To do this I determined the convexity defect, which is furthest from the centroid of the contour. This is done by the following code:

Find fingertip in the frame using Contour, Convex Hull, and DefectsImage for post

Image for post

Contour in Frame (1)Image for post

Image for post

Contour in Frame (2)

Then it determines the largest contour. For the largest contour, it finds the hull, centroid, and defects.

Image for post

Image for post

Defects in red circle and Centroid in purple circle

Now that you have all these defects you find the one that is farthest from the center of the contour. This point is assumed to be the pointing finger. The center is purple and the farthest point is red. And there you have it, you’ve found a fingertip.

Image for post

Image for post

Centroid in purple color and Farthest point in red color

All hard part is done up until now, now all we have to do is to create a **list** to store the changed location of the **farthest_point** in the frame. It’s up to you how many changed points you want to store. I am storing only **20** points.

Image for post

Image for post

Finger Detection and Tracking using OpenCV and Python

Lastly, thank you for reading this post. For more awesome posts, you can also follow me on Twitter — iamarpandey, Github — amarlearning.

Happy coding! 🤓

No Comments Yet