Virtual Mouse using Computer Vision

As computer vision became increasingly more advance and readily available, it has created a tapestry of helpful use cases. I came into encounter with computer vision while I was learning machine learning and saw the CNN, Convolutionary Neural Networks. With just a glimpse of it, I was immediately intrigued to create my own. As Iron Man has been my youth inspiration, I thought of an idea to recreate the holograms from the movie by tracking my hand to move the cursor on my laptop. This is the project that started my delves into the astonishing computer vision world.

Illustration created by Dall E

Adventure, Conclusion & Software setup

I used my native language, python with the following libraries:
- OpenCV

- Mediapipe

- Pyautogui

I started out by creating the standard structure for the OpenCV and learning the basics of it. Later on after various couple YouTube videos and documentations on sketchy 2011 websites, I figured out the tracking algorithm and function for computer vision. Firstly, I tracked the position of the hand, with indexes that respond to each of the fingers. I created a directly proportional 16:9 bounding box, which transcribe to the position of the mouse on the 1920:1080 screen on my laptop. I move the mouse using pyautogui, to the new current position localized using my index finger.

Then, like all my projects, I encounter a problem, the frames per second dropped to 2. However as I was busy, I took a 2 week break. Then, when I come back to it, I realized I forgot to comment & annotate the code, in which I reasonably added them. However during the 2 week break from this project, I learned threading, which was the key to my project's frame rate. I simply added threading and it just miraculously changed from 2 frames per second to 63 frames per second. I have achieved my goal of replicating a fiction to reality.

To view the code