Mobile port for CameraMouse

Danny Cooper

Problem Definition

The primary problem to be solved here is not one of computer vision theory, but rather one of application and integration-- a functional piece of software (CameraMouse) already exists, but it needs to be extended and made more accessible. With this goal in mind, I plan to devote my project time toward bringing the current port up to the point where it can build and pass basic testing on at least one mobile platform (IOS or Android). It is unclear how large of an effort will be required to accomplish this; as such, I've defined some additional stretch goals:

  1. Convert arbitrary facial articulation into a binary or trinary output. The intention here is to create something that will be usable for even severely motion-restricted individuals. A potential "cheat" for this approach would be to generate candidate areas from user input (possibly something as simple as flow detection), and then have the user (or their caretaker) apply a small mark or sticker to the area of high articulation to enable template-based tracking. This should work even if the area being tracked is a relatively non-featureful portion of the face.

  2. Integration with mobile OS accessiblity features on IOS or Android, possibly by having the CameraMouse software emulate input from a hardware switch.

  3. Integration with OSX or Windows accessibility features, once more, possibly in binary switch emulation mode.

Desired Result

It is my hope to get beyond the minimum viable product discussed above (port of core CameraMouse functionality to be cross platform); assuming I manage to implement some of the OS integration features, I would want to assess the actual usability of the end-user device using the CameraMouse as the sole method of input once setup is finished. An easy test of this would be to see if it is possible for me (as a user with a full range of facial articulation) to navigate the OS menus enough to send a text message in a reasonable amount of time. If this test succeeds, I would plan to move on to harder tests on myself and other students (send a text using only your eyebrows, for instance). If the software also passes these tests, I would be interested in testing it with individuals with constrained range of motion, but do not want to even consider such tests until I am reasonably certain the software will actually work.


My current plans do not involve any major changes to the underlying algorithms in CameraMouse, although I have been looking into work done in the world of motion captures using facial markers to determine whether a marker-assisted capture might be useful for some individuals.

Some potential readings on this topic (I've skimmed/read the abstracts):

With regard to operating system integrations:

Android switch support appears to be adequate: