Taking something in a hand requires a complex coordination of sight and hand. However, many people with neurodegenerative diseases or other coordination problems are unable to correctly perform this action. This research paper presents a hand motion interpretation system, which uses a video flow captured from under the user’s wrist, known as active perspective. To assess the algorithm, we have placed a variety of objects in a work area in front of the user. Through the video flow, our system can classify different hand movements with regard to the objects on the scene. For motion classification, we have proposed a set of descriptors, which are used by the classification algorithms kNN and HMM. Results show that our system is capable of detecting over 90% of approaching and lateral motions, regardless of what the objects on the scene are.