Hand gesture recognition and hand tracking for medical applications

Sharif, Hajar

Permalink

https://hdl.handle.net/2142/117779 Copy

Description

Title

Hand gesture recognition and hand tracking for medical applications

Author(s)

Sharif, Hajar

Issue Date

2022-12-02

Director of Research (if dissertation) or Advisor (if thesis)

Kesavadas, Thenkurussi K

Doctoral Committee Chair(s)

Salapaka, Srinivasa M

Committee Member(s)

• Sreenivas, Ramavarapu S
• Phillips, Heidi

Department of Study

Mechanical Sci & Engineering

Discipline

Mechanical Engineering

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Hand Gesture recognition
• Activities of daily living
• Hand grasps classification
• Surgical tasks classification
• Metric development
• surgical proficiency assessment
• Quantitative evaluation of rehabilitation
• Video processing of surgical tasks
• Leap Motion Controller
• Azure Kinect DK
• Myo armband
• Artificial intelligence
• Evaluating suturing and knot tying

Abstract

Hand gestures are a mean of communication and a prevalent type of body language that conveys messages through different shapes constructed by palm and fingers. Hand gesture recognition (HGR) has been of interest in many research fields such as sign language translation, musical creation, and virtual environment control. There are also several studies on HGR for robotics, prosthetic, and rehabilitation applications. In this dissertation, the application of HGR for addressing two challenges in the medical field is presented. The first challenge is to develop a quantitative metric to improve rehabilitation of neurological conditions, with a focus on improvement in performing activities of daily living (ADL), while the second challenge is to develop ATTENTIVE, an automated and quantitative assessment system, to enhance a better evaluation of surgical skills proficiency. Many neurological conditions lead to motor impairment of upper extremity that includes muscle weakness, altered muscle tone, joint laxity, and impaired motor control. As a result, common activities such as reaching, picking up objects, and holding onto them are compromised. Therefore, such patients will experience disability in performing ADL such as eating, writing, performing housework, and so on. Several evaluation methods are commonly used to assess problems in performing ADL. Despite the wide application of these methods, all of them are subjective techniques, i.e. they are either questionnaires or qualitative scores assigned by a medical professional. We hypothesize that providing a more quantitative metric can enhance evaluation of the rehabilitation progress, and lead to a more efficient rehabilitation regimen tailored to the specific needs of each individual patient. Since the first step of developing a metric is to distinguish different ADL activities using hand gesture data, in this dissertation the focus is on classification of ADL tasks using hand gestures. Data analysis pipelines were developed to take in data, collected by the leap motion controller as well as the electromyography and inertial measurement unit sensors, from the lower arm during completion of certain ADL tasks. These pipelines output classification accuracies to distinguish the ADL tasks. Different preprocessing, feature extraction, and classification methods were tested on the data from healthy adults to detect the best structure and parameters for the proposed pipelines. The developed pipelines can be trained and their parameters can be tuned based on data from an intact-adult population. Then, The tuned pipelines can be set as the references. Subsequently, hand motion data from a neurological patient completing the same tasks in the same data collection setup can be fed into the reference pipelines to obtain the classification accuracies. The achieved accuracies indicate how close a patient’s hand motions and muscle activation are to the hand motions and muscle activation of the healthy population. This method enhances assessment of the overall performance of a patient in a quantitative fashion. In addition, the acquired confusion matrices provide insight into the patient’s performance in completing each individual task. The second section of this dissertation includes the design of ATTENTIVE; an evidence-supported, automated, robust, real-time, comprehensive, quantitative assessment system for evaluating proficiency in basic surgical skills. Since ATTENTIVE provides quantitative feedback, it can have a variety of applications in teaching surgical skills either in traditional settings or within incorporation of the augmented reality systems. As of now, the presence of an automated and quantitative assessment system to provide feedback on surgical tasks performance is lacking, and expert surgeon’s involvement is necessary to provide feedback to the surgical trainees. As a result, a trainee’s opportunities to receive feedback on one’s performance is restricted to the availability of an expert surgeon, which is limited due to pre-existing high workload of the expert surgeons. ATTENTIVE can eliminate such restriction that in turn may result in surgical trainees’ performance improvement and superior surgical outcomes over the long run. In this work, the idea and pipeline for developing ATTENTIVE are presented. Next, the apparatus and experimental setup and protocol to investigate the feasibility of ATTENTIVE were designed and built. Afterwards, data was collected from 65 participants completing four basic surgical tasks. The participants were students, residents, and expert surgeons in the fields of veterinary and human medicine. To benefit from both sensor-based and vision-based HGR methods for solving the problem in hand, Azure Kinect DK, Leap Motion Controller, and Myo armband were used to collect data from the lower arm of the participants. ATTENTIVE’s workflow consists of three major steps including separating the main task part from the preparation and cleanup after the task completion, classifying the input surgical task, and assigning a performance score to the input task. In this dissertation, many parts of these three steps are completed, and algorithms to complete the rest are determined and implemented to a vast extend. The details of the data analysis steps are beyond the scope of the abstract and are presented in the second section’s chapters. The last chapter of the current dissertation contains preliminary work on design and fabrication of a wearable device, named iBand, to collect biosignals and kinematic data from the lower arm. Different components of iBand have been selected and calibrated to read synchronized data from the lower arm, transfer them to a computer via Bluetooth, and save them as separate files. An easy-to-work user interface has been developed for iBand to enable user to save the data in the desired folder and with the desired file name. In addition, the user interface enhances a real-time data observation in which the user can choose the sensor from which the collected signal is displayed. Upon completion, the iBand can replace the discontinued Myo armband for research and daily life applications.

Graduation Semester

2022-12

Type of Resource

Thesis

Copyright and License Information

© 2022 Hajar Sharif

Owning Collections