American Sign Language Recognition Using Adversarial Learning in a Multi-Frequency RF Sensor Network

Simple item page
dc.contributorGan, Yu
dc.contributorCrawford, Chris
dc.contributor.advisorGurbuz, Sevgi Z
dc.contributor.authorMacks, Trevor F
dc.contributor.otherUniversity of Alabama Tuscaloosa
dc.date.accessioned2022-04-13T20:33:41Z
dc.date.available2022-04-13T20:33:41Z
dc.date.issued2020
dc.descriptionElectronic Thesis or Dissertationen_US
dc.description.abstractHuman computer interaction (HCI) based technologies, such as Alexa or Siri, have become increasingly prevalent in the daily lives of American citizens. However, access to these technologies is gated by the need to communicate with spoken commands, which subsequently precludes the Deaf community from benefitting from the quality of life improvements they provide. Current approaches to providing HCI technologies or ASL conversion largely rely on video and image processing techniques, haptic gloves, and wifi based systems. However, wearables restrict users from engaging in their normal daily activities, while video raises privacy concerns. To help ASL compatible HCI technology, we propose a multi-frequency RF sensing network for the recognition of a basic lexicon of signs. When validating the network on a daily activities dataset, we had good performance, with classification accuracy's around 90% or higher. For ASL data, we have 2 datasets: native and imitation. The native dataset is small, and was collected from Deaf individuals. Our imitation dataset is larger, and was collected from hearing individuals prompted by copysigning videos. We show that imitation data cannot be used in lieu of native ASL data for training and benchmarking classifiers because the two datasets possess disparate distributions in feature space. Alternatively, we investigate adversarial learning as a means for mitigating the challenge of insufficient training data. Cross frequency training is one option for augmenting the training dataset which suffers from severe performance degradation when data from one frequency is used to pre-train a network for classification of data at another frequency. We show that data synthesized using Generative Adversarial Networks (GANs) can be used to reduce but not completely eliminate cross-frequency training degradation. An auxiliary conditional generative adversarial network (ACGAN) with kinematic sifting is used to augment and classify human activity data and recognize ASL signs. While the proposed network performed well with daily activities, its performance could not be adequately validated on ASL data due to sparsity of native ASL data and statistical inconsistencies of imitation signing data. Future directions for overcoming these challenges and extending the proposed techniques to ASL recognition are discussed.en_US
dc.format.mediumelectronic
dc.format.mimetypeapplication/pdf
dc.identifier.otherhttp://purl.lib.ua.edu/176866
dc.identifier.otheru0015_0000001_0003660
dc.identifier.otherMacks_alatus_0004M_14233
dc.identifier.urihttps://ir.ua.edu/handle/123456789/8365
dc.languageEnglish
dc.language.isoen_US
dc.publisherUniversity of Alabama Libraries
dc.relation.hasversionborn digital
dc.relation.ispartofThe University of Alabama Electronic Theses and Dissertations
dc.relation.ispartofThe University of Alabama Libraries Digital Collections
dc.rightsAll rights reserved by the author unless otherwise indicated.en_US
dc.subjectASL
dc.subjectASLR
dc.subjectGAN
dc.subjectHCI
dc.subjectRF Sensing
dc.titleAmerican Sign Language Recognition Using Adversarial Learning in a Multi-Frequency RF Sensor Networken_US
dc.typethesis
dc.typetext
etdms.degree.departmentUniversity of Alabama. Department of Educational Leadership, Policy, and Technology Studies
etdms.degree.disciplineElectrical engineering
etdms.degree.grantorThe University of Alabama
etdms.degree.levelmaster's
etdms.degree.nameM.S.
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
u0015_0000001_0003660.pdf
Size:
17.71 MB
Format:
Adobe Portable Document Format
Download
Collections
Theses and Dissertations
Theses and Dissertations - Department of Educational Leadership, Policy & Technology Studies