Found in translation

The Clark Hackathon is an opportunity for students to come together in teams with the goal of creating an app in 36 hours. The winners of the fall 2024 Hackathon were tasked with creating a project under the theme of community development and used the challenge to create an American Sign Language translation program.

Four graduate students studying data analytics teamed up to build the app. The group, Jothsna Praveena Pendyala, M.S. ’24, Mohana Uma Sai Manem, M.S. ’24, Kunal Malhan, M.S. ’25, and Keerthana Goka, M.S. ’25, all drew from their prior experiences working for the Indian government: Pendyala and Manem worked in electrical engineering; Malhan in transportation; and Goka in project management.

“We really wanted to do something that utilized our expertise in motion capture and neural network machine learning and could help people. So, we came up with SignPal, which is an application that translates sign language gestures into English captions,” says Pendyala.

“The app works in real time so there is no translation gap. It is meant to serve people in the deaf community to help them communicate with others who do not know ASL.”

The app was developed with an internal technical architecture that uses the phone’s camera to convert movements into words as they are being signed. The program utilizes a framework called Media Pipe to map facial expressions and hand movements and convert them into what is called a memory layer. It converts those layers into vectors and then into mathematical representations that can be read and decoded by the app’s algorithm to determine what learned word or phrase the program will display to the user.

The team settled on the scope of their project because most translation software relies on static images and singular words. Since ASL is a motion-based language and a lot of translation comes from facial expression along with signing, it is important to have translations processed in real time, similar to watching something with closed captioning.

The team praised Clark Center for Technology, Innovation and Entrepreneurship, Clark’s Competitive Computing Club, or C4, and all the coordinators and participants in this year’s Clark Hackathon.

“Everything that went into the planning was amazing. It made for a seamless and fun experience for all those participating. The energy was positive, the other competitors were awesome, and the food was great. It was a perfect working environment — even if it was for 36 hours straight,” Malhan said with a laugh.

The team also competed at a Harvard University hackathon this past fall and placed third. The group hopes to present a polished and final version of the SignPal app sometime in mid-spring of 2025 and release it to the public.