Clark University computer scientist John Magee applies his expertise in human-computer interaction to help others – people with disabilities and, more recently, middle-school students. And by inviting Clark LEEP Fellows and other undergraduates to join his research projects, he’s giving students the opportunity to do the same.
“There are a lot of opportunities in this research to improve people’s lives. That’s my main motivation,” says Magee, assistant professor of computer science.
Recently, Magee and his collaborators at Boston University, University of Massachusetts Amherst and Worcester Polytechnic Institute began a four-year project to develop Smartutors, an online tool that would employ advanced computer vision techniques to measure middle-school students’ emotions as they tackle math lessons. Based on that data, the tool would adjust and personalize the learning experience to keep students engaged.
The project, titled “Detecting, Predicting and Remediating Student Affect and Grit Using Computer Vision,” has been awarded nearly $2.5 million in grants from the National Science Foundation’s Cyberlearning and Future Learning Technologies Program.
“We’re trying to create more automatic estimates of what students’ emotional state is, how engaged they are and whether they’re frustrated,” Magee explains. “The computer vision analyzes their faces and expressions. Along with a computer camera, we’re using existing visual recognition software in new ways to see which face expressions correspond to a student’s various emotional states.”
The researchers are studying the experiences of more than 1,600 New England students using the online MathSpring tutoring system. Developed by faculty at UMass Amherst and WPI, MathSpring is used by teachers to improve and assess mathematics skills in middle and high school students and prepare them for standardized tests.
Magee is working closely with BU Computer Science Professors Margrit Betke and Stan Sclaroff to refine computer vision techniques.
“I also am analyzing mouse movements as students use the smart tutor as an indicator of their engagement and levels of frustration,” he says.
Magee already has involved Clark undergraduates in the project. Jiri Roznovjak ’18 conducted preliminary research, and three others – Lyle Pierson Stachecki ’17, Mishel Sikder ’18 and Jason Vazquez-Li ’17 – worked on related projects. Over the next several years, Magee plans to involve more students.
Improving access to technology
Since receiving his doctorate from Boston University and coming to Clark as a visiting assistant professor in 2011, Magee has collaborated with BU researchers on a number of projects. He also has invited Clark undergraduates to work in Boston with his collaborators and their doctoral students. Last summer, six undergraduates – Roznovjak, Vazquez-Li and LEEP Fellows Pierson Stachecki, Sikder, Trung Ngo ’18 and Rafael Zuniga ’18 – conducted research at BU.
Magee began researching computer vision, an interdisciplinary field that studies how computers process and analyze digital imagery, while a doctoral student at BU. But after meeting people with disabilities who sought to improve their access to technology, he expanded his research focus.
For several years, he and his colleagues worked closely with Rick Hoyt, a 1993 BU alumnus with spastic quadriplegia and cerebral palsy who became famous for his 30 years of participation – in a wheelchair pushed by his father – in the Boston Marathon.
Hoyt, who can’t speak or use his hands, communicates by using a computer via a button that he presses with his head. At BU, he helped test the Camera Mouse, a motion-tracking mouse interface. Developed by Boston College and BU researchers, the Camera Mouse employs a webcam to link movement in a person’s face or head to a computer mouse controlling the cursor on a screen.
“The Camera Mouse has had over 3 million downloads since it was made available in 2000. It means a lot to me that I contributed to something that has been used by so many people to improve their lives.”
When Hoyt used the Camera Mouse, “Magee noticed that Hoyt moved his head more easily in a diagonal motion, … making it difficult for Hoyt to propel the mouse pointer in a straight line,” according to a BU news story. To overcome that issue, Magee wrote adaptive mouse functions software so that Hoyt and other people with motion impairments could better use the Camera Mouse.
“The Camera Mouse has had over 3 million downloads since it was made available in 2000,” Magee says. “It means a lot to me that I contributed to something that has been used by so many people to improve their lives.”
In 2010, Magee met German researcher Torsten Felzer, who used his personal experience with the debilitating neuromuscular disease Friedrich’s ataxia to make advances in assistive technology. Magee collaborated with Felzer on scientific publications and development of a detector for the Camera Mouse that would better understand intentional muscular movements in people with neuromuscular diseases as they used computers.
Felzer died in March 2016.
“He was such an amazing man — he didn’t let his situation hold him back,” Magee says. “He was traveling to conferences and publishing papers and approaching researchers like me and saying, ‘Hey, let’s work on this collaboration.’ He had developed a way of controlling many functions on the computer just with a USB keypad, and with his thumbs, he could reach the buttons. He wrote thousands of lines of code with that device.”
Undergraduates join research
Magee continues the work he started with Felzer and Hoyt and brings undergraduate researchers onto the projects.
Last summer, five computer science students joined Magee on his research to improve computer access for people with disabilities: Ngo, Pierson Stachecki, Sikder, Vazquez-Li and Zuniga.
“LEEP was the highlight of my college career. This project gave me the opportunity to learn more about computer vision and machine learning, and at the same time I got to help people with disabilities,” says Zuniga, a computer science major from Lawrence, Massachusetts. “The fact that I could follow my passion and do the thing I love to help people – I couldn’t ask for anything more.”
Zuniga developed an algorithm to make the Camera Mouse software more accurate for people with Friedreich’s ataxia. Currently, the Camera Mouse tracks only one feature of a person’s face. “But that can be problematic because the interface keeps clicking and going to other web pages even if the user only wants to browse and stay on the screen,” he explains. “My approach relies on visually tracking a second feature on your face. It won’t click a link without checking this second feature.”
“LEEP was the highlight of my college career. This project gave me the opportunity to learn more about computer vision and machine learning, and at the same time I got to help people with disabilities.”
Magee continues to take satisfaction in the fact that his research and that of his students could improve someone’s quality of life.
“I’m interested in what I can do to help people with disabilities who have communication challenges,” Magee says. “Often when people meet someone with severe motor disabilities, they jump to the conclusion that the person also has a cognitive disability.”
Famed physicist Stephen Hawking, who struggles with Lou Gehrig’s disease and “speaks” via a device connected to movement in his cheek muscles, is proof that “the lack of a communication ability is not the same as the lack of an active mind,” Magee says.
And although much of Magee’s research aims to improve interfaces for people with disabilities, it also can be applied to the Smartutors project. In both cases, he is analyzing mouse trajectories vis-à-vis the intentions of users.
“We’re trying to interpret what their actual motions are in relation to their intended motions. The idea here is that systems can be developed so they can adapt to users’ abilities,” he says. “We’re trying to think about design for all.”
