SFSU Engineering Achieves Major Milestone with Sony SSUP Collaboration
We are proud to announce that the School of Engineering has successfully achieved the primary research goal of the ongoing collaboration with Sony’s Sensing Solutions University Program (SSUP). The project, titled “Neural2: Efficient Deep Neural Network for Neural-controlled Bionic Arm with Spresense Microcontroller,” has reached a significant milestone by developing the very first high-density electromyography (HD-EMG)-based real-time neural-machine interface (NMI) using the Sony Spresense edge board. Engineering is excited to continue the collaboration with Sony on an expanded project titled “Efficient and Robust Deep Learning Models for Neural-Machine Interfaces with Spresense Microcontroller.”
This groundbreaking innovation integrates deep learning techniques to enable precise hand gesture recognition by processing HD-EMG signals from human subjects’ forearm muscles in real time. The system’s deep learning compression techniques ensure that the neural network models run efficiently on the resource-constrained microcontroller board, offering the potential for real-time movement intent-controlled systems, such as bionic arms and rehabilitation applications.
The Neural2 project was presented by the graduate research assistant Peter Chudinov at the SSUP US End-year Review Meeting to Sony’s Japan and Europe teams in September 2024, highlighting the successful implementation and future potential of this cutting-edge technology. The collaboration demonstrated the critical advancements made in neural-controlled interfaces, showcasing the integration of efficient deep learning techniques for resource-constrained environments.
(Left) Graduate research assistant Peter Chudinov is setting up the HD sEMG system. (Right) The research progress on sEMG and deep learning based HMIs.
In addition to our research goals, Sony Spresense product has been integrated into the engineering curriculum such as ENGR 859: On-Device Machine Learning, undergraduate capstone projects, and outreach activities that engage community college and high school students in our research. This provides students with hands-on experience in developing and optimizing machine learning applications for edge devices.
(Left) The SoE Summer 2024 Research Internship Program Closing Day Event. (Right) Community college interns Srihita Panati and Jay Motamarry presenting their research on HD-EMG-based real-time NMI using the Sony Spresense edge board.
Continuing Collaboration for the Future
Building on this success, SFSU Engineering is excited to continue the collaboration with Sony on an expanded project titled “Efficient and Robust Deep Learning Models for Neural-Machine Interfaces with Spresense Microcontroller.” This next continuing collaboration will focus on further optimizing the robustness and efficiency of deep learning models in neural-machine interfaces (NMIs), enabling more reliable and scalable applications in prosthetics, rehabilitation devices, virtual input devices, and augmented reality. In addition to research, the collaboration will enhance educational efforts by integrating cutting-edge Sony products into the engineering curriculum, fostering student engagement through hands-on learning opportunities and project development.
The SFSU research team is led by Dr. Zhuwei Qin, Assistant Professor and Director of the Mobile and Intelligent Computing Lab (MIC Lab), and Dr. Xiaorong Zhang, Professor and Director of the Intelligent Computing and Embedded Systems Lab (ICE Lab). Both labs play key roles in optimizing deep neural network models and exploring their potential in new human-machine interface applications.
“Our continued collaboration with Sony on the Spresense platform allows us to push the boundaries of NMI technology, creating AI systems that are not only efficient but also robust enough for real-world applications,” said Dr. Qin.
Looking Ahead
As the project progresses, the SFSU team is focused enhancing the functionality and accessibility of next-generation NMI technology. The success of these projects will have broad-reaching impacts on AI-powered NMI, offering hope and innovation in fields such as rehabilitation robotics and biomedical engineering.
Stay tuned for more updates as SFSU Engineering continues to innovate at the intersection of efficient deep learning, biomedical signal processing, and neural-machine interface!