Sublynk (MIT Reality Hack 2024 version)
A downloadable app for Windows
What is it?
Sublynk is a mixed reality Quest3 app that displays AR face tracked English subtitles and an original motion captured database of SEE (Signed Exact English) to encourage learning of sign.
This project was made in two-ish days for MIT Reality Hack 2024. Currently there are only a couple test phrases such as: "Where is the bathroom", "I need help", "Say hello to my little friend", "I love you", "This is delicious". It's more of an open-source proof of concept, but we encourage folks to tinker.
License
Sublynk is licensed under the MIT open-source license (see PDF file 'license' in project).
Team Members:
Development
Steven Harmon @stevenharmongames - Programmer
Animation Pipeline / Usability
Vivian Ngiam @Vivianngiam - Designer
Xiaohan Qiu @raychiuxh - Designer
Production & Research
Mahsa Goodarzi @mgoodarzi - Project Manager
Steven Bueno @buenosteven02 - Storyteller
Download
Install instructions
Setup
The build export is windows only for now (remnants of commented out Meta Speech SDK for standalone builds and should require no setup aside from ensuring speech input is enabled)
Make sure in your Privacy settings -> Inking & typing personalization that "Getting to know you" is toggled on. The game utilizes Windows's dictation feature.
Hardware Required
Quest 2 or 3 (only tested on 3)
Ensure you have an external webcam plugged in. (The experience uses a ducktaped webcam ontop Quest3 since passthrough is blocked for security reasons) Technically, the external webcam is in addition to the built-in webcam. So you need two webcams for it to take the input of webcam 2. It's hardcoded currently. Sorry!
Software Dependencies
This project is created using Unity 2022.3.18f1, Microsoft's Speech API, MediaPipe's BlazeFace model, and the Leap2 Motion Controller (for mocap of the signs)
Run
Download Sublynk.zip (to be added upon completion)
Extract & Enjoy!
For the standalone executable in the zip, no dependencies needed. Just make sure to calibrate the face tracking by holding the left/right arrow keys to align the hands so that they are properly oriented if they look askew. (this will be fixed later)
If the program is using the wrong webcam, try the other build version v3/v4
Alternatively open project source in Unity and take a look under the hood and build for yourself (regular export process for Unity, one click build)
Comments
Log in with itch.io to leave a comment.
Oh my god this is insanely cool!
Thanks!