This is one of the many uses for Expressive Pixels . The Windows 10 development platform that lets you create animated renderings is now available for free in the Microsoft Store. In addition to the application, it has a set of software thanks to which you can use LED displays for communication using visual effects or emoji.
All of this expands the possibilities for non-verbal communication, opens up new ways for developers to use creatively through the open source API, and lowers the barrier to entry for aspiring programmers, designers and researchers.
Expressive Pixels also serves as a springboard for hobbyists or more experienced programmers, allowing them to create new creations without having to figure out how to program animations and images on small devices - there's a dedicated library for that.
No matter where you work, study or socialize, Expressive Pixels offers an animated way to personalize your space and amplify your presence, just like images, souvenirs and various knickknacks do.
“We pay a lot of attention to basic everyday functions, but when you think about what makes us human, it's emotions. We want to express our emotions and keep in touch with each other, ”said Bernice Yu, general manager, strategy and projects, Microsoft Small, Medium & Corporate Business. She is part of the Expressive Pixels project team and has contributed a lot to the 2019 release of Eyes First , which uses Windows 10 Eye Control , a key accessibility feature for people with visual impairments and mobility impairments. “It's great to be productive, but we all want to be human. This is an inviolable need. "
Expressive Pixels builds on years of deep and fruitful collaboration from Microsoft with the Enable Group and people with severe visual impairment and mobility. The purpose of this work is to try to understand the preferences of such people, their needs and problems arising in communication and interaction with other people, in creative self-expression and self-identification, as well as to expand the capabilities of traditional Augmentative and Alternate Communication (AAC) devices with new functions for non-verbal communication, self-expression, broadcasting social signals and device status.
Examples of Expressive Pixels
Communication is not just what you write and say. This is largely a non-verbal process that includes knowing when it’s your turn to speak.
Harish Kulkarni
“The solution can be used for serious cases, or as a fun creative way to tell something about yourself,” says Harish Kulkarni, technical manager of the AI frameworks team in the cloud computing and Microsoft Cloud and AI group. He's been a member of the Enable team for several years and has done a lot to integrate Eye Control into Windows 10.
Along the way, the team embraced the most unexpected innovations, overcoming obstacles with fresh ideas, inspired by the benefits that technology can bring to people.
When developers realized they needed a form factor that could be easily seen from anywhere in the room and in any lighting, they turned to user-friendly LED displays offered by third-party manufacturers such as Sparkfun, Adafruit, and Silicon Squared Displays (latest displays are custom all-in-one devices designed to solve resolution, communication, storage, and animation issues by sole engineer on the Expressive Pixels team, Gavin Jahnke). However, Expressive Pixels can be used without an LED display or other device by simply inserting animated GIFs into emails.
Then another problem arose: developing software that would make it easier to display animation on displays. Once again, Jahnke took up the challenge and created a firmware that, together with a development application, forms a platform that can be run on multiple devices. With this firmware, manufacturers can use certain aspects or extend them. Creators can share their original animations in the Cloud Gallery of the app.
Jahnke was instrumental in the development of these key elements of Expressive Pixels - all in his spare time, outside of the main job.
Silicon Squared Display: Prototypes and Display Evolution
The first version of the device was a simple display of 16 pixels arranged in a circle. To make this prototype, a cutter was used - circular pieces of translucent material were cut out, which covered the displays in order to diffuse the bright light from the LEDs.
“Usually, when I’m doing something, I make a royal dish out of it,” jokes Jahnke, general manager of development for Microsoft Research. But he gets serious and adds: "My vocation is to help those segments of the population and communities that do not receive the proper service, to give them what is not available to them due to their technical capabilities, but which is within my technical capabilities."
Jahnke is a jack of all trades, having acquired new skills, including in the field of electrical engineering, over the three years devoted to the project. Along the way, he created a new category of devices with a higher resolution RGB LED display than was previously available. He also figured out how to provide users with maximum mobility using Bluetooth technology and various mechanisms to trigger animations, such as switches or MIDI music signals.
“What's really interesting is the unintended consequences of getting out of your comfort zone and learning something new,” he says. "You get amazing ideas and abilities."
The Expressive Pixels team meeting in Microsoft Teams in Together Mode. Back row (left to right): Anne Paradiso, Bernice Y. Second row from top (left to right): Noel Sophie, Gavin Yanke, Jarnael Chaj. Third row from top: Duane Lamb, Stacey Stutz. Bottom row: Christopher O'Dowd.
The Expressive Pixels journey began more than five years ago with the Enable Group's interest in improving Augmentative and Alternate Communication (AAC) devices, which are usually too expensive because they are part of the medical market.
Expressive Pixels across devices.
Chief Development Officer Anne Paradiso took the lead in this effort, wanting to channel her energies into serving others. She worked with Jahnke's team at Microsoft Research before joining the Enable Group, and Jahnke did not refuse her help, especially as the project was gaining momentum.
They first worked with former NFL player Steve Gleason, who inspired and stimulated research after he was diagnosed with ALS (amyotrophic lateral sclerosis). While Gleason was adept at using eye tracking technology, he still found it difficult to communicate when technical problems arose and had to look at the screen rather than the person trying to communicate with him.
“When talking to a person who only controls a voice device with their eyes, you will notice that the pace of the conversation is much slower compared to normal speech. Often conversations over AAC are 12 to 25 times slower, even with better predictive accuracy and eye movement recognition, says Paradiso. “So we often observe how the interlocutor either cannot stand the prolonged silence, or cannot understand that they are preparing an answer for him, and continues to speak, and the person with speech impairments is left without the opportunity to speak.”
This led to user-centered research involving people with ALS and their families. Paradiso and her staff met with them regularly, accompanied them to appointments with doctors, observed their interactions with neurologists, speech therapists, physical therapists and many others. The meetings were held both at home and in Microsoft laboratories. The resulting trust has allowed researchers to truly get to know the ecosystem around people with disabilities, learn about support systems and the attitudes of family members, examine equipment, and find out what works and what doesn't.
We quickly realized that optimizing LED displays for emoji-based communication was a great opportunity.
Anne Paradiso
“Emojis are already ubiquitous in digital communication platforms, including text messaging, email and social media. They take less action to convey intent, context, or mood. A single emoji icon can drastically change or improve the perception of a message. We think they can be used as additional expressive mediators for people who are unable to speak or control the muscles responsible for facial expressions, says Paradiso. - The people with ALS who work with us are some of the funniest, most thoughtful and creative people I know, but the opportunities available to them to express their thoughts are limited due to the disabilities and imperfections of the available voice devices and the technologies underlying them. We know,that people want to express a lot more with their AAC devices, not just communicate in simple phrases. We wanted to create something that would help people participate more actively in conversations, would be visible in low light and from a distance, and would also provide another opportunity for unique expression, communication and transmission of emotions. "
The developers tried various additional screens, but returned to LED displays for several reasons: they are inexpensive, they work reasonably well, and they are “cool”. The people with ALS in the research also made it clear that they don't want to use anything that could have unintended negative social consequences for users.
“Observing and empowering people living with ALS so that they can do what they never hoped for is motivation enough to work hard,” said Duane Lamb, a developer who joined Enable Group in early 2017, who creates user interface and user interface. - More often than not, when you are talking to someone who can only communicate with their eyes, that person enters text while looking at the keyboard of the device in front of him. And you, although this is bad form, from the point of view of social etiquette, usually look over your interlocutor's shoulder in an attempt to see what he is typing. "
The emergence of Expressive Pixels is partly due to the desire to solve this problem.
Emoji in the Expressive Pixels app
Expressive Pixels can be used to animate displays of various sizes, up to 64 × 64 pixels, said Christopher O'Dowd, who helped fill the hardware gaps in the project. O'Dowd noted that LED displays are ubiquitous at Maker Faire, in homes during the holidays and more. LED displays are so versatile that they can often be found even on fabric (such as masks, caps, and backpacks) or banners.
One of the first applications of LED displays by the Expressive Pixels team was the award-winning SXSW Music without Hands project .
In this project, a specially designed LED matrix with midi support and music sync was used. It was an optional visualization tool for an eye-controlled physical drum kit designed for one of the Expressive Pixels members, a Seattle musician living with ALS. Winner of the 2018 SXSW Interactive Innovation Award: Music and Audio Innovation, the project includes a suite of new eye-guided applications for composing and performing music and collaborating.
“How can a person who is unable to speak and move to compose or perform music, command what is happening on stage or contact a live audience? What about performing together with other musicians, rehearsed or improvised? How can we lower barriers to making school music programs more inclusive, not separating or minimizing the participation of children with disabilities? We were driven by the desire to find answers to these fundamental questions, - recalls Paradiso. "We wanted to adapt our technology and design so that they fit the person's creative goals and real-life situations, and not vice versa."
Lamb had the idea to add support for Musical Instrument Digital Interface (MIDI) to send different signals to different instruments, and this idea was later carried over to Expressive Pixels.
“Every time we worked on a new goal, we pushed the platform further,” says Paradiso.
The Expressive Pixels platform is open to a wide range of creative people. This is not only a community of developers, but also students who can develop projects to help other people and learn JavaScript using Microsoft MakeCode . According to Jahnke, MakeCodes typically have a very small number of LEDs, so full display capabilities open up a lot of scope for students to create, experiment, and program using something like Silicon Squared displays connected to a MakeCode gadget.
“Expressive Pixels gives students an easy way to integrate in-app animations into their MakeCode programs and hardware creations with a minimum of effort,” says Jahnke.
The application itself will know what animations are stored on the device, so different commands will cause it to react accordingly.
This and other Enable projects in Paradiso rekindled her passion for the company she has been with for almost 20 years.
“So many people come together, and many spend their personal time to work for something greater than each of us on our own - it inspires and aligns with the values and mission of our company, and it makes me happy and gives hope for a more inclusive and creative future for all, ”said Paradiso.
Expressive Pixels.