Heres a novel fix for the headache of interacting withall sorts ofconnected devices: researchers at Carnegie Mellon University have devised a system that lets smartphone users tap their phoneagainst an IoT device in order to have a contextual menu automatically loaded on screen thereby saving them from having to scramble around looking for the correct app to control eachdevice. Or fiddling with actual physical buttons and trying to navigate less consumer friendly menus.
So, in other words,no more scrolling through endless pages of apps on your phone to control with your supposed smart home’, as CMU researcher Chris Harrison puts it.
The system, called Deus EM-Machina (see what they did there?), leveragesthe fact thatelectromagnetic noise is emitted from everyday electrical objects to power a device classifier theyre usinga smartphone kitted out with an EM-sensor that can detect what IoT device its resting on enablingcontextual functionality to be pushed tothe smartphone screen so it can be a dynamiccontrol device.
Theres no reason this couldnt be integrated into smartphones in the next few years, saysHarrison, anticipating miniaturization of the EM sensing tech to pave the way for future commercialdeployment.
And while researchers at CMUs Future Interfaces Group havepreviously shown a similar electromagnetic sensing system running on awearable device also for powering contextual awareness of otherdevices the use of a smartphone as the control device in thislatest researchscenario meansricher menus canbe made available to users, allowing more control functions to be supported.
Harrison confirms the latest EM sensing research is an extension of the earlier wearables project, telling TechCrunchit has improved accuracy, as well as being much smaller and self contained. But the big difference is the exploration of a very different form factor and how that can be utilized at the application layer.
It isnt that one project/technology is better than the other, its more that switching form factors opened up different interactive opportunities, he says. In particular, unlike a smartwatch, you perform distinct activities on a smartphone. Its more of a tool than an accessory.
In a researchpaperon the new research, the team writes:
We propose an approach where users simply tap a smartphone to an appliance to discover and rapidly utilize contextual functionality. To achieve this, our prototype smartphone recognizes physical contact with uninstrumented appliances, and summons appliance-specific interfaces. Our user study suggests high accuracy 98.8% recognition accuracy among 17 appliances. Finally, to underscore the immediate feasibility and utility of our system, we built twelve example applications, including six fully functional end-to-end demonstrations
Examples of the apps the researchers built to demonstrate the sensing systemare shown in the below video including controlling a thermostat; configuringa router; printing a document thats on screen on the mobile device with a single print button push; sending text from a mobile to a desktop computer; and more.
The researchers created a background Android service running alongside their IoT device classifier that pushes so-called contextual charms onto the screen for certainapplications aka small floating buttons that appear at the right edge of an app when the phone touches a supported device, and which can execute commands (such as a cast charm to stream video content to a smart TV, or a print button to print whats on screen).
We envision that future smart appliance applications would register their devices EM signature and a set of verbs with the charm system service upon installation, which would enable existing apps to immediately take advantage of appliances and devices in a users environment. This is analogous to the current paradigm of applications registering Android share handlers to support system-wide sharing of content to e.g., social media, they write on this.
Harrison adds that the contextual charms idea was developed for wherethere are co-occurrences of two things such as reading a PDF on your phone, then tappingit on aprinter. This all of a sudden opens up a much richer set of synergistic activities between phones and devices whereas right now, your phone doesnt really have any clue what is going on around it or what functionality it should reveal), he adds.
Discussing limitations of the system in general in their paper, the researchers emphasize the need forIoT devices to haveopen APIs, noting:We initially set out to produce full-stack implementations for all of the network-connected devices on our list. However, we were stymied by the lack of public APIs on several of them. Furthermore, even when APIs were available, some were vendor-locked (e.g., the Apple TV casting APIs were only open to Apple devices). In order for the future Internet of Things to have true impact, open APIs are a strong requirement. Until then, our system will be limited by the inability to talk to all smart devices.
Other limitations include difficulties recognizing multiple instances of the same device (e.g. more than one connected thermostat); and external interference from powerline noise which can confuse the device classifier. The sensing system also cannot work if a device is truly powered off although the researchers note that low power or sleep modes might still render an IoT object detectable.
The research is being presented this week at theACM CHI Conference in Denver.CMU isalso presentinganother interesting bit of interface research at the conference which involves usinga conductive spray paint and an array of electrodes to turn any surface into a touch-sensitive surface.