Patent drawings show a HomePod with cameras, which could now become the anticipated Apple HomeHub
HomePod owners may not even need to call out the word “Siri” in the future, with Apple looking for ways to use gaze detection to let a device know it’s being searched for.
If you own multiple Apple devices, you know it’s difficult to get Siri to respond on the one you want. When you’re in a room with an iPhone, an iPad, and a HomePod mini, Apple has all kinds of systems to gauge which device you want, but they routinely fail.
Additionally, not everyone feels comfortable with the “Siri” prompt, even though it is better than the original “Hey Siri.” You can still pronounce either version, just like your TV. It’s common for something said on a show to be close enough to “Siri” to prompt a request you didn’t ask for.
It’s also possible that users may need to interact with devices without using their voice at all. There may of course be situations in which a command must be issued remotely, or in which it may be socially awkward to speak to the device.
In a recently issued patent titled “Controlling Devices Using Gaze Information,” Apple suggests that it might be possible to control Siri visually. Specifically, it proposes that devices can detect a user’s gaze to determine whether that user wants that device to respond.
It would take HomePods, or other devices, with cameras and other sensors that can determine a user’s location and gaze trajectory to determine what they’re looking at. This information could be used to automatically configure the device under review to enter an instruction-accepting mode where it actively listens, in the hope that instructions will be given to it.
This would be similar to how an iPhone’s “always on” screen will turn off until you look at it. So there already exists a device capable of detecting when we look at it.
Apple could extend this to interpret a look as being the equivalent of a verbal trigger. Users could still call “Siri” if they weren’t looking at the device, but this would give them an additional option.
Using gaze as a barometer for whether the user wants to give a command to the digital assistant is also useful in other ways. For example, gaze detected while looking at the device could confirm that the user actively intends the device to follow commands.
A digital assistant for HomePod could potentially only interpret a command if the user looks at it, the patent suggests.
In practical terms, this could mean the difference between the device interpreting a sentence fragment such as “play Elvis” as a command or as part of a conversation that it would otherwise have to ignore.
The patent filing mentions that just looking at the device does not necessarily mean it is listening to instructions, as a set of “activation criteria” must be met. This could simply consist of a continuous gaze for a period of time, such as a second, to eliminate minor glances or false positives from a person turning their head.
The angle of the user’s head is also important. For example, if the device is located on a nightstand and the user is asleep in bed, the device could potentially consider the user facing the device as looking based on how they are standing, but could dismiss it as such to realize that the user’s head is sideways instead of vertical.
This would be a judgment taking into account whether the user’s eyes were open and what the angle was. This would prevent unintentional glances from triggering Siri, but as useful as that might be, then there is a concomitant problem.
This means that if the device does not always respond to a glance, the user must know whether this is the case or not. It can’t be that the user says a long command only to have Siri say, “Are you talking to me?”
So, in response to an intentional gaze, a device could provide the user with a number of indicators that the assistant was activated by a gaze, such as a noise or light pattern from built-in LEDs or a screen.
Given the ability to record a user’s gaze, it would also be possible for the system to detect whether the user is looking at an object they wish to interact with, rather than the device containing the virtual assistant.
For example, a user could look at one of multiple lamps in the room, and the device could use the context of the user’s gaze to determine which lamp the user wants to turn on from a command.
The patent also describes situations in which the user can begin a command before turning to the device to complete it. Or, that they can look at the device and turn away before completing the order.
In both cases, the device must have determined that it is indeed the person being addressed. But if the gaze is not detected immediately, Apple suggests that the device could ask the user what they want and thus attract their attention.
Nearby devices could detect the user’s gaze on other controllable objects in a room.
This isn’t the first time Apple has asked parents to be able to remotely operate a device, which has appeared several times in previous patent filings. For example, a 2015 patent for “learning-based estimation of hand and finger pose” suggested the use of a 3D optical mapping system for hand gestures, which may well have led to the Apple Vision Pro.
The new patent, however, is also not the first time Apple has been awarded one on this exact same patent. A version of it was initially filed in 2019 and then was granted in 2020.
Apple files hundreds of patents every year, and it’s not uncommon for it to reapply even after a patent is obtained. There may, for example, be a minor but significant update in the new version.
In this case, the fact that Apple initially filed for a patent in 2019 means it has had time to see at least parts of it become shipping products. Face ID had already launched on the 2017 iPhone
It remains true, however, that even the existence of repeated patents on the same technology does not constitute proof that this specific idea will ever be launched. But the patent drawings consistently show a HomePod, and we’re now closer to having a similar HomeHub device that might well have cameras.
Originally filed on August 28, 2019, the patent lists its inventors as Sean B. Kelly, Felipe Bacim De Araujo E Silva, and Karlin Y. Bark.