Apple’s 2023 Worldwide Developer Conference got underway this morning at 10am Pacific time with a keynote full of product announcements like the M2 Ultra featured in an updated Mac Studio and the new Mac Pro. The most exciting news, though, that Cupertino took the opportunity to announce its new headset, the Vision Pro. It seems the rumors, which have been swirling for months, were mostly true, but in true Apple fashion, the most interesting aspects are what’s kept in the software.
The Vision Pro is Apple’s first augmented reality device — or in the company’s parlance, spatial computer. The idea is that rather than looking at a display like a typical VR headset, users will strap on the Vision Pro and look through the lenses, still seeing the area around them with the user interface overlaid, not unlike Microsoft’s current HoloLens 2 AR offering. Apple has been pushing augmented reality for years as part of iOS and iPadOS and companies have dabbled with it, with perhaps the most notable example being Pokémon Go.
Apple’s AR Operating System: visionOS
The main interface looks like a scaled-up version of the Apple Watch app launcher, and Dye says that in practice it will feel like the UI is a part of the room. UI elements have depth, respond to light, and cast shadows as a way to give a hint about the scale and distance of elements in the room. Voice controls include dictation and talking to Siri to open and close apps. While we haven’t gotten to try this out for ourselves yet, the demos seem to be pretty slick. The only thing we’re not totally sold on is the virtual keyboard, but Apple says that Vision Pro is compatible with Apple’s Bluetooth devices like Magic Keyboard and Magic Trackpad; hopefully that’s just their way of saying it supports standard Bluetooth accessories support like it is on iPadOS and macOS. And of course, you can control your Mac with the Vision Pro just by looking at it and placing its screen wherever you want in the room with a resolution up to 4K.
Apple says it filed over 5,000 patents during the development cycle for Vision Pro and visionOS. Dye says the whole visionOS experience feels like “magic,” but that word gets tossed around a lot, and we’re sure this is more the result of a lot of new research and not magic. All the same, visionOS seems pretty polished in demos, as apps effortlessly slide around to make room for newly opened ones, filling in the space around users as they navigate the system. Speaking of navigating, Apple’s demos included people walking around a room while wearing a Vision Pro headset, a handy byproduct of the AR focus of the device.
Applications, or “experiences” as Apple calls them, aren’t bound by a display, and they can be adjusted to be any virtual size or position around the user. Apps don’t remove one another, but just take up a different space in the room. Environments allow the UI to scale beyond the dimensions of the room and get an immersive feeling of being just about anywhere. Want to watch Ted Lasso from a spot on the lake shore? Apparently, that’s an option. The Digital Crown (another Apple Watch feature that Vision Pro borrows) lets users dial in just how immersed they are. Landscapes have been captured volumetrically so that they don’t get that flat photo movement effect that Facebook’s 3D photos often exhibit.
While most headsets need at least a little extra hardware to track hands, heads, and users in the room, Apple’s input design goals were based around not using any additional hardware. Dye said users will control it using nothing but eye movements, hand gestures, and voice commands. Eye tracking controls seem to basically mimic a mouse hover or cursor position, a concept we got to play with in an Alienware m15 R2 a few years back, though Apple showed of a host of sensors around the eyes of the Vision Pro to enhance accuracy. Flick to scroll, or pinch fingers together to select an object. The interface seems to be pretty intuitive and simple, a pair of words that typically describe Apple UI work. Everything seems to be just really polished.
Unlike VR headsets that necessarily isolate their users from the world around them, Vision Pro has features to let users continue living in the real world. That’s as much for the user as it is the people around them; as anybody who’s ever been startled or pranked while wearing a headset can attest. When someone is around, Vision Pro users can see them and thanks to an outward-facing display called EyeSight, those other people can see the eyes underneath the headset. EyeSight also gives cues to other people, showing them how immersed you are when you’re busy. Also, when someone is nearby, they’ll appear in the immersive view. Every little detail about this suggests that Apple has gone through an awful lot of scenarios, and suddenly the reported long-term investment starts to make sense.
The apps included are the same basic apps that are available on every other Apple device. Browse the web in Safari, send and receive SMS and iMessages, make phone calls with FaceTime, and sync all your data automatically with iCloud. Spatial audio with life-sized FaceTime calls seems like a great way to get immersed with other people, too. Apps aren’t just two-dimensional, either; Apple’s demo included receiving a 3D model attached to message in the Messages app and pulling it out to look at the wireframe on a table in the same room. If that’s not the promise of some crazy Minority Report-type stuff right there, this author doesn’t know what is.
Apple Vision Pro: M2 and 23 Megapixels
As for the Vision Pro itself, the hardware looks pretty spiffy. The shell is built from a big slab of aluminum, with a modular design that takes various head sizes and face shapes into account. The head band is a 3D knitted, single-piece design that’s meant to hold the main display and its flexible face mount close to the user’s eyes and the integrated audio solution strapped to their temples. While the default configuration doesn’t make room for eyeglasses, Apple collaborated with Zeiss to make magnetically-attached prescription lenses.
The display itself is a micro-OLED Apple Silicon backplane with an RGB arrangement involving the red sub-pixel sitting in a horizontal orientation above the green one. These OLED pixels are placed on an Apple-designed backplane with one of its in-house developed controllers running the show. Apple Technology Development Group VP Mike Rockwell says there are 64 pixels in the same space as an iPhone pixel, but iPhones have different pixel densities. The Vision Pro has a pair of displays “the size of a postage stamp” with a total of 23 megapixels. That’s more than a 4K display’s ~8.3 megapixels per eye, so it’s super dense, with support for 4K HDR video playback. All of that is viewed through a pair of three-element lenses.
Rockwell says Apple wanted the audio to sound like it was coming from the room itself, not the headset. The audio setup seems equally complex as a result, with a pair of “audio pods” that deliver sound not directly to the ears, but around the user. Spatial audio uses a proprietary technology that Apple calls Audio Ray Tracing, which includes sensors to read the room around the user and tune playback to match the environment. The result, Rockwell says, is that the user feels like the sound is from the environment. There are a host of other sensors built into the system for head and hand tracking, IR illuminators, downward-facing and side cameras, LiDAR, and more.
Needless to say, all of these sensors and a high-quality interface need a lot of computing power, and to do that Apple has endowed the Vision Pro with its M2 Arm64-based SoC. Found in the Mac mini, MacBook Pro 13″, and MacBook Air, the low-power processor handles most of the computational needs. Running in parallel is a new chip called R1, and it handles all of the real-time sensor input. It has to handle a dozen cameras, five sensors, and six microphones. Rockwell says that R1 is necessary to keep the response time low enough for sustained use, and that the combination of M2 and R1 can get new images to the display in just 12 milliseconds.
Being a first generation device, there’s a bit of clunkiness that early users will have to live with, specifically with how the device gets its power. Apple CEO Tim Cook talked up the aluminum-clad tethered battery pack like it was some kind of fashion accessory, but that means users will have to deal with wires. Users are going to feel that woven fabric-covered wire whenever they put the headset on. Because the battery is not integrated into the headset, the Vision Pro is lighter than it would be otherwise, and we don’t really take issue with that.
The biggest drawback to the battery pod is that Apple chose to not make the battery bigger; as it stands, the company says users can expect around two hours of runtime. The battery size is apparently a result of form over function, because Apple says that it’s intended to be pocketable. Cook hyped up the entertainment interface, saying that users will be immersed in movies and sporting events, but two hours doesn’t cut it for either of those. Most users will probably want to use Vision Pro while it’s plugged in, with only short bursts of tether-free activity, and the device does support such a configuration.
Vision Pro seems like a really slick and intuitive piece of kit with plenty of potential, but all of this tech doesn’t come cheap. Apple says Vision Pro will cost $3,499 when it’s available for purchase. That’s an awful lot of money, but to put it into perspective it’s the same price as the base model of Microsoft’s HoloLens 2. Where HoloLens is a business-focused product, Apple seems to be taking aim at consumers with deep pockets who are already invested in the fruit-flavored ecosystem. Will it sell? We’ll just have to find out; Vision Pro will be available starting early next year.
