Apple Intelligence Initial Analysis: Apple Shows the Market How AI Should Be Done on Devices

Apple Intelligence Initial Analysis: Apple Shows the Market How AI Should Be Done on Devices

AI features shared at WWDC 2024 include:

Writing Tools. For rewriting, proofreading and summarizing text. Apple will implement the writing tools as system-level services, available across Mail, Notes, Pages and in third-party apps. I once asked Steve Ballmer why Microsoft had not implemented a spellchecker in Windows (given that many HDs at the time held multiple calendars for app-specific spellchecking). His response: “No one needs a spellchecker in a browser.” Well, now they do. And in almost every other app. Writing tools will provide thread summaries in e-mail, not just summaries on individual e-mails.

Priority Notifications. These will also be managed by Apple Intelligence, ensuring that the most important ones appear.

iPhone Audio Transcription and Summarization. The iPhone will become a professional conferencing system with the ability to record, transcribe and summarize audio. However, I don’t believe FaceTime calls were included. Those features can’t be far behind. Everyone on a call will be notified when the recording starts. This is not an AI feature but a smart transparency choice.

Image Playground. Write a note. Circle a blank space using the Image Wand, and Image Playground will extract information from the note page to inspire an illustration. That is an example of thoughtful teams at Apple taking the time to figure out how to embed AI without shouting its presence in the UI. Images will be created on the device and can be rendered as animation, sketches or illustrations. The feature will also transform sketches into more finished artwork. Image Playground may be the end of many clipart services.

Image Playground will even use an image from the owner’s personal library to create a rendering of a friend or family member. I will not be surprised if Apple doesn’t have some privacy discussions around this feature as people challenge what can be done with their likeness.

Genmojis will handle smaller images by creating custom emojis—so much for the emoji committee. Apple devices will be capable of crafting custom emojis that include likenesses of the sender or receiver with just about anything you can imagine tacked on to them. Dan wearing a Dodger cap? Sure. Janet wearing aviators? Why not? Genmojis will be available as emojis, tapbacks and stickers.

No one will be surprised that Apple is incorporating more AI. With Samsung and Google already in the AI photo game, the third space for photos, Apple’s Photos, now facilitates the removal of people and objects that mar an image. Generative AI can be used to create a memory movie from online photos. As the generative organization of images becomes more sophisticated, I can imagine a hit to montage makers for weddings and bar mitzvahs.

I can imagine a hit to montage makers for weddings and bar mitzvahs as the generative organization of images becomes more sophisticated.

Siri reimagined. So far, these features are not Siri-based, meaning that AI has become infused within apps and experiences. Siri is no longer the primary entry point for Apple Intelligence. That said, Apple did not forget Siri. Siri now acts as a proper assistant. And like the other Apple Intelligence features, its glowing orb disappears, becoming a flash across the display.

While Siri becomes more invisible, what it can see expands. Siri will now be aware of previously seen content and the current content on the display. Phrases like “Bring up that article I was reading on backpropagation,” and the article will appear; “Summarize the web page” should also work.

The most important feature isn’t the local LLM but the idea of Personal Intelligence. This semantic index creates a representation of the owner through relationships between data types and an understanding of what those data types mean to each other. Today, Siri can’t handle the query, “When is Mom’s flight landing?” Current Siri returns a web search that includes mothers on TikTok talking about plane landings. It’s not very helpful for those picking up Mom from the airport.

With the semantic index, Apple devices will recognize data like who “Mom” is and find e-mails from her or saved files with today’s date and flight information. Follow-up queries like “How long will it take to pick up Mom?” will also likely work, recognizing that Mom is at a particular airport and where the device’s owner is at the moment.

Interestingly, Microsoft’s AI Recall + Copilot seeks to do many of the same things that Apple discussed. Apple, however, shared the semantic index in support of end-user experiences with repeated shoutouts to privacy. Microsoft already saw a backlash because it doesn’t learn from its own history. Recall suffered from the same botched marketing as the initial Viva launch: a lack of user and privacy first design and messaging. Apple, on the other hand, touted privacy, and they did so in the context of selling an idea that won’t hit devices for several months, giving them time to set and adjust expectations.

Apple also gets that basic generative AI, words and images, are already table stakes. In many ways, generative AI already verges on boring for some, and for others, it creates new friction as people seek free ways to leverage the tools through clipboards and files rather than expensive subscriptions. Generative AI, while it still has a wild side, is good enough to support most commercial writing scenarios and, with some caution (like adjusting for abhorrent spelling), is pretty good for illustration.

The current state of AI isn’t good enough, and Apple knows that. Real AI needs to understand its users and act on their behalf, understand intent, and find a way to work with data to fulfill that intent. None of the AI systems can do that yet. Apple promised personal intelligence-informed collaboration today at WWDC.

OpenAI can’t move the needle without Apple or Microsoft because they don’t own the data or the devices. Apple’s local-first approach offers the best of both worlds: proprietary, device and data-aware applications and the choice to go beyond.

ChatGPT integration. Initially, that choice to go beyond on-device processing involves a conscious decision to engage with OpenAI’s ChatGPT. Regardless of the LLM model compression, Apple can’t fit all the world’s knowledge onto a device. Apple chose ChatGPT integration, but not without asking first. Apple claims not to send anything beyond the query to OpenAI’s servers. The privacy-first model extends to partitioning queries sent to third-party services.

The outreach to ChatGPT does not require a subscription, although those with a subscription can incorporate advanced features into their ask.

None of these features are available today on any Apple device except for developers. Public betas will begin soon, giving regular users the ability to test them. They won’t be fully baked until this fall, and some will likely arrive well after that. I highly encourage readers to explore the public betas on a secondary device.

Learn what you can and give feedback. That’s what betas are for. Don’t jump onto forums to complain or rant. Send thoughtful feedback on the experience to Apple through the mechanisms provided in the betas.

Apple Intelligence: Implications for the Market

Apple holds a unique position in the AI market. It offers phones, tablets and laptops with market share beyond those of rival Samsung and Google, at least when it comes to tablets and laptops. Though Android OS leads in devices, its hodgepodge of implementations leaves leading-edge features to leading-edge companies like Samsung and Android maker Google. Eventually, Android devices will reach feature parity with today’s announcements, but it will be years before second-tier owner devices with the right chips and enough RAM can do much with it.

Microsoft doesn’t play in the phone market anymore, so it doesn’t own personal images like Apple, Google and Samsung, making any Microsoft AI impact on personal photo repositories and image editing negligible. The other three will keep upping the ante on image editing and storytelling.

While Microsoft does own productivity, its clever Apple team leans into Cupertino’s innovations. This means that free Apple Intelligence will likely enhance Microsoft’s productivity suite on macOS and iOS/iPad OS before free AI becomes available to Microsoft’s productivity suite.

Apple Intelligence is going to force Microsoft to reconsider its Copilot revenue stream of $30 a month per user. Apple eventually forced Microsoft to give away Windows. Apple’s non-pricing model will likely force Microsoft to give away AI (though it may take Redmond a hot second to implement), significantly cutting into their big bets on AI to drive add-on sales. Microsoft will still be able to leverage Azure AI as a stream with developers further supported by the positively reviewed coding Copilot in GitHub. While Apple did announce servers today, their servers are aimed at protecting data for personal experiences, not at selling AI as a service to enterprise developers.

Apple’s free implementation (as part of a routine OS update) of an on-device semantic index to complement local processing and access to ChatGPT 4o in a very transparent, partitioned fashion—and the near invisibility of AI within the UI—strongly counter Microsoft’s approach of charging for app-level Copilots and brandishing Co-pilot with logos on Windows, Edge and AI PC keyboards.

Beyond pricing, Apple’s new privacy model will also likely result in changes to privacy expectations across the market, forcing everyone from Microsoft to Salesforce to reconsider how they hand off data to OpenAI’s ChatGPT and other models.

Apple’s foray into AI will also likely erode older “AI” tools like Grammarly, the popular grammar checker that has grown AI features of late. If the built-in system at the OS level does as good a job, there is no need to buy a service.

With Apple Intelligence, Apple is giving its M1 and better processors with the Neural Engines something serious to do. Those who haven’t adopted the new Apple Silicon will likely see this as the killer app that gets them to finally abandon their Intel-based Macs and older iPhones and iPads. For all those people wondering about Apple’s M1 endgame, this is it. The shift to Apple Silicon and the neural processors have created millions of devices ready for Apple’s AI—machines that won’t need to meet an end-of-life calculation for replacement value like many current PCs that will prove underpowered for handling on-device AI.

Apple moved its users to adopt its platform ahead of the need for AI, positioning them uniquely, at least for a few months, as enterprises decide if they want to cut their losses and abandon what would normally be mid-life PCs and replace them with AI PCs. With M1 and beyond, (and A17 Bionic and beyond on phones), all Apple devices are already AI devices.

Apple Intelligence: The Bottom Line, at least for now

I have long touted a few principles for intelligence devices. They need to recognize and leverage distributed intelligence, meaning they need to be part of an ecosystem of compute and data. Do what can be done locally but reach out and back to create better experiences. They need to create local indexes of knowledge based on what people do, have done, and the data they own.

The local indexes need to be in the form of semantic maps that index relationships, not just metadata. AI needs to interact with personal data very securely, though it also needs to give the owner of that data the ability to share with others at their discretion (rather than via guardrails that restrict user choice). Apple seems to align with these ideas.

Apple’s delay in entering the generative AI space reflects its choice to enter only when confident it can control the narrative, meaning it has tested its ability to deliver quality results.

Apple’s semantic index will likely be the most revolutionary component of Apple Intelligence, as it will change the fundamental assumptions about storing and retrieving content and interacting with devices. If it isn’t initially clear, people will need to unlearn years of constraints associated with windowing operating systems. Apple’s vision offers the first comprehensive view of next-generation personal computing in a way that looks inviting rather than frightening.

Developers have only had a few hours of experience so far. Over the next few months, we will all see how much of Apple’s vision from today was hype and how much of it is reality. Apple’s delay in entering the generative AI space reflects its choice to enter only when confident it can control the narrative, meaning it has tested its ability to deliver quality results.

The initial implementation of generative AI on Apple devices may fall short of its vision, but it will likely be more cohesive and understandable than any other approach the market will see this year.

For more serious insights on AI, click here.