Designing for natural experiences
Last week, we discussed the emerging world of natural user experiences. Digital services that are increasingly conversational, personal, agentive, and embedded.
As if on cue, Apple took it up a notch later in the week by introducing Apple Intelligence.
“Finally!” some industry observers exclaimed. Apple stepped into the arena of generative AI in its grandiose style.
Apple's vision is to combine personal context—what is on our phones—with the most powerful generative AI models, like GPT-4o, to help us get stuff done.
Apple Intelligence brings generative AI to the core of the Apple experience. (Image: Apple)
Siri will finally hopefully understand us and take action across all of our apps. For example, you can ask the assistant to “Play that podcast Jim recommended”.
The small on-device AI model will handle simple tasks, like editing your writing to sound more casual. A state-of-the-art model, now GPT-4o, will handle more complex tasks, like explaining the importance of the Renaissance to a 6-year-old.
Andrej Karpathy, formerly of OpenAI, summarized the launch succinctly (below).
In short, we’re getting closer to a world where we’ll just say things to our devices, and they’ll get it done. They’ll even proactively tell us when it might make sense to do something. The comparisons to the Spike Jonze movie Her are not so far-fetched anymore.
What will this mean for those of us who design digital services? What are the emerging principles of designing well for these natural experiences?
It’s still early, of course, but I think there are five key considerations:
Aligning goals
Anticipating needs
Balancing control
Transparent explanations
Seamless multimodality
Aligning goals
Traditionally, we’ve accomplished things by navigating digital services manually.
Booking a dinner and a movie? We browse different, often clunky, movie theater and restaurant sites and coordinate with friends on messages to get the night organized.
Soon, we’ll ask our AI assistant to book us a highly-rated movie and a hot new restaurant that fits our schedules.
We’re going from navigating services to directing AI assistants to reach our goals.
This change leads us to the first key to success in designing natural experiences - aligning closely with user goals.
The models and digital services must understand our goals as they fulfill them. We’ll move from designing steps in a flow that the user navigates to multi-step processes that fulfill users' goals without necessarily their input.
The AI-powered services must have high-quality contextual awareness of needs and goals to serve us well.
2. Anticipating needs
Increasingly, digital services will be able to anticipate our needs. We might get a reminder to answer an important message or book a time to switch to our winter tires based on the weather.
To do this well, we’ll need to understand the full user journey. When would be the most relevant time to notify the user? This requires a fine balance between being helpful and not intrusive.
As designers, we’ll have to look beyond the app or website to see when and where the user might need help. Apps might already do this with simple things like notifications about a new TV show might like to watch based on our past favorites.
In the near future, digital services will be able to use the personal context of our phones to deliver (with permission) increasingly relevant help before we ask for it.
For example, a fitness app could nudge us to leave early for a meeting across town and walk instead of taking the subway.
In its keynote, Apple showed an example of highlighting the landing time of the user’s mom it got from a message. To continue the example, the phone could suggest leaving early to account for a change in the flight time and traffic.
Apple showed how Siri can locate details on an upcoming flight. (Image: Apple)
3. Balancing control
Another balancing act is between human control and AI autonomy.
For agentive experiences to be helpful, some level of control will be ceded to the AI assistant. Gone too far, however, the experience might become unhelpful and, at worst, dangerous.
If we are to trust the AI assistant to do personal things for us, we’ll need to know it won’t go haywire and send the wrong message to the wrong person.
Humans must still regain overall control of the experience. The AI will only complete actions as directed and not operate with complete autonomy.
4. Transparent explanations
A closely tied concept is transparency. We’ll most likely want the AI to explain what is happening.
Continuing our restaurant example, the AI might ask, “To reserve a table at a restaurant, I’ll first browse the web for the best restaurants and cross-check their availability with your calendar. Is that OK?”
Generative AI is largely a black box to humans, so it’s increasingly important to provide an understandable, high-level explanation of what’s going on.
5. Seamless multimodality
In our conversations between people, we intuitively switch between audio, text, video, and images based on the context.
Over the next few years, this opportunity will increasingly exist in digital services. If we’re chatting with an AI assistant about a kitchen renovation, it makes sense to show our current kitchen on a live video feed from our phones.
Foreboding the opportunities to come, someone hooked up their Home Assistant-powered security camera to GPT-4 vision to find things around their house. (via @Shpigford on X).
As designers of digital services, we’ll have to become familiar with blending modalities from audio to text and video seamlessly. This will take some practice as we’re so accustomed to the current graphical UI-based paradigm.
From science fiction to daily life
Apple didn’t invent the smartphone or tablet. Instead, it packaged the new capabilities in a device with a far superior user experience than the competition, paving the way for an explosion in the category.
Will the same Apple effect hold true for AI?
Up until now, access to AI models has mostly required using a dedicated app like ChatGPT. Although wildly successful, this has limited their user base to relatively early adaptors.
With Apple Intelligence, generative AI will be baked into over 2 billion Apple devices worldwide. Most people will not know or care that they’re using generative AI; they’ll just go about their day, and an assistant will increasingly help them get stuff done.
AI assistants are moving quickly from science fiction like Her to something we take for granted in our daily lives.
To design experiences for this future, we’ll have to update our mental models. We’re no longer designing passive experiences navigated on a graphical UI.
We’re designing conversational, personal, agentive, and embedded experiences. We’ll have to think deeply about aligning for users goals, anticipating their needs, balancing AI vs human control, how to explain things and how to blend modalities.
It’s never been a more exciting time to be a designer.
PS. Thanks to everyone who already signed up for the Designer 2.0 Live training in October.
Seats are limited, but they are available for an early-bird price until June 30. We’ll have a chance to dive deeper into designing for natural experiences and using Gen AI in design work.
