Why LLM Users Still Need Apps
The theory others propose: it will be the "Death of Apps" tomorrow, or maybe next Tuesday. We will arrive at a place where users speak their wishes to a well-connected LLM, and it just gets done. No knob twiddling, button pushing, or even a glance at the UI of an application.
While I am no naysayer of conversational interfaces, I don't think app interfaces are going away. I'll give you the two big reasons.
1. Apps Deliver Precision Interfaces for Users
I use Krita, (their website) an open-source Photoshop-style app, to create images. Its screen has a UI that is something like the cockpit of an airplane. Each little icon I might press needs at least 5 minutes of newbie time to learn how it works. And there are countless panels and dialogs that expand the possibility space of Krita well beyond what is immediately visible within its UI.
I love Krita, but compared to the glorious AI future on offer, it does seem pretty complex and tedious, right?
Could we not just lounge in our chair while being fed grapes and call to our AI servant, "Craft me an image. And be quick about it!"
For the image above, that's basically how I started using Midjourney. You can see one of the more successful prompts below that I built from.
Midjourney did a fine job of generating different candidate images matching my prompt. I picked one and continued refining. It's important to note that Midjourney is very much an app that allows specification beyond prompting. I used UI-presented features like upscaling, regional masking, and character references to arrive at a base image that showed Bill Gates handing a bag of trash to the heroic woman from Apple's "1984" commercial (YouTube). And then I overlayed text in the exact font and position I wanted using Krita.
My point is not to impress you with my dime-a-dozen photo manipulation skills. Rather I am calling out the usefulness of application UI when combined with generative AI.
As a user, you can say what you want in language, and that feels quite natural. But just because speaking or writing is a familiar activity to us, doesn't mean it's the most efficient way to specify every desired outcome. The visual interface of drawing a rectangle over the blonde man's head to select it for replacement is fast, precise, and hard to beat with a conversational interface.
Imagine choosing a specific color. Do you really want to have a conversation about the desired color? ("Uh... a stormy kind of red or actually... mauve, but not a dark mauve...") Or do you want to pick it from a swatch? If I care about the color, I'll take the swatch. Maybe if I memorized the names of all those Crayola crayons when I was a kid, I'd feel different.
The second reason why we need apps for LLM use cases is even more important.
2. Full Automation with LLMs is Usually a Bad Idea
The dream that seems to fund a thousand AI startups is full automation. Ask the bot to build a website, book a flight, write a legal document, invest in stocks, and the bot just handles it for you. And you can spend all your free time watching your favorite AI-generated movies.
But due to hallucinations and errors, we don't actually want LLMs to do our work in a fully automated way. Despite promises that the accuracy problems are going away, I believe it will take a real breakthrough to get there - not just interative improvements to LLMs with more scaling and training. And you don't put AI breakthroughs as a scheduled event on a roadmap. They happen when they happen.
Depending on the task and model, your LLM is going to be right 50% to 99% of the time, but never 100%. If the task has a consequential result, (e.g., booking an airplane flight) you must have a human at least verifying the proposed action before it is performed. And that brings us back to the usefulness of apps with their UIs for presenting information and interacting with users.
There are also times when work performed by an LLM can be simplied by the structure of an app. It's a very challenging problem for an LLM+RAG system to index and retrieve a large corporation's tens of thousands of documents. (I believe nobody is doing it well currently, including Microsoft.)
But if you had an app where the user curated a list of documents that were important just to themself, the LLM+RAG system need only deal with hundreds of documents. And docs from that smaller collection can be retrieved with more accuracy. This simple trick of limiting the context to something smaller is key to handling LLM use cases successfully.
Setting Sights on What is Possible Now
Rather than assume apps will be wiped out by "just do it" commands given to bots, let's meet the technology where it is right now. We can make LLM-based apps do useful things for actual people, rather than make claims about what we might eventually be able to do. With this practical attitude, there are countless opportunities to create excellent software.