Forget multi-touch, it’s time for the Instinctive Interface
We have seen the User Interface (UI) evolve from the days of the command prompt (C:>) to today’s rich graphical environment. The UI is the most important part of the user experience and the user’s ability to get stuff done easily, far more than the hardware on which it runs. We have seen a number of technologies appear to this end over the past few years but they’ve all fallen short of providing the ultimate user environment, at least the way I see it. Many of these technologies have done a good job of attempting to make our computing lives easier but have not quite reached the pinnacle. The mouse made a huge improvement over the command prompt but can only do so much for us. Speech recognition is the technology that shows such great promise in the area of UI but not the way it’s implemented today. Touch interfaces have recently expanded the ability to interact with the items on the screen, and multi-touch has the potential to get people excited.
I’m afraid that all of this is not enough, not even multi-touch, even though it’s the big buzz phrase these days. I think about how I work with my computers a lot and I realized that none of these things are going to give me the rich interaction that will make a difference in the way I work. I think it’s time to be thinking about what I term the Instinctive Interface (II), which I do believe is totally realizable today given current technology. So what is the II? One of the things that I notice about the way I work is that I quite often do the same things the same way for the most part. Imagine this scenario:
Most mornings I fire up my PC at 8am and check my email. My computer should be smart enough to learn from my actions which is what the Instinctive Interface provides my PC. I resume my computer at 8am and tell my PC "good morning" which flags this as a normal day. It opens up my email client, Outlook if that’s what I always use, or the web browser opened to GMail if I use that. It downloads new email from overnight if needed and presents it to me in the application. I can open an email by moving the cursor over it like I do now, or telling the computer to "open Kevin" since that’s who the email is from, or maybe I touch the email on the screen. All of these actions will do the same thing which makes my life much easier because all I want to do is read Kevin’s email. The II has readied mousing, speech and touch control to handle whatever I throw at it. It has learned over time what I normally do and it knows I will continue to do so this morning and it is focussed on letting me do it.
After I finish doing my email, or even before I’m done if there are too many emails to do them all, I want to go to Google Reader to check all the items from my RSS feeds overnight. I can open up Firefox or just say "check the feeds" or the equivalent and the II knows to fire up Firefox with the Google Reader page loaded. The key to the learning capabilities of the II is that just because I use Firefox doesn’t mean you do. If it’s learned from your actions that you use Opera or Internet Explorer then that’s what it will use for you. No overt training required, the II can learn volumes about your preferences and what you normally do just by paying attention when you do them. After just a short time of doing this the II can be working WITH you, not just for you. It will become a very intelligent personal assistant that works the way you do when you do. It’s always watching what you do and WHEN you do it as most people’s work days are very routine when it comes to schedule.
The Instinctive Interface can also learn what you normally do at a given location using GPS functionality that is integrated in a lot of devices. If you always turn off your wireless connectivity at Client A’s office due to security restrictions then the II will do that for you. If you always log into the hotspot when you hit your local Starbucks then the II will do that based on your location. It will also know where your 3 o’clock appointment is and based on how far you currently are from that destination will make sure the appointment reminder happens in plenty of time for you to drive there because it knows exactly how far away it is. You probably see where I’m going with this- your system will always be learning from your actions and will always be working with you as never before.
Think about how effective your gadget would be if it usually could figure out what you needed to do next. Sure you could use the various UI technologies to make your point but most of the time I believe the II could accurately determine what you were going to do and help you do it with little or no effort. The beauty of the II is it could be device independent. Maybe it’s helping you when you use your laptop but the same technology will reside on your cell phone and work there. No matter what gadget you use the II would not only help you get your things done but also be contributing to the learning portion. It would be smart enough to know that you use Opera Mini on your cell phone to check your RSS feeds and use that but using Firefox on your laptop as I described above. The point is that your technology would be helping you all the time and learning what you do and how you do it constantly. The cloud could help here big time. I am convinced that this could be done today with the current technology if the right people, people a whole lot smarter than me, would get to thinking about this. So come on, smart folks, lets get the Instinctive Interface going.
>>>One of the things that I notice about the way I work is that I quite often do the same things the same way for the most part.
Get out of my head, Kendrick!! What did I tell you about planting bugs in my mind?! eavesdropping fiend!!
I’ve been thinking about the same thing, almost included a few lines about it in a recent post, but cut the lines as verring too far off the topic.
And now YOU come out and say it.
I’m going to sue for mind plagiarism, I will!! I’ll have X-rays taken to show the judge and jury the bugs you’ve planted! (Probably designed by MS and fabbed by HTC, knowing you!!)
voice input is a long ways off. The problem is that it just does not work very well. While geeks love to create things they think mainstream would love like the non keyboard UMPC’s; guess what 350k sold in all of 07′ and 08′ looks just as weak? What is needed is just to make a simple touch type keyboard in a jacket size UMPC. There are creative ways yet to be designed that could take lessons from the Samsung sph p9200.
Pen input basically sucks for mainstream and voice is even more decades away if ever in our lifetimes. Technology is supposed to make our lives easier and the reality is that most people find the touch type keyboard easier than all these touch screens and voice attempts to change things.
I’d settle for a computer I could use in a dock, with a pen or with my fingers for multi-touch. Sometimes each of those is the best interface. Computers that anticipate my needs are fine for deciding what to put in RAM or knowing to do housekeeping when I’m plugged in and not when I’m walking around, but if an intuitive interface is like the “I’m guessing what you want” button on cell phones or the MS Office menus, it needs to be very good before you turn it loose.
This is something that is coming, though I’ve heard that programming it is much harder than it would seem. Its not just a matter of the OS/shell “knowing” what to do, but also learning what it is that you do best and then delievering the best interactions.
I would not be surprised if the next version of Windows offers this in a small way. I think to do this within the next 2-3 years for complete device interaction might freak out people more than it makes them go wow. That being said, mobile devices (tablets, smartphones, and feature phones) should be doing this already. They not only have the user data to do such things for items like call lists, web bookmarks, etc., but they are already personal enough that the bridge to such functionality is easier to receive.
Didn’t you basically just detail Microsoft Bob 2.0?
Thus far the marketplace has shown that people don’t like when their computer tries to guess at how it can help them. That said, if it had major advances in AI, I suppose that could change people’s minds.
Thank you James.
While I don’t think I want my computer “knowing” me that well (who’s to say it won’t “decide” a better way for me to do something?), I sure would like to be able to wake up in the morning and, as I’m stumbling to the bathroom, tell my computer to open my e-mail app or go to my feed app or start up Pandora.
Others might not like it, but I would much rather talk to my computer than touch it.
I’ve experimented with voice control applications, while they aren’t perfect, they are much further along than some people would like you to believe.
I think we’re going the other way. Vista is driving us back to XP and I think the command line is next. Interesting clip art. Is that the Fabeo interface or the Michael Bolton interface?
I’ve often had similar thoughts. Most of the time I’m working on the same small set of files, but over the course of a year I will gradually move from concentrating on one set of files to another. Why doesn’t the computer notice this? Why do I have to set up a system of short cuts when I find myself constantly negotiating a hierarchy of folders to get to the files I want.
As for voice input, I can’t see it. I notice the trend in office design (insanely in my view) is towards open plan, to the extent that if you want to do anything that requires thought you have to go somewhere off-site. I suppose ‘thought’ is an inefficient use of time and not to be encouraged. It would be 10 times worse if everyone was talking to their machines. I would, in any case, feel an idiot talking to a machine. But maybe that’s a generational thing.
If your computer guesses what you want to do, or learns what you do regularly, what happens when you need to do something different?
What happens when Windows’ Start Menu’s top level shortcuts suddenly include a program you only used a couple of times, bumping off a more-regularly-used program? (Why is there a 400×24 WM6 Emulator in my start menu shortcuts? I haven’t launched that program for the better part of a MONTH!)
What happens when Office helpfully hides all your menu items, causing you to click around extra to find the one that you need *right now*?
I don’t think we really want computers to guess what we want to do. We want computers to solve problems for us. We want to say, “computer, I have a problem, it is this: blah.” and we want the computer to say, “Okay, I have done X Y and Z.” (I’m getting an image of Scotty talking into a Macintosh mouse…)
*Other People* can’t guess what we want to do. How often do you get gifts that are only ‘the thought that counts’?
If computers could talk back like people, then maybe we would feel comfortable talking to them. If you could have a conversation with one, instead of having to repeat yourself like you do with those phone systems… “Hi, I’m the customer service phone droid! Tell me your home phone number.” “555-1212″ “I’m sorry, did you say 642-1975?”
Assuming Kevin White’s questions weren’t entirely rhetorical, I would say I didn’t have a problem with Windows Start Menu showing me only some of the installed programs, or with Office menus only showing me some items.
For me, the problem was that neither application told me what it was doing, nor offered me a quick and *explicit* route to the full choice if that’s what I wanted. Hence I was confused and exasperated when I first encountered it, as, in my experience, everyone is.
Now that I know these programs are offering me a limited choice, and know how to get the full choice, I don’t mind – and actually get a lot of benefit. But it has cost a lot of bad temper and frustration on my part to get to this point. That’s the trouble, as I see it.
Somehow, this post makes me think of the Terminator series and all the “mischief” that Cyberdyne lead…
What’s the first thing Vista users need to disable in order to get decent performance out of their machines ? SUPERFETCH. What’s the purpose of superfetch – to analyze and predict your usage patterns. Microsoft continues to try to do this and fails every time – because by definition people are willful and unpredictable.
Sorry, I don’t do the same things at the same time of day on my computers, and I don’t like to be slowed down because the OS is recalculating my priorities for me every time I do something a bit different.
An instinctive interface has to somehow read my mind and *know* when I want to go free-range. Otherwise, I will turn off these “conveniences” that throttle performance and limit my use of the machine.
Dragon NaturallySpeaking and Windows Vista speech recognition has most of the functionality you talked about to recognize and train certain commands but as has been said by yourself Kevin it isn’t intuitive enough yet.
But let me tell you as a disabled person who uses many of the accessibility options in Windows and who also uses his computer 16 hours a day these voice recognition programs do work once you give it a time and effort it deserves.
However what really gets to me these days is that a dual/quad core CPU has many of its cores underutilised. Surely Microsoft in the 21st century has the time and financial wherewithal to utilise all these cores in the next version of Windows as they have done efficiently in Microsoft flight simulator-x.
A perfect example of repetition the damned computer should be smart enough to do.
I do a bunch of vlc screensnaps. I want them all resized to 440-pixels wide. I should do one, them tell the PC to do the rest itself.
I do a bunch of web screensnaps. All must be cropped the same way, then resized. Instead of me having to manually crop and pay attention to the dimensions, the PC should be able to do everything itself after the first one is manually done.
Its always great to dream and think of new things but seriously the market has yet to make a simple laptop in your pocket and do you really think a talking PC in your pocket is realistic?
To me there are too many geeks thinking up computer designs and they are people not in tune with what mainstream people want. It is like the cyborg type wear a computer on your face or a TV watch, the same goes for the UMPC’s with no touch type keyboard;they are cool gadgets for geeks but the mainstream does not buy them.
Technology needs to enhance what the mainstream people prefer and feel is easier to use. I think techno computer geeks opinions do not match well for that part of the equation.
Hey, doesn’t the Ribbon in Office 2007 adjust according to what you have been doing? It sucks!
In Office 2007, nothing is ever in the same place twice, and you have to seek the hidden functions that you have rarely used.
Everyone who was familar with the Office 2003 menus and toobars hates the Office 2007 pardigm. There is even a company that sells an addon that provides the 2003 layout, and they are making a fortune. Too bad MS did not provide a simple toggle to let you choose your UI. Bring back Clippy!