Thursday, April 30, 2009

How To Start Designing A Multitouch Interface

This morning, Bill Buxton published an interesting piece in Business Week that effectively said that it is incredibly difficult for engineers to become good designers (and vice versa), so you should budget for a really good, highly experienced designer (like myself) for a lot of money. While I obviously agree with this statement, it's a big business approach to software...and it's often inappropriate.

In the current economy, using a high end designer for small projects is frequently untenable. Recently, several engineering teams have asked me to teach them how to think through design problems for themselves, so that they can better utilize existing OPEX budgets.

Many of these engineers have found it useful to start by breaking the user experience down into three basic chunks: Graphics, Input and Feedback. It would be great to get your feedback on additions to this list.

Please, check out Bill Buxton's insightful response/rebuttal to this piece. As always, he has smart things to say.


A graphics strategy for multitouch should make it easy for users to think through:
- What the computer is doing

- How to navigate from the current application state to all of the user’s goal

- How to navigate between the windows/content of an application

- How the controllers are conceptually grouped

- If content is highly freeform (like in a Surface Scatterview) how to organize it

into a recognizable structure, like a grid.

- How many views/windows are open and how users got to the current application state.

A good graphics strategy is made up of two components:

How the interface hierarchizes and prioritizes the relative (and often changing) importance of content and controllers

Example: Time sensitive content may get larger and eventually take over the screen when it becomes critical.

Metaphor: How users predict what a controller will do. A good "universe of metaphors" should think through how the way each interface element looks and animates communicates what it does, what content effects and how to undo it.

Example: One of the most frequently cited metaphors is Apple’s trashcan.
It makes sense because it echoes a real-world experience. The more controllers and interactions mimic the realworld…through physics and meaning, the easier it will be to intuit how the device will respond.

Inputs are sensor or other data that impact the interface. An effective input strategy has two components:

User Generated Events: How users control the device
Waving the device in the air, pressing a soft button, pressing a hard button

• Input multiple predefined values – GUI equivalent:Radio buttons
• Choose predefined values – GUI equivalent:application icons, buttons
• Input highly customized values – GUI Equivalent:Command line searches
• Group content/Remove content – GUI Equivalent:Drag and Drop
• Choose values along a range – GUI Equivalent: a slider, a color picker

Contextually Generated Events: What the software understands about the outside world

Examples: Changes in user, mood, noise-level/mood in the environment


An effective feedback strategy informs users that their task has been accomplished and that they are progressing toward a desired objective.

Examples: The color of a button might get brighter. There may be a noise. There may be a task path that progressively lights up.

Feedback can be broken up into two basic categories:

Device Behavior: How the software/Device responds to input, conditions and events
Examples: The devices rocks, shakes, flashes, etc.

Device Conditions: The state of the device and software
• Examples: Time, power, processor load, modes, users online, etc

Hope this is helpful. Courteous comment and additions are always appreciated.

Wednesday, April 29, 2009

Wyoming, why do you hate multitouch?

It's interesting to me how new technologies grow in some areas and not others. Why did the internet spring out of northern California instead of Boston, where LAN was big?

For those of you who are curious about the rapidly developing interest in multitouch industry, here's a heat map of last week's searches for Multitouch:

Here's what it looked like in December:

Why on earth did Georgia and Texas suddenly get religion????

A Bit of Lusty Interactivity

Sometimes, an interaction that would never make it through usability testing is just completely addictive and needs to be in the world.

In multitouch design, this is a fine line to walk ... The lack of tactile feedback makes usability vital, but if users don't want to play with the interface, a lot of time and money has been wasted.

Game designers have faced this problem for years and they have some great lessons to teach about creating lusty interactions (they would call them game mechanics).

Check out this sweet video from the independent game developers conference. From a usability perspective, the game isn't working yet...but the interaction is just lusty.

Thanks to our friends at Core77 for hunting this one down.

Crikes, I wanna play this game.

Multitouch LCD Trackpad on a Netbook

It seems like there is a lot of opportunity for improvement, the potential is unbelievable. I present to you the Sharp Mebius:

While I don't think that Americans will ever use the stylus feature, the ability to use Kanji or similar systems should be incredibly popular in Asia.

I'll need to look into it, but the spokesman claims that the Mebius is optical and can see outlines object...Potentially, this is the most exciting news of all!

Last year, I made a post predicting this product I am incredibly happy to see that it's a reality.

From a productivity point of view, I still wonder if a different layout like this wouldn't be more effective though...

It seems like it would be awefully hard to type and "mouse" at the same time.

You can see my review of the Asus Touch-enabled trackpad here

Monday, April 27, 2009

Epson Tries to Knock Off Surface with the X-Desk

Sometimes it's hard to see what a great job Microsoft has done with Surface...because there isn't anything to compare it to...until now.

This video is a testament to the fantastic work that Ali, Evan, Daniel and their teams at Surface UX did making the Surface demo apps insanely easy to use.

Friday, April 24, 2009

Multitouch Design Roundup

The volume of demoware is just exploding. I wasn't able to review everything this week, so this means a double helping next week. Well, Here's your weekend entertainment:
Let's start with the Future of Haptic Interfaces!

MS Surface Mobile Demo

I really don't know what happened here. This seemed like such a good idea when it was conceptualized. The MS Surface UX team is pretty good, yet there are any number of subtle affordances that would make this application more usable. Did they lose control of the project?

Micro Rolls

I don't know much about this guy. He's from France, but he drops an intriguing video every couple of weeks. If you are watching, please drop a line!

Update: Thanks to Anne Roudaut, the author of this video for piping up. As she notes below, she is most definitely not a "he"'s a website with links to her exciting work We all look forward to seeing more soon!

Cool Controller Scheme from Gilles Bailley

He's a PhD. student at Telecom ParisTech has also been dropping some cool stuff. The French are really starting to take a leadership position in multitouch!

Random Eyecandy From Obscura

I was accidently at their first project in ~2003 and have been a fan ever since. It's not multitouch, but check this out!

Aquima on Surface

I'm not sure what or why, but it's probably useful to one of you out there. :~} According to the website, Aquima is a functional software development environment that enables you to easily model and design applications that match your business needs. By combining process management, document creation, content management, business rule management, and presentation functionality into one powerful development environment, Aquima enables you to engineer your business. All these parts are not coded by a programmer but can be configured by you.

Intuilab Photosorting Demo

While I think there is room for visual improvement, I really like the "tractor-beam" affordances.

Immotouch Real Estate Sales Tool

I'm unsure that their is a business behind this idea, but it's a great video for two reasons:
1. The use of ambient animation in the background creates a nice emotional experience
2. The living room set. Man, I wanna live in that future.

Reporting on the UK Launch of Surface

Identity Mine Talks About Their Gesture Engine API

The first half of the video is yet another variation of the Surface Wine Demo from CES 2008. The good stuff starts at ~6:00.

Thursday, April 23, 2009

Basic Human Factors For Multitouch/Interactive Displays

Yet another prospect came to me today looking to reenact the displays in Minority Report.

Don't get me wrong. I'm huge fan of the movie, but the requests for unusable interfaces have become so frequent that I am putting together a little "Human Factors 101" pamphlet for marketers.

I would really appreciate your comments and help getting the word out about good design.

4 Reasons You Don't Want to Be Tom Cruise:
Basic Human Factors for Interactive Displays

Minority Report, Microsoft Surface and Jeff Han's touch demos have created a global sensation. These compelling videos represent the technology that marketers dream of: it is big, splashy and visceral. While these massive interactive displays are really cool in movies, real life solutions need to take the needs and limitations of users into account.

When selecting a multitouch solution, there three human factors issues that you should take into account when choosing a screen size for your interactive display.

Distance From Screen/Foveal View

While we thing we see the world in a “panoramic view”, this is an illusion that is created by our brains. Behind the scenes, our eyes are constantly scanning and updating this view. Our foveas, the part of our eyes that see in “Technicolor” only scans about 5% of our field of view at any given time. The rest of our eye is primarily built to detect motion...changes in the environment that should be updated.

While we may be able to “sense” a 5’ screen at arm’s length, we can only really pay attention to about 18-24” of it at any given time.

Try this experiment. Look in a mirror and stare into your own eyes. Put both hands out in front of you and wiggle your fingers. Continue to move them away from each other, you will feel your eyes pulling to try to track your fingers instead of staring into the mirror when they come in line with your shoulders.

Arm Fatigue

While typing and mousing might cause mild fatigue, making your upper arms defy gravity for hours or even minutes on end is straight up painful. In fact, this position is so painful that CIA used this "Stress Position" to break Guantanamo Detainees.

In extended usage situations, it's important that there be proper support for the users arms and fingers.

In tabletop situations, it is important to avoid scenarios where users need to hover for extended periods of time and where the wrist is hyper-extended (the hand and fingers should be as parallel to the forearm as possible). Top two situations to watch out for here are placing the table at too low (like Microsoft Surface) or placing content too close to the user (so that their wrists need to move perpendicular to the forearm to access it).

In wall mounted situations, interactive content should be above chest height and controllers should be placed as low on the screen as possible. Ideally, users should be able to support their elbows on a table or similar when the device is not in use.

Neck Fatigue

30 years of industrial research by furniture manufacturers has led to commonly available guidelines for table/chair heights, keyboard trays, monitor heights and lighting that make computers comfortable to use. Use it.

One of the findings of the research is that neck pain is a primary causes of decreased task efficiency in the office. Hyper-extending the neck by looking down as one might when typing with a notebook computer on one’s lap is a prime example of poor ergonomics.

A number of typical multitouch formfactors, including the Tabletop computer, the handheld tablet and wall-mounted displays that have content below chest height. This can cause intense pain if used for extended periods of time.

Eye Fatigue

The normal human eye likes to focus at ~28 inches in front of it (approximately the length of a human arm). If type sizes, task patterns or physical factors require users to be closer or farther away, your application will be hard to use. It’s important to note that older users in particular prefer larger point sizes.

While we would all like to look like Tom Cruise, Minority Report should be viewed as the cautionary tale that it is. In reality, none of us want his job. It's a pain in the neck.

If you have other thoughts about content to include, I'd really appreciate your comments. ~Jonathan

Wednesday, April 22, 2009

Two Videos Multitouch Designers Must See

UX Week '08 had two great talks that you must see if you are serious about multitouch and NUI.

The first is by Darren David and Nathan Moody of Stimulant. Currently, they are acting as "out-sourced brains" on the Microsoft Surface Team. I met them when I was there and was very impressed by the quality of their thinking and execution.

Darren David & Nathan Moody | UX Week 2008 | Adaptive Path from Adaptive Path on Vimeo.

The second video is by Dennis Wixon, the Usability Research Director on Surface. He gives an incredible talk about the Surface vision. I first heard this spiel from August De Los Reyes and was asked to give a keynote reflecting on these ideas myself, so I don't know where the core content came from originally...but it's insanely inspirational thinking.

Dennis Wixon | UX Week 2008 | Adaptive Path from Adaptive Path on Vimeo.


Tuesday, April 21, 2009

5 Questions To Ask When Planning Multitouch Interfaces

As of yesterday, I have consulted on, managed, planned or been otherwise involved with $2 million in multitouch research and application development. What has shocked me is the fantastic ability of these projects to go over time and over budget.

There are a number of reasons for this. The biggest and brightest is that teams are so excited with the technology that they forget to define their objectives first.

The second reason is the designer is brought in at the end of the process instead of the begining. The third is that user testing starts too late in the process.

There are five "must ask" questions that will help you to quickly define your application and avoid much wheel spinning.

1. What is the interaction model for each task module?

One strike aspect of multitouch devices, particularly large multitouch devices, is their social nature. Multiple users want to use them at the same time. This seems obvious, but it requires rethinking interaction models that are assumed in GUI interfaces.

Single User
- Freeform (One person manipulates one screen. The objective and order of tasks are not the Surface Sample Apps)
- Time/order Sensitive (The order of input is unimportant, but the order and rate of task completion the classic videogame Millipede)

- Turn-based (I do, then you Poker)
- Synchronous (We both work on different content at the same Halo)
- Asynchronous (We manipulate different content at different an email thread)
- Collaborative (We manipulate the same content together)

2. How are content and controllers accessed?

In GUI simple applications, it's easy to get away with a single mode interface. In multitouch, the innate space limitations make it very difficult to have complex, modeless interactions where every window/controller is available all of the time.

A strategy for prioritizing, indexing and quickly accessing content and controllers is vital to application usability.

3. What is the minimum amount of contextual information that users need at any point to move toward their goal?

It's important to limit the amount of visual noise on screen for two reasons:
- The inherent "pixel inefficiency" of multitouch interfaces
- Unnecessary information is frequently misinterpreted by users, decreasing usability.

A clearly prioritized, minimalist approach to information almost always enhances the usability of new interfaces. One great example of this is the Surface attract app.

Users first notice the bright reflection on the pebble in the center. This subtle cue encourages users to make a first touch. When users get bored of the water ripples, subtle animation of the Surface logos in the corner suggest that they might be worth touching as well.

4. How clearly does the design of controllers need to articulate what they do?

There are a number of "short hands" in GUI...We know to read web pages from top to bottom, left to right. In multitouch applications (particularly in tabletop applications) these customs go out the door.

It's important to design controllers so that they are as clear and simple about how to achieve tasks as possible. If an asset looks, sounds and animates like what it does, it will be highly usable.

Leveraging the real world experiences that users have of physical artifacts is a key communicating what they do and how to use them. Metaphors that are both graphic and tangible seem to be the simplest to deduce...buttons, knobs, hinges, moving objects forward and backward in space...etc.

5. How do users know what each controller does, how to use it and what content it affects?

Discoverability of features is intimately linked to visual design. It's important to have a strategy for communicating whether controllers have the ability to manipulate a single window or multiple windows.

Monday, April 20, 2009

A Haptic Solution for Multitouch

Chris Harrison and Scott Hudson at CMU hav been experimenting with using air bladders to enhance input for touch displays.

This an insanely cool idea. It's been floating out there for a number of years, but this is the first example I have seen.

How do you explain that you popped your iphone?

Multitouch Exhibit Design Projects

I just ran across Digit, a group of Spanish Multitouch Junkies.

DIGIT Promo from digit on Vimeo.

Nice work.

Sweet Multitouch Student Project

LICHTEN a virtual memorial from julalakritz on Vimeo.

The Best Multitouch Game Ever!

via our friends at

Tuesday, April 14, 2009

New Surface Demo Videos

Phenom Blue plays with some physics on Surface. Note the markup menus. I think this is the first example I have seen publicly on surface. We likey.

GENESIS - A Closer Look from Phenomblue on Vimeo.

Infusion and Microsoft make their pitch for linking Health Vault and Surface

House Calls with Bill Crounse, M.D. featuring Microsoft Surface from on Vimeo.

Microsoft's Randy Fusco talks about the Surface/Healthvault strategy, starting at ~8:00

Chris Sullivan, Randy Fusco, Microsoft from Health 2.0 on Vimeo.

Razorfish drops yet another car sales demo. It has some interesting ideas in it...particularly at the end of the video when a domino (fiducial)tag is printed out and given to the customer, so they can come back and use the application at a later time. What a fun way to differentiate the customer experience and track customer usage.

Carville - A Razorfish Surface Application from Bryan Hamilton on Vimeo.

BTW. Sorry to be slow on post recently. I've been working on something you're really gonna like. Always Inspired ~Jonathan

Thursday, April 9, 2009

Beta of Bump Top Released. Get it now!

I can't wait for the multitouch version of this!

I love all of the strategies for:
1. Managing screen realestate...Stacking files, skewing the content around the edges of the screen.
2. Great tools for managing sorting tasks.
3. Markup Menus

It would be great to see feedback or video from someone with an HP TouchSmart or similar!

Update, Point & do readers Ben and Laurent pointed to demos Anand has already done with multitouch devices. Alright, I know how I will be wasting my weekend :~}

Wednesday, April 8, 2009

Multitouch Application Round Up

Over the past couple of weeks a number of interesting multitouch apps have shown up on the interweb, and there are some great lessons to learn from them.

Illusion brings Their Skateboard Game to a Multitouch Table

One of the things that frequently gets forgotten in multitouch applications is that scale is incredibly important. This game is one of the top sellers on the iphone, but on a table, the interaction simply looks cumbersome. Lesson: Multitouch Apps won't necessarily port well between form factors.

New Game Demos from the Surface Team
Check out how the captured checkers and chess pieces be played with. This is a great, simple way to keep users entertained in turn-based applications. Also check out the "tiles" game (formerly known as circle pie). It's a great 360, multiuser interface.

Smart Table
Applications for the Smart Table use a range of simple affordances to encourage collaborative interaction. I like how applications have to be cancelled by clicking on two opposing corners simultaneously.

Art+Com's Touch Table for Otto Bock
These guys have been playing with touch tables as long as anyone around. Notice how they have minimzed any extranious content and oriented content in the direction from which users will actually use it. Can anyone say thank you for killing the friggin' scatterview?

Wednesday, April 1, 2009

Great Multitouch Talks at SIGCHI

While there is no lack of conferences on human computer interaction, SIGCHI is the big mama and it's in Boston next week. I've been looking at the smorgasbord of papers on multitouch being presented CHI this year. These 9 talks seem really interesting. What do you think? What would you add or subtract?

1. User-Defined Gestures for Surface Computing
Jacob Wobbrock, University of Washington, Meredith Morris, Microsoft Research, Andrew Wilson, Microsoft Research

2. IMPAD - An Inexpensive Multitouch Pressure Aquisition Device
Ilya Rosenberg, Alexander Grau, Charles Hendee, Nadim Awad, Ken Perlin, New York University

3. SLAP Widgets: Bridging the Gap Between Virtual and Physical Controls on Tabletops

RWTH Aachen University / University of California, San Diego

4. Back-of-device Interaction allows creating very small touch devices
Patrick Baudisch + Gerald Chu, Microsoft Research

5. Bezel Swipe: Conflict-Free Scrolling and Multiple Selection on Mobile
Touch Screen Devices
Volker Roth, Thea Turner, FX Palo Alto Laboratory Inc.

6. MicroRolls: Expanding Touch-Screen Input Vocabulary by Distinguishing
Rolls vs. Slides of the Thumb

7. Occlusion-Aware Menu Design for Digital Tabletops

Peter Brandl, Thomas Seifried, Jakob Leitner, Michael Haller, Media Interaction Lab, Upper Austria University of Applied Sciences | Bernard Doray, Paul To, Nortel Networks

I'm really excited about this, but haven't found the paper yet! If you find a link, please pass it on. Update: I found this talk by Michael Haller that appears to cover the content. ~pp. 54-77

Tabletop Displays for Small Group Study: Affordances of Paper versus

Digital Materials
Anne Marie Piper, James D. Hollan, University of California, San Diego

9. VPlay: An Interactive Surface for Collaborative VJing

David Kirk, Richard Harper, Armando Garcia-Mendoza, Stuart Taylor, Shahram Izadi, Microsoft Research

VPlay: An Interactive Surface for VJing from Stuart Taylor on Vimeo.

Other Talks of Interest

The Performance of Touch Screen Soft Buttons
Shumin Zhai, IBM Almaden Research Center, Seungyon Claire Lee, Georgia Institute of Technology

Empirical Evaluation of Finger Input Properties in Multi-touch Interaction
Feng Wang, Xiangshi Ren A315, REN LAB, Kochi university of technology