Friday, February 27, 2009

Let the Multitouch Patent Wars Begin!

This time, its Canada's Smart Technologies vs. New Zealand's Next Window!

Court Case Number:

File Date:

Friday, February 20, 2009

Smart Technologies ULC

Plaintiff Counsel:
Martin T. LeFevour, Timoth J. Vezeau of Katten Muchin Rosenman LLP


NextWindow Ltd.


35:145 Patent Infringement


Illinois Northern District Court

Honorable John F. Grady

Thursday, February 26, 2009

Razorfish Engineers Tossing Things at Microsoft Surface

I love this game mechanic. They say they are going to network this. I look forward to trying it out.

Table Toss (Microsoft Surface Game) from razorfish - emerging experiences on Vimeo.

I can't wait for these guys to get a touch wall...maybe they'll throw darts at it :~P.

Surface Needs A Faster Vision System

The physics really bring the experience to life, but Surface's vision system continues to be too slow really make the experience seamless. This is fine for a game where the graphics aren't critical to the interaction, but it really limits complex application development. We have been hacking around in XNA in the search for more performant solutions. It's way faster for 3D applications.

Wednesday, February 25, 2009

Stimulant Talks About Designing For NUI

My favorite quotes:
1. NUI has been around for decades. It's only now being commercialized.
2. Natural Does Not Mean Intuitive.
3. Are buttons bad?

Apparently, these are the first two parts of a 5-part interview. Hopefully, they will get around to answering why, if Surface is so different, are we using fingers as mice?

Tuesday, February 24, 2009

Surface Development Job in Beautiful Boulder

My coauthor, friend, and occasional flamer Richard Monson-Haefel is looking for a Surface developer. He's great and so are the people he works with.

Check out the posting here. If you have 5 years MS stack experience and want to work on a high-visibility, breakthrough product, give Richard a ding.

Always Inspired.


Pressure Sensitivity, The Next Multitouch Horizon

It seems that the next multitouch feature is going to be pressure sensitivity. One of the major issues in managing the sensitivity of touch event. To much sensitive and they are all over the place. Too little and users need to try 14 times for anything to work.

Pressure sensitivity provides a much better way to register intent than blog size. Great idea!

There are at least two players in the field at this moment, Sima Systems, a small startup out of San Mateo and Stantum, makers of the Lemur.

Checkout engadget's video of the Lemur reference platform:

Friday, February 20, 2009

Microsoft Design Unleashes New Design Tool

Alright, so I will admit it. I'm an Adobe slut. I've been hooked for going on two decades and I think that with the acquisition of Macromedia, it has become and will stay the most flexible creative suite around.

That said, if you really, really want to, you can make the MS stack do some insane $#!%

Why the Failure of Surface is a Success for Microsoft.

MS Surface never was and is never going to make real money. Any senior manager at Microsoft could have looked at the price point and custom development model and come to this conclusion. This isn't to say it's not the right product for the right customer...but it is a super niche product.

If MS knows they picked a loser, why did they do it?

and I have been discussing this and we have a little conspiracy theory:

Only a company as large and profitable as Microsoft could pull off a public alpha test that is larger than most mid-sized companies in the midst of a recession.

This decision is consistent with Microsoft’s history of doing Ready, Fire, Aim product planning…and it’s a great strategy…if you can afford it. You get to be a first mover, you pick up IP, and you do your heavy development based on real market data.

For ~$50,000,000 they learned how, why and if customers might buy into multitouch. While this seems like a lot of money, 1/10% of earnings is pennies to test a bet the farm idea.

Few companies consistently make .27 cents on the dollar…and the ones that do are decidedly not dumb.

Microsoft Gets a Lot More Out of Surface Than Market Validation

- Greater institutional knowledge of closed-system hardware/software platforms
- Preparation for a consumer version of Surface
- A great reference platform
- A gesture library
- Exposing preferred vendors to Multitouch
- A shitlot of IP
- A great foundation for the Win7 Multitouch SDK

What most people don't realize is that Microsoft Surface is just a prototype. The real Surface will be what hardware manufactures create to run Windows 7. If they pull it off, it will be one of the largest market research programs ever attempted.
If Microsoft pulls this off, it will be one of the largest alpha programs ever attempted. It’s the scale of the experiment that has confused the market.

What company besides Microsoft could afford to build a business unit just to float an idea?

Thanks to Josh for your collaboration in writing this post! Check out his blog at: It's only slightly less interesting than mine :~P

Using Transparent Artifacts On Top Of Surface

When I first saw Surface, the thing that blew me way was the use of transparent tiles as ID tags.

Over the course of development, Microsoft stopped pushing this idea, likely because the acrylic pieces create too many false touches. This seems short sighted to me, so I did a some materials research and developed a range of options that work pretty well.

1. Choose a material with a more appropriate refraction index.

2. Use an IR reflective coating.

Here is an IR reflective paint:

3. Scuff the acrylic to turn it into a screen. We have had good success with this.
If you machine this really well, you can also get contact feedback from on top of the artifact. It feels REALLY nice.

A word to the wise: Messing around with interactivity on top of lenses can rapidly get you into mathematics that you might rather avoid.

4. You can mask or etch a frame around the artifact. Laser etching is a nice,inexpensive way to do this. You can also create a partial screen on top of the surface. This is a pretty cool effect.

Thanks to Staph for reminding me about this.

Thursday, February 19, 2009

Complex Dataclouds and Multitouch

Reza Ali recently mapped usage data from the Seattle Public Library to the Dewey Decimal System to better understand how the Library was being used. Complex data clouds like this are just begging for multitouch interativity.

Check out the interacive version here:

What do you think?

Monday, February 16, 2009

Great talk on Physical Computing by David Merrill

A number of years back, we created a number of prototypes like this for a game manufacturer.
Really cool stuff.

Thanks to our friends at for pointing this out.

New Demo Videos From Infusion

A few months ago, I called up Greg Brill (no relation), founder of infusion to talk about what they are doing on Surface. Needless to say, it was a hilarious cold call. He graciously invited me over to chat. I was impressed and think they are a great team.

Infusion is an ~200 person Microsoft VAR with a large focus on sales in the middle east and a dedicated Surface UX team based out of Canada. To my knowledge, they are the largest committed buyer of Surface units.

Last week, Infusion put a bunch of new marketing content out on the interweb. Here they are in all their glory.


With the Stimulus package's focus on electronic medical records, everyone's trying to sidle up to Microsoft Healthvault. Infusion is no exception.

Internknowlogy dropped an incredible similar demo last year.

News Reader
Video of this has been sitting on infusion's website, so I assume it is them...While the interaction is really natural, I look at a number of "advanced" features like the rotating ring and wonder how discoverable they are.

I am also concerned about speed of use on this...maybe I am different than the rest of the world. I monitor ~200 posts a day on 50+ RSS feeds to bring this blog to you. Browsing on a reader like this would be torture.

Falcon Eye
Infusion has been out pounding the pavement with this for a number of months now. It has a range of interesting functionality, but as with most GIS multitouch demos, it seems to lack in depth...a petro-geologist will have very different needs than a city planner.

I like the integration of 3D overlays onto what looks like Virtual Earth. From a development viewpoint, this is a neat trick...but the hardware may not be ready for prime time...take a look at the lag time in loading 3D assets.

Lazzarra Yachts
I'm not sure if they are serious about this...but it looks like infusion is attempting to skin Falconeye with interface components from Newsreader. It's worth a a watch just for the over the top video!

If you want more infusion goodness, you can find more here.

I am always glad to showcase Multitouch Design Leaders. If you have content you think I should be showing, please ping me at the email to the left.

Friday, February 13, 2009

Why Microsoft Is Smart To Open Digital Lifestyle Stores

Today's Wall Street Journal Reports on Microsoft's decision to build its own retail stores and goes on to mock it:

“The failures of other stores opened by technology companies will loom over Microsoft as it launches its stores. In 2004, computer maker Gateway Inc. shuttered a network of more than 188 company-owned retail stores after weak sales. Microsoft itself operated a Microsoft store inside a movie-theater complex in San Francisco beginning in 1999, but two years later shut down the store — which showcased, but didn’t sell, Microsoft products.”

Frankly, comparing these two store concepts to Microsoft's business model and portfolio is absurd.

While MS makes no shortage of dumb decisions,
this one its really, really smart


Retail Real Estate

Flagship brand retail isn't about selling the goods in the store. It's about selling product in the surrounding 20 mile radius. Nike discovered this with its Niketown stores. It made more off of sales in the surrounding area than it did in store...and at its height, Nike's Chicago flagship was doing $1,200 a square foot.

The Microsoft store in San Franciso was an attempt to sear eyeballs, not to sell product. The location closed when Sony Metreon (the development) completely failed to attract the projected crowd. Things got so bad at this mall that Discovery Channel, an anchor tenant, found it was cheaper to pay rent on a closed 20,000 sf prime storefront than it was to keep it open.

Unlike 1999, high street retail space is going for pennies on the pound ... especially for a stable company like MS. At one point, Metreon offered to advance my client $600k to refit the Discovery Channel space for a new store concept. Even better deals will be available to MS. If they are smart, they won't pay rent on the spaces while flagships are under construction (likely 18 months).

If I were MS, to speed the rollout, I might be looking to gobble up a high-end electronics chain. I can only imagine that Bang and Olufsen's retail is sucking wind. Radioshack might be a larger acquisition target.

Microsoft Merchandise

Microsoft has an easier pitch than Apple. It's a lot easier to sell accessories to a system users have already made large investments in. Apple is trying to sell users up ... Buy the ipod first, buy the computer second.

Gateway's problem was that it had its own perishable equipment tied up on its shelves.
Microsoft isn't going to pay for HP's hardware ... they are going to sell shelf space. Microsoft will only tie up capital on long shelf life, high-margin/volume items like keyboards, xboxes and Zunes.

Let's take a look at MS's consumer portfolio 18-36 months from now. The combination of Win7 and the products the Entertainment and Devices Division is likely to roll out over the next 24 months make a very compelling story.

- The vast majority of prospects have will have bought into Vista or Win7.

- Windows mobile/Zune will be fixed.

- Surface-like devices will be more reliable and price competitive.

- MS will have many of its cloud computing datacenters up and running.

- XBox will likely stay in a #1 or 2 position.

Market Positioning
Apple has positioned its stores as museums that derive their character by being devoid of it. Yes, the effete liberals are hot on this...but what about everyone else?

If Microsoft simply provides endless tables of desktop computers, it's dead in the water. Frankly, Apple has already done this as well as it can be done.

If Microsoft is able to tell an egalitarian story using the very technologies it sells ... a sort of executive briefing centers for real people it will be a hit ...people like places that are full of great stories told through gee whiz tech.

This Microsoft Store is a place I will want to play ... and it's the most compelling scenario for a digital lifestyle store that I can imagine ... Oh, and everything will be demo'd on multitouch.

New Demo from RENCI

Visual search tasks like those in airport security and medical imaging are often easier to perform when the objects are rotating (as opposed to static images). At the same time, it's easy to lose one's orientation when stuff is flying all over the place.

The team at RENCI has come up with a nice way to maintain orientation while leveraging the power of 3D rotation. Check out :58 - 1:32

RENCI Multitouch Footage in HD from Renaissaince Computing Institute on Vimeo.

Thursday, February 12, 2009

Dell's New Multitouch Tablet

Dell Latutude XT2 from asadal on Vimeo.

I don't know what the ongoing fascination is about walking around with multitouch tablets. It's an incredibly akward idea for two reasons:
1. Balancing a 5 lbs. box on your arm is hard enough without poking it all over the place.

2. The natural standing position for users is to stabilize the device with their abdomen. If the user balances the tablet with their left arm, the bottom left third of the screen will be uncomfortable to access.

Pick up a computer, book, cereal box or magazine and try it.

More importantly, I don't understand why hardware manufacturers are providing these products without a meaningful SDK. Hardware is useless without easy to develop software.

Roundup of Talks From Interaction '09

Design conferences normally leave me clammy ... Is it just me or is this an industry of self indulgent divas who talk about the need for empathy?

Design conferences seem to have one of two themes:
1. We are the champions of the world
2. Who the @#%$ are we?

It seems that this one fell into the latter category and came up with the utterly unbillable answer, best articulated by Frog Design's R. Fabricant: we design behavior not technology.

While I agree that this is true, it's an abstract idea. Personnel Managers, the Army and Ben Bernanke also design behavior...and in much more meaningful ways.

Interaction designers design nonlinear interactive interfaces. They do. That's what they do.

The trend toward mobile and integrated hardware/software organizations might blur the line a bit. Having built buildings, manufactured physical objects and designed software, I can assure you that these are different skillsets with unique project definition, legal, production, documentation , profitability, business development and process models.

It would be interesting to ask frogdesign, Smart or IDEO about the margins on web/application development vs. industrial design. Maybe the growing margin pressure on ID and behavioral research is driving Fabricant's definition.

If we are coming up with new industry slogans, how about:
IX Designers increase operational efficiency by decreasing user errors and increasing users' ability to understand complex datasets
We make it really, really easy to buy stuff.

If we are gonna get all abstract, these seem like more appropriate themes for the time.

Sorry to be so Pithy.

Anyhow, here's some fun stuff from the conference:

The folks over at have put together an exhaustive blow by blow of the whole conference.

For the fluff friendly, Aynne Valencia has grocked it down into a picture-heavy designer friendly slideshow.

Simon King of Ideo gave a nice talk about privacy and data-driven design. I think he threw down the gauntlet for an impending national debate...web privacy 2.0

Dan Saffer busted out what one twitterer called a Danifesto.

What do you think? Flame Away!

Wednesday, February 11, 2009

Creating Intuitive Gestures Using Object Orientation

Recently, I've been thinking a lot about how to create intuitive, high precision gestures for a context menu system.

In the process, I remembered this interesting research by Ravin Balakrishnan's group at the University of Toronto:

Here are some other interesting projects coming out of U of T

Pointing Lenses

As an American, this is getting embarassing. Why do the Canadians have the good UI? They are doing the cool academic research. They have taken over at Microsoft and Autodesk.

Is it something in the coffee at Tim Horton's? :~P

What do you think of Ravin and Co.'s ideas?

Multitouch and Public Accomodations

I've been having a brilliant offline conversation with JoshB. He brought up an aspect of MS Surface Multi-touch Othodoxy that I just disagree with:

"My understanding of NUI was that you have fewer paths but the frequency of interaction is higher and more contextual relative to GUI."

I'm going to get heretical here and say that multitouch in existing kiosk applications (ATMs, Ticket kiosks, etc.) is a usability disaster waiting to happen.

In "single serving" applications, you want modeless interactions and a linear decision tree. If you are going to design a linear decision path and minimize interaction modes, why not go a step further and just do single point directed every ATM in christendom.

It cuts hardware costs in half and increases usability.
I'm going to get heretical here and say that I think modeless interaction is a flatout bizarre model for multitouch ... and that multitouch in existing kiosk applications (ATMS, Ticket kiosks, etc.) is a usability disaster waiting to happen.

Multitouch really shines in multiuser productivity apps and in situations where there is more content and data than can be analyzed

If you are going to design modeless interactions, why not go a step further and just do single point directed every ATM in christendom. That's the most usable touch interface framework.

Tuesday, February 10, 2009

An interesting Approach to Pointing and Selection in Multitouch

Here's a nice piece of work from Tovi Grossman's group at the University of Toronto

I think this concept could have some interesting applications in multitouch situations where there is a lot of overlapping content.

Monday, February 9, 2009

Running Around Apple's IP Minefield

A couple of weeks ago, Richard Monson Haefel started asking around about the impact of Apple's most recent multitouch patent.

I look forward to the clarification when Microsoft inevitably brings this big boy to court. Can gestures be patented with any meaningful breadth? For a patent to be valid, it needs to be inobvious, yet good gestures are obvious by definition.

This morning, gizmodo dropped an excellent piece on the subject. Check out the teaser below and the link at the bottom.

Dissecting Apple's "Multitouch" Patent: Can It Stop Palm?
By John Mahoney, 1:00 PM on Fri Jan 30 2009, 37,754 views

The iPhone’s multitouch patents are the equivalent of a cold war nuclear arsenal—dormant for now, but Palm’s Pre is looking for a fight. Here’s why we think Apple’s multitouch monopoly won’t last.

To help guide us through, machete in hand, what is one of the more confusing jungles of U.S. law, we talked to R. Polk Wagner, a professor of patents law at the University of Pennsylvania Law School. He specializes in patents and intellectual property as it relates to technology, and teaches hundreds of Penn Law students every year how to decipher the Enigma-level encrypted language of patent filings. We couldn't have done it without him.

As others have thoroughly and eloquently explained this week, it's impossible to identify a single patent that has a lock on the iPhone's multitouch magic as we know it. That patent probably does not exist. But here's the key—patent wars are intrinsically cold wars. They entail both sides jacking up their arsenals (reams of legalese replacing megaton warheads) with as many patents as possible, with hopes of scaring their adversaries out of even attempting to try something. These cold wars, thankfully, rarely turn hot, but under our legal system, lack of courtroom action means there's almost no way to determine whose armada of patents actually cover what.

More After This Link

Saturday, February 7, 2009

How Do We Dance With Computers?

Blogger extraordinaire Lynn Marentette sent this over the transom this afternoon. She's been thinking a lot about the design of freeform gestures and their interrelation with the realm of interactive dance.


Michael Moschen is probably the greatest juggler in the world. He single-handedly evolved the craft from a carnival sport to a fine art. Here he is talking about the design of complex gestural systems.

My personal passion is contact juggling. Check this out:

Friday, February 6, 2009

The MIT Media Lab Uses Portable Projectors To Build Contextual Multitouch Interface

It seems that Patty Maes Fluid Interaction group at the MIT Media Lab has been taking Minority report seriously. There are too many interesting ideas to describe.

I would simply visualize this in After Effects. These coots actually went and built it. Kudos!

Special thanks to Staph for pointing this out.

Problems With Mulitouch Content Selection In Win7

I ran across this today. It looks like Microsoft is developing some snappy content selection interface components in Win7...but they seem more flash than function.

Am I missing something or is this simply a folder without the ability to multiselect or a box to define it. it would be very difficult to:
- Select and drag multiple objects at one time.
- Reorganize the order of groups of objects

They also seem extremely pixel inefficient...While I like how the "folders" compress, users could only have ~3 folders open at a time before running into visual chaos.

Takeo Igarashi's team at the University of Tokyo has come up with a range of interesting interaciton methods may blow Windows out of the water:

What do you think?
What do you think?

Monday, February 2, 2009

Wanna See What Happens When Real People Use the "Minority Report" Interface?

A couple of months ago, I blogged about Oblong, the new venture of Minority Report UI guy John Underkoffler.

He showed up at sundance with his hell contraption and tried to use it on innocent spectators. See the results.

Here's 9 minutes of usage:

Here are the results:

What do you think? Is there a big idea in here somewhere?

Crazy New Surface Demo From Stimulant

I've been talking for a while know about interaction between handheld devices and Surface. Check out this new video from Stimulant

Stimulant: XRay from Stimulant on Vimeo.

Eyecandy for Multitouch Designers

One of the difficulties of multitouch interaction is that GUI design tools are really set up to make things look perfect. I think there is a lot to be learned from the tangibility of stop motion animation. Take a look.