Natural sound is as essential as visual information because sound tells us about things we can’t see and it does so while our eyes are occupied elsewhere. Natural sounds reflect the complex interaction of natural objects; the way one part moves against another, the material of which the parts are made. Sounds are generated when materials interact and the sound tells us whether they are hitting, sliding, breaking or bouncing. Sounds differ according to the characteristics of the objects and they differ on how fast things are going and how far they are from us.
Bill Gaver

Detour: Location-Aware Audio Walks

Detour is a brand new way to experience the world. Gorgeous audio walks in San Francisco that reveal hidden stories, people and places all over SF. Each hour-long Detour takes you at your own pace, your own schedule, alone or synced with friends.

Lovely idea, an “immersive location-aware audio walk service”, perfect for people like me who might land in a place alone and might appreciate a guide to the more interesting places off the beaten path. Similar to something I explored before.

Moff – a wearable smart toy

Moff bills itself as a “wearable smart toy”. Grab a broom and strum, and the Moff bracelet will emit a guitar sound. Grab a banana and point it like a gun, and your shots ring out. A pretend tennis racket, you’ll hear a swat sound when you swing.

I love this idea, it’s weakness and strength is it’s reliance on a smartphone.

Ambient wants to make the world calmer

With Ambient the physical environment becomes an interface to digital information rendered as subtle changes in form, movement, sound, color or light.
Current information interfaces are either interruptive or too detailed. For the first time in history, ubiquitous wireless networks can affordably deliver digital information anytime, anywhere. The result for most of us is cacophony. Ambient wants to make the world calmer. Ambient Devices.

From an old ambient interface project of mine called unoriginally, Girls Ambient Room.

Approaches to the design of sounds

In his thesis ‘Auditory Information Design‘, Barrass (1998) describes 7 types of approaches to the design of sounds that in particular support information processing activities – syntactic, semantic, pragmatic, perceptual, task-oriented, connotative, and device-oriented.

The syntactic approach focuses on the organization of auditory elements into more complex messages. The semantic approach focuses on the metaphorical meaning of the sound. The pragmatic method focuses on the psychoacoustic discrimination of the sounds. The perceptual method focuses on the significance of the relations between the sounds. The task-oriented method designs the sounds for a particular purpose. The connotative method is concerned with the cultural and aesthetic implications of the sound. The device-oriented method focuses on the transportability of the sounds between different devices, and the optimization of the sounds for a specific device”. (Barrass 1998).

It’s great reading if you are interested in Audio Interfaces.

Unique qualities of sound

Some of the ways that sound is unique are as follows:

  • Sound can provide information about the interior of an object. Our ears perceive patterns of moving air from vibrating objects and sound can carry information about the consistency and hollowness of objects.
  • Sound also communicates information very quickly (Brewster 1998)
  • “Sound exists in time and over space, vision exists in space and over time” (Gaver 1989)
  • Sound is not bound to a specific location

A draw back is that one cannot turn away from sounds nor can one turn off our ears to unpleasant sounds.

Audio User Experience – Q&A

Kathy Sierra was nice enough to send me an email asking me some thoughts on audio/sound. I sometimes need this impetus to write down even the briefest thoughts on a subject (and these are just brief sketches). The following are her questions and my answers.

Do you agree with me that the power of audio/sound is being greatly overlooked in so many areas of product design, user experience, etc. (as opposed to areas where sound is recognized as crucial, like movies and commercials)?

Yes I agree but there is a good reason – I would also extend your characterization of crucial to include games and toys.
Movies and commercials are passive shared experiences. Task based products are interactive and not generally shared. It’s an obvious but crucial difference. Everyone outside of China may agree that noise is something that we would rather not experience. But sound is not noise.
Sound is distinguished from noise by the simple fact that sound can provide information.
Sound answers questions; sound supports activities for tasks, so sound is inheritly useful. Consider the information provided by the click when the bolt on a door slides open, the sound of your zipper when you close a pair of pants, the whistle of a kettle when your water is finished boiling, the sound of a river moving in the distance, the sound of liquid boiling, of food frying, and the sounds of people talking in the distance. In the workplace there are the sounds of keys being pressed on a computer keyboard.
Natural sound is as essential as visual information because sound tells us about things we can’t see and it does so while our eyes are occupied elsewhere. Natural sounds reflect the complex interaction of natural objects; the way one part moves against another, the material of which the parts are made. Sounds are generated when materials interact and the sound tells us whether they are hitting, sliding, breaking or bouncing. Sounds differ according to the characteristics of the objects and they differ on how fast things are going and how far they are from us.
An extension of the statement that tasks are not shared is that the environment in which the tasks are competed are – one person’s sound is another noise. Visual displays are not as intrusive as auditory ones.
So the question of whether or not auditory interfaces would or should be used is primarily a question of implementation – how to restrict the receiving of the information inherent to sound to the person meant to be receiving it? When we solve this problem cheaply then I think we will see a great deal more use of sound in other products’ development.

Do you see any areas of great leverage — places where audio/sound could be incorporated that could make a big difference in either usability, user experience (even if simply for more *pleasure* in the experience)?

I hesitate to use these buzz words but with the popularization of Ajax/Web 2.0 interfaces it may be a good time for people to start experimenting further with sound in online application interfaces. Since these interfaces load data in real time, we lose a vital visual clue from the pages loading or refreshing. Sometimes the data change happens so fast we can’t follow any clues.
But these ideas are always met with criticism. An example from Jeffery Veen, “Sounds I stopped counting how many times I tore the headphones from my ears when a site started blaring music or “interaction” cues like pops, whistles, or explosions whenever I moused over something. Am I the only one who listens to music while using my computer?”
I love childrens toys and gain much inspiration from them. Cheap cheap sensors which illicite wonderfully fun feedback. We should have these in everything. Imagine buying a jacket that when you closed the snaps it sounded “heavier” than it feels or looks. Like the difference in sound between the door closing on a Lada and a Benz. Lots of possibilities.

Any other comments on your “Adult Chair” experience? What you learned from observing users interacting, etc.?

The adult chairs were just a small part of a broader set of objectives in creating non-elitist interfaces to musical expression. Though all of my work at that time were prototypes, just some manifestations of some ideas I had, I was harshly criticized because of the lack of “new science” or extended interactivity. Basically my work was too simple due to using off the shelf tech. and short lengths of time that people were engaged in the activities. I rejected this criticism, mostly, because I knew the criticizers didn’t understand the goals of the project and they weren’t looking at people actually using the prototypes. Though it was never intended to be so, this project ended up being the greatest champion of user centered design for me personally. We video taped allot of sessions and gathered allot of anecdotal data which drove later iterations of the design.
Some of the conclusions:

  • Its really hard to design interfaces that have no visual responses. In a game we developed around an interface similar to Adult Chairs (hulabaloo) children kept looking for flashing lights or some kind of physical response. Eventually they learned to use their ears only which was good as it was a music appreciation game. Children here are very conditioned to visual response.
  • People love being surprised and they want to have fun. They don’t care if the technology came from radio shack – they care if you can make them smile.
  • Features, options, and controls are not needed to allow people to have fun for a short period of time. To keep them engaged for long periods of time people want that control.
Any other thoughts or tips for the rest of us?

I think too often when people thing of audio interfaces they immediately think of the horrible implementations in Yahoo IM, icq, and flash sites with hip hop sound tracks. It can be intelligently and elegantly designed.
Another thought is the difficulty in designing “gray sound”. Computer user interfaces are gray – not thought provoking – sit in the background and purposely boring. Icons and language localizations aside I think they work everywhere. But how to design auditory signals that work everywhere – cultural differences abound and what data is there to help us?
I live and work in Taiwan, arguably the noisiest group of people anywhere (i’m guessing). They “appear” to have a tolerance for noise and a need for sound that is far different than my own. Because their environment is so full of aural cues how do we design for them? A Japanese garden is a place of tranquility. A Canadian park a place of clean nature. A Taiwanese park is frequently experienced with a soundtrack as they pump in music and nature sounds to keep it from becoming quiet. Quiet seems to make them uncomfortable. This is just one example of what is acceptable or normal for levels of aural cues across 3 different locations and cultures. I think localizing audio interfaces will be quite challenging.

Richard Etter’s Melodious Walkabout

A project quite similar to my proposal Guidebot which unfortunately I couldn’t get enough interest in the project to get some budget to take it beyond a simple exhibition poster. It’s great that Richard Etter was able to take something similar, and likely more capable, and make it real. His project is likely a far better fit for the the Fraunhofer Institute for Applied Information Technology than mine was for where I was working at the time. Fraunhofer investigates human-centered computing in a process context.

A variety of navigation systems have been developed that use a GUI-based interaction style. However visual navigation systems are often inappropriate in the dynamic mobile context since the user has to watch the device and cannot keep his eyes on his surrounding environment. Auditory navigation systems are more convenient, mobile users can easily interact with the system and are not visually distracted. But most auditory systems navigate the traveler by using precise spoken instructions and speech requires high attention.

Read: Melodious Walkabout – a new approach to navigation – Richard Etter

Adult chairs exhibited in Taipei

Never underestimate the power of fun.
Today one of my tangible interface experiments is being exhibited in Taipei as part of some industry showcase – apparently the President of Taiwan is going to have a look though after yesterday’s election I doubt he will have much enthusiasm for the fun my piece seems to provide. The continued quasi-popularity of Adult Chairs always suprises me as it was without a doubt the simplist interface I had made. I think it proves how important simplicity, discovery (surprise), and fun can be in creating these type of products (interfaces). At least in the context of everything else that was being produced by the company.
Adult chairs were shown in a greater exhibition I had back in January and were a part of a project I was running called smenms. They were very simple prototypes which consisted of 7 pressure sensors (couldn’t afford 8) hidden in 4 pillows which when activated controlled a simple parameter of music. I wanted to embed sensors in ordinary objects which would allow people to interact with music and sound with a form that had a completely different use. It was hoped that by making the interface invisible and a part of ordinary objects we could invoke a sense of wonder, surprise, and hopefully engage people in the creation of sound and music in a whole new way. The whole project followed an iterative project development cycle, with every cycle producing an increasingly expressive musical interface. This was a slightly different version of the first iteration which utilized a simple on/off interaction metaphor. In first experimenting with different sounds, music, and parameters in which to control (I originally wanted people to control wildly different parameters but no one found it fun – it sounded too “post modern”) I finally settled on controlling the volume of separate pre-composed tracks. I was disappointed in the amount of expression but the audience was enthused – perhaps the lively music I wrote and produced carried the day.
Yesterday Chientai and I were setting it up for likely the last time. I will miss projects like this.
Here is an example output of the song the chairs controlled called Sit and Dance.
More info. here and a related article which perhaps should have been the title of this one Never Underestimate the Power of Fun.

Notes on (Audio) Iconic Interface Design

Notes of interest to audio interfaces from an aging but good paper, “The Use of Metaphors in Iconic Interface Design“.
“Icons are used extensively for communication purposes. The term icon has been adapted from its Russian origins -‘ikon’ meaning a religious painting or statue. Within the context of computing the word is used to refer to a small image which embeds ‘meaning’.”
“… some of the most popular of the new developments are ‘picons’, ‘micons’ and ‘earcons’. Picons are essentially icons that embed a picture (as opposed to a symbol). Similarly, micons are composed of moving pictures or video clips. Earcons, or auditory icons, are based upon the use of sounds and are usually embedded in sonic sequences …”
“… metaphors can play a part in the development of a functional specification. Indeed, metaphor usage should be made explicit at the design stage of application development so that maximum benefits can be attained. In this way a whole range of functions can be identified for which icons are required. In addition, metaphors can assist interface design by providing ideas for individual icon designs.
Second, the use of metaphors can have a significant impact upon end-users. Within an end-user interface metaphors can provide cues for the recognition of iconic symbolism …”
“As well as textual augmentation, audio augmentation may also take place.
Of course, serious problems can arise from the use of textual and audio augmentation – particularly, with respect to international interface designs. Obviously, if an interface is intended for international use then any textual labels which have been attached to it must be dynamically switchable between the target languages.”
How do you localise audio interfaces? The desktop metaphor seems to be transcending culture but what about audio metaphors? A click sound is a click sound but what about more complex functions.
Read the full paper: Metaphors in Iconic Interface design

Beeps and blips

Quote:“Sounds are critical,” said Donald Norman, co-founder of the Nielsen Norman Group and a University of California San Diego professor emeritus in the departments of cognitive science and psychology. “You have to spend the same type of attention to designing sound as visual appearance. Companies these days always hire graphic artists. They need to hire sound artists.” ”

Audio in the Computer User-Interface

“A number of studies have shown how audio contributes to the interaction process in order to provide a richer, more robust environment than with mere graphic feedback. Auditory feedback can present further information when the bandwidth of graphic information has been exhausted, as is often the case with the present emphasis on graphic presentation. By expanding conventional interfaces in another dimension, sounds make tasks easier and more productive. Other studies have even shown certain types of information to be represented better by sound than through graphics or text. Additionally, audio feedback may complement graphics and text to create valuable redundancy, reinforcing or reconfirming a concept in the user’s mind.”
Noise Between Stations: Audio in the Computer User-Interface