A paper by Rafael Ballagas, Jan Borchers Michael Rohs, and Jennifer G. Sheridan.
Mark Weiser envisioned ubiquitous computing as a world where computation and communication would be conveniently at hand and distributed throughout our everyday environment.  As mobile phones are rapidly becoming more powerful, this is beginning to become reality. Your mobile phone is the first truly pervasive computer. It helps you to both keep in touch with others and to manage everyday tasks. Consequently, it’s always with you. Technological trends result in ever more features packed into this small, convenient form factor. Smart phones can already see, hear, and sense their environment. But, as Weiser pointed out: “Prototype tabs, pads and boards are just the beginning of ubiquitous computing. The real power of the concept comes not from any one of these devices; it emerges from the interaction of all of them.” Therefore, we will show how modern mobile phones (Weiser’s tabs) can interact with their environment – especially large situated displays (Weiser’s boards).
The emerging capabilities of smart phones are fueling a rise in the use of mobile phones as input devices to the resources available in the environment such as situated displays, vending machines, and home appliances. The ubiquity of mobile phones gives them great potential to be the default physical interface for ubiquitous computing applications. This would provide the foundation for new interaction paradigms, similar to the way the mouse and keyboard on desktop systems enabled the WIMP (windows, icons, menus, pointers) paradigm of the graphical user interface to emerge. However, before this potential is realized, we must find interaction techniques that are intuitive, efficient, and enjoyable for applications in the ubiquitous computing domain.
In this article, we survey the different interaction techniques that use mobile phones as input devices to ubiquitous computing environments, including two techniques that we have developed ourselves. We use the word “smart phone” to describe an en-hanced mobile phone. In our analysis, we blur the line between smart phones and personal digital assistants (PDAs), such as the PalmPilot, because the feature sets continue to converge.
The Smart Phone: A Ubiquitous Input Device Paper (.pdf)
Jan Chipchase is Principal Researcher at the Mobile HCI Group in Nokia Research. His personal insights can be found on Future Perfect, Jan’s wonderful photo-intensive weblog. As he says: “… if I do my job right you’ll be using it 3 to 15 years from now.”
The Convivio Network Interview
“A number of studies have shown how audio contributes to the interaction process in order to provide a richer, more robust environment than with mere graphic feedback. Auditory feedback can present further information when the bandwidth of graphic information has been exhausted, as is often the case with the present emphasis on graphic presentation. By expanding conventional interfaces in another dimension, sounds make tasks easier and more productive. Other studies have even shown certain types of information to be represented better by sound than through graphics or text. Additionally, audio feedback may complement graphics and text to create valuable redundancy, reinforcing or reconfirming a concept in the user’s mind.”
Noise Between Stations: Audio in the Computer User-Interface
“Ambient intelligence appears poised to cause remarkable changes in the way people live. With digital information, the ease of interaction between humans and computers can be greatly increased by broadening the interface media available and allowing for mobile and portable communication free of inhibiting wires and stationary units. Additionally, some forms of ambient intelligence allow computers to adapt to their user’s preferences. The result of ambient intelligence is ultimately a more empowered computer with the benefits of added convenience, time and cost savings, and possibilities for increased safety, security, and entertainment.”
Read the full paper