Chapter 8 Interaction Devices

8.1 Introduction

  • Progress in speed and storage has been matched by improvements in input/output devices
    • QWERTY keyboard will remain primary device for text input
    • But even it is evolving to accommodate changes in technology
  • Pointing devices provide freedom from the keyboard
    • Mouse and touchscreen
  • Innovations in pointing devices
    • Eye-trackers
    • DataGloves
    • Haptic or force-feedback devices
    • Further reducing the interaction gap between human and computer
      • Provide alternatives to disabled users
      • Brain-controlled mouse movement
      • Implanted input devices
  • Speech recognition is still improving
  • Increasing emphasis on speech store-and-forward
  • Non-speech auditory interfaces
  • Speech generation
  • Highly varied range of display devices
    • Small LCDs on digital cameras and mobile devices
    • Wall-sized (or larger) high resolution displays
  • Additional topics
    • Multimodal interfaces that combine several modes of input and output
      • Simultaneous modes have had limited application
      • Bigger payoff to switch between modes depending on need
    • Context-aware computing
      • Useful in mobile computing when a device is aware of the context of its environment
      • Provides information about surrounding places, objects, or other devices

8.2 Keyboards and Keypads

  • Speed of input
    • Traditional keyboards allow only one key press at a time
    • Higher rate of data entry possible if multiple keys are pressed simultaneously
      • The piano keyboard allows several finger presses at once and recognizes different pressures and durations
      • Chording – chords represent several characters or entire words
  • Keyboard size and packaging influence user satisfaction and usability
    • One-handed keyboards
      • Useful for manipulating physical objects
      • Or when user’s tasks require simultaneous data entry
    • Tiny keyboards
      • Acceptable for limited text entry

Keyboard Layouts

  • In the 1870s Christopher Latham Shole’s design for typewriter key layout became successful
    • Clever placement of letters that slowed down users
    • Placed frequently used letter pairs far apart to increase finger travel distance
  • Dvorak layout
    • It increased speed and reduced errors
    • Despite these improvements the perceived benefit did not outweigh the effort to learn a new, nonstandard interfaces
  • ABCDE style
    • Keys laid out in alphabetical order
    • Designed to be easier to locate keys yet studies have shown no real advantage
  • Number pads
    • Issues arise because of 2 different standards
      • Telephone uses 1-2-3 keys on top
      • Calculator uses 7-8-9 keys on top
  • Keyboard ergonomics
    • Awkward wrist and hand placement
  • Users with disabilities
    • Keyless keyboards
    • Reliance on pointing devices to select keys
    • Large menus with fixed choices
    • Dasher – predicts probable characters and words as users make their selections in a continuous 2D stream of choices


  • Tested standards
    • Slightly concave surface
    • Matte finish to reduce glare
    • 40 to 125 grams of force with 1 to 40 millimeter of key travel
      • Provides suitable feedback to the user
    • Light click when key has been depressed far enough
      • Tactile and audible feedback important in touch typing
      • Lack of tactile response limit the use of virtual, membrane, or cloth keyboards
    • Certain keys should be larger than others
      • Space bar, Enter key, Shift key, Ctrl key
    • Some keys should clearly indicate their state
    • Caps lock and Num lock
  • Cursor movement keys
    • Placement is important for rapid and error-free movement
      • Inverted-T arrangement for using three middle fingers
      • Cross arrangement
      • 8 key arrangement for diagonal movement
      • Reassign letters keys to movement keys for games in order to minimize finger movement
  • Usability
    • Large print keys for the visually impaired
    • Adjustable key auto-repeat that can be slowed for very young users, older users, and users with motor impairments

Mobile Devices

  • Most laptops have full-sized keyboards
  • Phones provide text and email functionality and a small QWERTY keyboard
    • Mechanical or virtual
    • Up to 60 wpm w/ mechanical
  • Predictive techniques
    • T9 and LetterWise
    • Useful in multi-tap systems that require users to hit a number key multiple times to specify a letter
  • Handwriting on a touch sensitive surface
    • Character recognition can be error-prone
    • Gestural data entry of simplified character sets
      • Graffiti on Palm devices
      • Requires training to learn the unistrokes
    • Shorthand gesturing on a keyboard using shapes that match tapping patterns
    • Usability
      • Can benefit Japanese and Chinese
      • Disadvantage for users with disabilities, very young users, and older users that lack motor control

8.3 Pointing Devices

It is often convenient to point at and select items.

  • Pointing devices, a direct manipulation with benefits such as:
    • Avoid learning commands
    • Reduce typographic errors with a keyboard.
    • Keep attention on the display
  • Results
    • Faster performance
    • Fewer errors
    • Easier learning
    • Higher satisfaction
    • Important for small devices and large wall displays to make keyboard interaction less practical.

Pointing Tasks Seven types of interaction

  • Select - choose from a set of items, used in traditional menu selection, the identification of objects of interest.
  • Position –Users choose a point in a one-, two- , three- or higher dimensional space. May be used to create a drawing, to place a new window or drag a block of text.
  • Orient - Choose a direction in a one-, two-, or higher dimensional space. May rotate a symbol on screen or indicate a direction of motion.
  • Path - Rapidly perform a series of positioning and orientation operations. Such as a curving line in a drawing program.
  • Quantify- Users specify a numerical value. Usually a one-dimensional selection of an integer or real values to set parameter
  • Gesture- Indicate an action to perform by executing a simple gesture, such as a swipe motion to the left or right to turn a page.
  • Text – Users enter, move, and edit text in a two dimensional space. The pointing deice indicates the location of an insertion, deletion, or change

Types of Devices

  • Direct control device: Easy to learn and use, but hand my obscure display
    • Lightpen, touch screens, stylus
  • Indirect control: Takes users time to learn
    • Mouse, trackball, joystick
  • Non standard devices and strategies: used for special purposes
    • Multitouch tables and display, eye-trackers, sensors, foot controls, digital paper
  • Criteria for success:
    • Speed and accuracy
    • Efficacy for task
    • Learning time
    • Cost and reliability
    • Size and weight

Non standard devices, the other white meat.

* Multiple touch touchscreens:
* Allows users to use both hands or multiple fingers, allows multiple users to work together on a shared surface.

  • Bimanual input:
    • Can facilitate multitasking or compound tasks
    • Non-dominant hand sets a frame of reference in which the dominate hand operates in a more precise fashion. Non-dominant hand selects actions while the dominant hand selects the objects of the operation.


  • Eye trackers:
    • Gaze detecting controllers that use video camera image recognition of the pupil position to give 1 or 2 degree accuracy.
    • Problems occur since every gaze has the potential to activate an unintended command. Combining eye tracking with manual input is one way to address the problem

* Ubiquitous computing:
* Depend on embedding sensing technologies into the environment. Such as Radio frequency identification tags can trigger reloading of personal files on a computer.
* Positioning of physical objects can specify modes or trigger actions.

  • Paper and Digital pens:
    • Using special paper and a digital pen that has a small camera in its tip that records pen strokes drawn on a special paper printed with a unique pattern that identifies the location of each stroke


  • Sensors:
    • Are added to handheld devices can enrich the interaction with the devices themselves. Example, the accelerometers in the iPhones that can detect changes in the devices orientation, causing the display to switch between portrait and landscape.


  • Data Gloves:
    • Made of sleek black spandex with attached fiber optic sensors to measure angles of finger joints.
    • This allows for commands such as closed fist, open hand, middle finger and so on.
    • Attractive to game developers and virtual reality devotees.
    • _GLOVE-2.gif


“Fitts' law is a model of human psychomotor behavior developed in 1954 [6]. Extending Shannon’s theorem 17 in information theory. Fitts discovered a formal relationship that models speed/accuracy tradeoffs in rapid, aimed movement (not drawing or writing). According to Fitts’ Law, the time to move and point to a target of width W at a distance A is a logarithmic function of the spatial relative error (A/W), that is [11]”
MT = a + b log2(2A/W + c)
MT is the movement time
a and b are empirically determined constants, that are device dependent.
c is a constant of 0, 0.5 or 1 (refer to [11] for details)
A is the distance (or amplitude) of movement from start to target center
W is the width of the target, which corresponds to “accuracy”

8.4 Speech and Auditory Interfaces


  • Obstacles to speech recognition
  • Increased cognitive load compared to pointing
  • Interference from noisy environments
  • Unstable recognition across changing users, environments, and time
  • Obstacles to speech output
  • The slow pace of speech output compared to visual displays
  • Ephemeral nature of speech
  • Difficulty in scanning/searching


  • When users have vision impairments
  • When the speaker’s hands are busy
  • When mobility is required
  • When the speaker’s eyes are occupied
  • When harsh or cramped conditions preclude use of a keyboard


  • Speech store and forward
  • Discrete-word recognition
  • Continuous-speech recognition
  • Voice information systems
  • Speech generation

Discrete-Word Recognition

  • Recognizes individual words spoken by a specific person; works 90 – 98% reliability
  • Speaker-dependent training – repeat full vocabulary once or twice
  • Speaker-independent training – not as accurate but eliminates training to expand the scope of commercial applications.
  • Quiet environments, head-mounted microphones, and careful word choices improve recognition rates to both

Continuous-Speech Recognition

  • Speech dictation products, IBM Via Voice: to help dictate letters and compose reports verbally for automatic transcription.
  • Can be used to scan and retrieve radio or television programs, court preceding, lectures or telephone calls for specific words or topics.
  • Summaries can be generated from audio conversations.
  • Close Caption generation
  • Used for security systems and voice graphs in court

Voice Information Systems

  • Interactive Voice Response systems: to check on airline flight departure/arrival times
  • Voicemail Systems: Telephone based speech systems enabling users to store or forward messages.
  • Apple’s iPod: allows to store large audio databases and retrieve selected segments.

Speech Generation

  • Widespread in consumer applications and telephone applications
  • Used in Automobile navigation systems, Internet services, utility-control rooms, and children’s games.
  • Inexpensive, compact, reliable
  • Applications are good for the blind for text-to-speech utilities like Microsoft Windows Narrator
  • Speech enabled readers: for documents, newspapers, statistical data, and maps
  • Web-based voice applications and telephone-based voice information systems.

Non-Speech Auditory Interfaces

  • The bells and whistle sounds from a computer.
  • Auditory Icons: useful distinction between familiar sounds
  • Earcons: sounds whose meanings must be learned by use of Auditory Icons to help reinforce the visual metaphors in a graphical user interface.

8.5 Displays - Small and Large


  • The display is the primary source of feedback to users from the computer.
  • Important characteristics:
    1. Physical size
    2. Resolution (number of pixels available)
    3. Number of colors available
    4. Luminance, contrast, and glare
    5. Power consumption
    6. Refresh Rates
    7. Cost
    8. Reliability
    9. Usage characteristics

Display Technology

  • CRT (Raster-scan Cathode Ray Tubes)
    • have mostly vanished, and have been replaced by LCDs
  • LCD (Liquid Crystal Displays)
    • Thinner
    • Lighter
    • Capable of higher resolutions
    • More energy efficient
    • Great for general purposes
  • Plasma displays
    • Brighter
    • Have a better side viewing angle
    • Use more electricity
    • Better for control rooms, public displays, and conference rooms
  • LED (Light-Emitting Diode) display
    • Mostly used for large public displays
    • LED technology is advancing making it usable for smaller applications.
  • Electronic Ink
    • Uses charged black and white particles
    • Only consumes power when the display changes
    • Used in eBooks
  • Pico Projectors
    • becoming more available
    • may soon be able to project images from a mobile device
  • Braille Displays
    • All other displays don’t accommodate the visually impaired
    • Consists of up to 80 cells, each displaying a separate character
    • Can be mounted on a mouse or above a keyboard

Large Displays

  • Three main kinds:
    1. Informational Wall Displays
    2. Interactive Wall Displays
    3. Multiple-desktop Displays
  • Hybrid combinations are possible

Informational Wall Displays

  • Common in control rooms, classrooms, and military operations.
  • Help to build awareness through a common understanding.
  • Used to use CRT matrices.
  • Now mostly use frontal or rear projection.
  • Do not require high resolution.

Interactive Wall Displays

  • Traditional desktop interaction techniques become impractical.
  • Digital whiteboard
    • SMART Board
  • They can:
    • Manage brainstorming information
    • Provide new creative tools

Multiple-desktop Displays

  • Usually employ multiple traditional displays.
  • Allows users to have multiple open windows.
  • Most used in creative applications.
  • Increased workspace or increased clutter?
  • Can detract the users focus of attention

Heads-up Displays

  • HUD projects information into the users field of vision.
    • Most often used in vehicles: cars, planes, etc.
    • User can remain focused.

Helmet-mounted Displays

  • HMD is used in virtual or augmented reality.
    • Lets the user see the information even when they turn their head
      • Maybe the see different information
    • Have to carry hardware

Mobile Device Displays

  • Four classes based on intended usage
    1. General Purpose work
      • BlackBerry or Pocket PC
    2. General Purpose Entertainment
      • iPod
    3. General Purpose Communication and Control
      • Cell Phone
    4. Targeted Devices
      • UPS DIAD IV
  • Most used for brief or repetitive tasks
    • Should be optimized for these tasks by hiding or eliminating less important functions, reducing data entry
  • There are 5 key pairs of actions to consider when designing applications for mobile displays
    1. Monitor dynamic information sources and alert when appropriate
    2. Gather information from many sources and spread out information to many destinations
    3. Participate in a group and relate to individuals
    4. Locate services or items that are not visible and identify objects that are seen
    5. Capture information from local resources and share your information with future users
  • Applications are designed to use every available pixel.
    • Being able to zoom is highly important.
  • Poor readability is an issue.
    • Low light or bad eyesight
      • May be improved with Rapid Serial Visual Presentation
  • Some applications remain ineffective on small displays
    • Data modification – summarizing and making smaller
    • Data suppression – eliminating or selectively sampling
  • Most mobile users only have one free hand
    • Guidelines for mobile interfaces support one-handed use
      • Placing targets close to one another
      • Allowing users to chose between left or right-handed operation
      • Placing targets towards the center of the device
    • May change due to diversity of devices and their input devices
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License