Chapter 5

Direct Manipulation and Virtual Environments

Go Back to Exam 1 Review


Direct manipulation interfaces enable users to visibly, rapidly, reversibly, and incrementally perform actions that had traditionally required typed commands to perform. The satisfaction users feel as the result of using these interfaces leads to beneficial behaviors such as an eagerness to show off an interface to novice users, and the desire of users to learn more advanced concepts of an interface.

There have been several new technologies that extend upon direct manipulation including virtual reality, augmented reality, and other tangible/touchable user interfaces.

Examples of Direct Manipulation

Direct manipulation may be thought of like driving an automobile. You turn the steering wheel and the car turns, press the gas pedal and the car goes forward. In contrast command based interfaces might be compared to a captain directing his ship. The captain gives a command, but the helmsman does the turning.

The Legacy of Word processors, Spreadsheets, and Office Automation

Word processors can be thought of as the great-great-grandfather of most direct manipulation interfaces. WYSIWYG word processors provide users with a complete page to edit. The placement and styling of text that a user sees on the screen is also what the document will look like printed. Coupled with the immediacy of actions and their results, the use of the mouse cursor to select and modify text in a physical way, and the ease of reversing ones actions and it is hard to imagine trying to type a professional document using something as rudimentary as Notepad.

The introduction of VisiCalc in 1979 introduced users to a visual way of manipulating data with immediate results. Spreadsheets offered users a way to test alternate calculation of the same data with the results being displayed on the screen at the same time.

Early office automation systems also used direct manipulation techniques. The use of a mouse cursor to interact with a constantly updating graphic interface, allowed users to enjoy simplified procedures for accomplishing tedious tasks. In short, users now had continuous and immediate feedback on their actions and inputs.

Contemporary examples of direct manipulation interfaces

  • Spatial data management
    • These systems create spatial models based on reality. Buildings, geography, and file systems are represented in 3D spaces with appropriate graphical representations and layouts.
  • Video games
    • Video games represent a perfect example of direct manipulation interfaces. Video games, even more so than spatial data interfaces, involve users in a dialog where users are given prompt and immediate feedback based on their their interaction with the game.
  • Computer-aided design
    • Direct manipulation in computer-aided design allows users to shape, build, design, and create objects and then manipulate them.

Discussion of Direct Manipulation

Direct manipulation interfaces let users involve themselves directly with objects instead of forcing them to deal indirectly through an intermediary. They are powerful tools for problem-solving and education because users naturally interact with them. It is generally easier for people to retain and manipulate physical, spacial and visual representations than it is for them to do the same with text based representations.

  • Problems with direct manipulation
    • Direct manipulation interfaces depend on spatial and visual representations. This means they may not be the best interface for use by those who are blind or visually-impaired. This spatial-visual representation may also force potentially important information off screen, requiring users to scroll to or perform multiple actions to get to that information. Representations must be meaningful and easy for users to understand. Problems may arise if users misinterpret the meaning of chosen symbols, and users may be mislead into over or underestimating the actual function of an application. Experienced users may find frustration in removing a hand from the keyboard to use a mouse if a simple command could have accomplished the task. Mobile devices with small screens can have problems with screen space being taken up and smaller representations being difficult to see or select.
  • The three principles of direct manipulation:
    1. Continuous representation of objects and actions with meaningful visual metaphors.
    2. The use of physical actions or labeled buttons instead of complex syntax.
    3. Rapid, incremental, reversible actions whose effect on objects is immediately visible.
  • Visual thinking and icons
    • Depending upon the context and users of an application, the use of text, visuals, or both may be justified. That is the major point to remember however, the use of either(text, visuals, or both) must be justified for the context of use and the users.
  • Direct-manipulation programming

3D Interfaces

3D interfaces are not suitable for all applications. Industries that benefit from 3D interfaces include medical, architectural, manufacturing, aviation, and of course video games. Microsoft Office is an example of when a 3D interface might not be the appropriate choice. 3D techniques that might enhance a 2D interface would be raised/depressed buttons, overlapping windows, and representations of real-world objects.


  • 3D techniques such as shadows and perspective should only be included when necessary.
  • Keep navigational steps to a minimum.
  • Make text readable and with no more than a 30 degree tilt.
  • Keep visual clutter at a minimum to avoid distractions.
  • Make user movements simple.
  • Prevent errors.
  • Object movement should be simple and predictive.
  • Keep similar items grouped together for easy visual identification.
  • Allow users to organize objects to facilitate spatial recall.

Enhanced 3D Guidelines

  • Include a dashboard or overview.
  • Allow teleportation.
  • Offer x-ray vision to see into or beyond objects.
  • Show logging and undo actions.
  • Permit users to perform actions on objects such as save and copy.
  • Allow collaboration with other users.
  • Let users control explanatory text such as pop-ups and screen tips.
  • Provide tools to select objects, mark them, and perform measurements.
  • Allow for queries to filter out unnecessary items.
  • Support semantic zooming and movement that will bring objects closer and provide more details.
  • Allow distant destination points to display themselves.
  • Show different views of the environment simultaneously.
  • Show 3D icons in ways that are more recognizable.


Teleoperation is the child of direct manipulation and process control. Teleoperation is closely linked to supervisory control. If the physical processes can be carried out from a computer at a remote location it is considered teleoperation or remote control. Typical scenarios involving teleoperation include manufacturing control, surgery, unmanned aircraft, home security systems/energy controls, and steering vehicles. It is more common for users to use point, click, and drag operation rather than enter commands. Hardware, environment, and network design are to blame for poor feedback and slow response times. Slow response times can confuse the operator as to what the current status of the system is. The military and medical communities heavily rely on this type on direct manipulation.

Virtual and Augmented Reality

VR applications are well suited for training and simulation in environments that are normal to humans. Flight simulators and architectural design simulators are persuasive arguments for "being in" rather than "looking at" environments. VR experiences can assist people in overcoming their phobias. VR can also be used as a distraction in surgery to helped dampen some pains.

Augmented Reality enables users to see the real world with an overlay of additional information. Picture looking through eyeglasses at a wall and being able to "see" the studs and electrical wires behind the walls. Desktop or fishtank displays are becoming more common because they don't cause the discomfort and can be used with standard equipment. To be successful, AR must integrate with multiple technologies. Motion must be smooth and sensor precision high. Haptic feedback and sound I/O is sometimes necessary to provide realism.

BOX 5.1 (Definition, benefits, and drawbacks of direct manipulation)


The most successful interactive systems give the user a sense of enjoyment and allow them to create their own content. Novices must find them easy to use and then be able to progress towards more detailed tasks. Actions must be fast and reversible. Feedback must be immediate and in turn this leads to less errors. Iterative design is of utmost importance when testing direct manipulation systems. These approaches in direct-manipulation can lead to unpredictable problems. The potential for these systems is compelling but not all applications are suitable for direct manipulation.

Midterm Exam
Chapter 4
1 2 3 4 5 6 7 Chapter 6
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License