Shneiderman · 1983
Before 1983, most people typed commands
Direct manipulation means working on the thing itself: you drag the file, move the slider or click the cell, and see the result right away.
To copy a file you had to remember a command, its spelling and its order of arguments. Shneiderman asked why some new systems felt so different.
Command line
You must recall the syntax, and success is silent.
Direct manipulation
Drag the file onto the folder.
The file and the folder are on screen; you see the result at once.
Shneiderman · 1983
The systems he pointed to
Shneiderman built his argument from systems people already loved. Try a small version of each.
Full-screen display editor
Original ↗Fix the typo “teh” twice: once in the line editor, once by clicking in the page.
Principle 1: the whole document stays in view.
VisiCalc spreadsheet
Original ↗Change any monthly amount. Every total updates as you type.
| A | B | C | |
|---|---|---|---|
| 1 | Item | Month | Year |
Principle 3: each small change shows its effect at once.
Xerox Star desktop
Original ↗Drag the document onto the folder, the printer or the wastebasket.
Principle 2: you point and drag; there is nothing to spell.
Arcade video games
Original ↗Move the pointer (or your finger) to steer the paddle. No manual needed.
All three: most people learn a game like this by watching someone play for a minute.
Computer-aided design
Original ↗Drag a corner of the floor plan. Area and perimeter follow.
His analogy: driving
Turn the wheel and watch the car’s path change, instead of typing “turn left 30 degrees.”
Shneiderman · 1983
Three principles of direct manipulation
The principles
- Keep the objects visible. The document, cells or files stay on screen the whole time.
- Act, don’t type. Point, drag or press a labeled button instead of writing a command.
- Small, fast, undoable steps. Each action shows its effect at once and can be reversed.
What users gain
- Novices learn quickly, often by watching someone else.
- Experts work fast.
- Occasional users remember how it works.
- Error messages are rarely needed.
- You see right away if you’re getting closer to your goal.
- You feel in control.
Try it: one task, two ways
Task. Find the High-priority block groups (TEPI, ACS 2019–23) whose census profiles look alike. Where are they on the map? Were they already High priority in 1990? Do it with direct manipulation, then switch to the command line and do it again.
Map: TEPI tier by block group ACS 2019–23
Similarity: UMAP of 12 indicators near = similar profile
Ask the agent
Choices it made for you
Assistant
Watching. Make a selection.
Select
Draw around points or block groups on either view. Hold Shift to add.
Selection
Effort
Data: Tucson Equity Priority Index (City of Tucson), recomputed for 14 census releases from 1990 to 2024 in Aditya Retnanto’s CSC 596 Project 2. 718 block groups; UMAP is fitted once on the 12 TEPI indicators (2019–23, percentile-ranked), so the release slider only changes colours. A few rural block groups fall outside the map frame.
Where it runs out
What direct manipulation struggles with
- Scale. 718 block groups fit on a screen; fifty thousand emails don’t.
- Repetition. There is nothing to drag for “do this every month.”
- Hidden things. Future events and rules have no icon to grab.
- Learning the visuals. A lasso or a UMAP plot still has to be explained.
- Access. It assumes you can see and point.
- Effort. You do every step yourself.
After 1983
Where direct manipulation went next
The same idea, grabbing the thing itself, moved off the desktop: into physical objects, hardware and code.
Tangible interfaces
More ↗Ishii & Ullmer’s “Tangible Bits” (1997). In Urp (1999), planners moved model buildings on a table and saw projected shadows move. Drag the buildings and change the time.
Extends: the object you touch is a real one.
Physical computing
More ↗Since Arduino (2005), anyone can wire a knob or sensor to code. Drag the knob up and down; the reading drives the light.
Extends: your body and the room become the input.
Direct manipulation of code
More ↗Bret Victor’s “Inventing on Principle” (2012): creators need an immediate connection to what they make. Drag any orange number sideways.
Extends: even programs get continuous feedback.
More offshoots
- Dynamic queries (1992). Sliders that filter a display instantly. The TEPI demo is one.
- Multi-touch (2006–07). Jeff Han’s demos, then the iPhone: pinch and drag with fingers.
- VR and AR hands (2020s). Grab and pinch virtual objects in headsets.
Maes · 1994
Too much to do by hand
An interface agent is software that does a task for you, like an assistant, instead of you doing every step yourself.
The problem
- Information overload. More email, news and files than anyone can sort.
- New users. Most people using computers were never trained.
- Direct manipulation doesn’t scale. You still do every step yourself.
Her proposal
- Indirect management. Alan Kay’s phrase: you delegate, the computer acts.
- Interface agent. A personal assistant that works beside you in the same program.
- Delegation. Handing over a task, and with it some control.
Maes, P. (1994). Agents that reduce work and information overload. Communications of the ACM 37(7).
Maes · 1994
Every agent has two problems
The two problems
- Competence. How does it know when to help, and how?
- Trust. Will you feel safe letting it act for you?
Three ways to build one
| Approach | Competence | Trust |
|---|---|---|
| You write the rules | Only as good as your rules; you must spot every pattern and keep the rules up to date. | High: you wrote them. |
| Experts build it in | Lots of work up front; fixed, and not fitted to you. | Low: you don’t know what it knows. |
| It learns from you (Maes) | Grows as it watches you work. | Grows as you watch it get things right. |
Maes · 1994
How her agents learn
Four sources
- Watching you. It records what you do in each situation and imitates it.
- Your feedback. It learns when you accept or undo what it did.
- Examples. You can show it a case on purpose.
- Other agents. It can ask agents that serve other people.
When it acts
- Confidence. It compares the new case with similar past ones.
- “Tell me” threshold. Above it, the agent suggests.
- “Do it” threshold. Above it, the agent acts on its own.
- You set both. You decide how much to hand over.
Works best when tasks repeat a lot and differ from person to person.
Try it: train an email agent
Mock-up · not a real AITask. Sort the incoming email with the buttons. A Maes-style agent watches. Once it has seen enough, it starts suggesting, then acting. Drag the two thresholds to give it more or less control, and undo anything it gets wrong.
Inbox
Handled
Agent 0 examples in memory
Why the most similar past emails
Who did the work
Agent states
Maes’s agents showed a face so you could see what they were doing.
A simplified version of Maes’s memory-based reasoning: every email you sort is stored with its sender, recipients and subject words; a new email gets the action of its most similar stored emails. Senders and messages are made up.
The TEPI task, handed to an agent
Mock-up · scripted steps, not a real AITask. Same data, same views as before. This time, ask the agent and watch it drive the controls for you. Then compare: whose analysis is it? What did it decide without asking?
The agent only knows three kinds of request; it follows fixed steps and then lists the choices it made. Data as on the earlier TEPI page.
Maes’s group, 1993–95
The agents they built
Four assistants
- Email (Maxims). Learned to prioritize, delete, forward and file mail.
- Meetings. Learned when you accept, decline or reschedule.
- News (NewT). Learned which articles you read.
- Music (Ringo). Matched you with people who rate music like you do.
Try Ringo’s idea: rate a few albums
Ringo: Shardanand & Maes (CHI 1995), “social information filtering.” The other listeners here are made up.
Where it runs out
What interface agents struggle with
- Slow start. It is no help until it has watched you for a while, as in the email demo.
- New situations. A sender or task it has never seen leaves it unsure, like the email from Dad.
- Silent mistakes. When it acts on its own, errors can go unnoticed, like the “URGENT” email filed away.
- Hidden choices. It decides things you didn’t ask about, like what “similar” means in the TEPI agent.
- Only repeated tasks. It needs a pattern to learn; one-off tasks give it nothing.
- Privacy. To learn your habits, it has to watch everything you do.
After 1994
Where interface agents went next
Learning from you
- Recommendations. Ringo became Firefly (1996); Amazon, Netflix and Spotify use the same “people like you” idea.
- Spam filters (2000s). Every “mark as spam” teaches the filter.
- Priority inboxes (2010s). Mail apps learn which messages you open first, like Maxims.
Agents with a face, and a warning
- Microsoft’s Office Assistant, “Clippy” (1997). Popped up with help nobody asked for, and became a famous failure.
- Its ideas came from the Lumière project at Microsoft Research, led by Eric Horvitz. Hold on to that name.
- Voice assistants (2010s). Siri and Alexa act on spoken requests.
Recap before the debate
Two schools so far
Direct manipulation · Shneiderman 1983
- The idea: you do it yourself, by grabbing and moving things on screen.
- Principles: keep objects visible, act don’t type, small undoable steps.
- Strengths: easy to learn, predictable, you feel in control.
- Struggles: scale, repetition, things you can’t see, and you do every step.
- In our demos: lasso and sliders on the TEPI map.
Interface agents · Maes 1994
- The idea: software does the task for you, like an assistant.
- How: it learns by watching you, and acts once it is confident enough.
- Strengths: handles overload and repetitive work.
- Struggles: slow start, new situations, silent mistakes, hidden choices.
- In our demos: the email agent and the TEPI agent.
In 1997 the two authors met on stage to argue it out. Which one should we build?
IUI ’97 and CHI ’97
The debate: Shneiderman vs. Maes
Shneiderman: keep people in charge
- Control. People want to feel they did it, not that something did it for them.
- Predictability. An agent that changes as it learns is hard to anticipate.
- Responsibility. When an agent errs, who is accountable?
- Fake people fail. Talking, human-like interfaces such as Microsoft Bob flopped.
Maes: people need help
- Overload. There is too much information to handle by hand.
- Not a replacement. Agents sit beside direct manipulation; you can still do it yourself.
- Not a person. An agent doesn’t need a face or a human personality.
- Personal. Each agent learns one person’s habits. Your agent is not my agent.
- Trust is earned. It learns from you, explains itself, and you set how much it may do.
Shneiderman, B. & Maes, P. (1997). Direct manipulation vs. interface agents. Interactions 4(6).
Shneiderman, in the 1997 debate
“4,000 or more items on the screen in an orderly way”
His argument, in short
- Human vision is remarkable, and interfaces barely use it.
- In 1997, a screen showed about 40 icons across two or three windows.
- He wanted 100 to 1,000 times more, laid out so people can see every option and move between them.
- His answer to overload: show more, better organized, instead of handing the work to an agent.
Discuss
- Was he right? Where do you see thousands of items on one screen today, and where do you see one text box?
- Does showing everything help people, or recreate the overload Maes was worried about?
- Who is a 4,000-item screen for: experts, or everyone?
- For a UMAP plot or an agent trace with thousands of steps: show it all, or summarize?
40 items or 10,052
Side by side
Where they really disagree
| Question | Shneiderman | Maes |
|---|---|---|
| Who acts? | You, through visible controls. | You, or the agent when you allow it. |
| What makes it trustworthy? | Being predictable and easy to understand. | A track record you can watch and correct. |
| What about scale? | Better visualizations of the data. | Delegate the repetitive parts. |
| Who is responsible? | The user, clearly. | The user, who set the thresholds. |
| Should it look human? | No. | Not necessarily. |
- They agree that users must stay in control.
- They disagree on whether handing work over can keep you in control.
Then and now
Their visions, and what actually happened
| Topic | Shneiderman’s vision | Maes’s vision | Today |
|---|---|---|---|
| Cognitive load | Let your eyes do the work: recognize what you see instead of recalling commands. | Let the agent do the work: less for you to read, sort and decide. | Chat removes command syntax, but adds writing prompts, reading long answers and checking them. |
| Human-like interfaces | No fake people. Software should feel like a tool, not a colleague. | Agents don’t need a face or a personality. | Assistants talk in the first person, have names, voices and personalities. |
| Speech and language | Pointing and seeing beat talking for most tasks. | Agents act quietly in the background, not through conversation. | Chat and voice became the main way to reach AI. |
| Who it is built for | The same visible tool for everyone, which you control. | One agent per person, learned from your habits. | One model trained on everyone, lightly personalized. |
| Who acts | You do. | The agent acts for you: files, schedules, filters. | Started as conversation; agents that act are only now arriving. |
Then and now
Questions: visions versus reality
- Whose vision is closer to today’s reality? Where did both of them miss?Use the table: which rows surprised you?
- Does chat reduce cognitive load, or just move it?From learning commands to writing prompts and checking answers.
- Today’s assistants talk like people. Would Shneiderman or Maes approve?What does a human-like voice make easier, and what does it hide?
- Maes imagined agents that act quietly, not ones you talk to. Is a chat window really a Maes agent?Or is it a command line with a friendlier voice?
- When is speech the right interface, and when is it the wrong one?Think about busy hands, accessibility, quiet offices, and thinking while talking.
Your turn
Where do you stand?
For each task, slide toward doing it yourself or handing it to an agent.
Discussion
Questions on the debate
In general
- When software acts for you and gets it wrong, who is responsible?You, the designer, or the software?
- Is predictability or a track record a better reason to trust a tool?Shneiderman’s answer versus Maes’s.
- Can you hand over a task and still feel in control?What would the tool need to show you?
- Should an agent look or talk like a person?When does that help, and when does it mislead?
- Which tasks should never be delegated?What makes them different?
In your project
- What does the person do directly, and what does the system do for them?Draw the line between the two.
- If you handed one step to an agent, which step would it be?What would it get wrong first?
- Would your users rather feel in control, or have the work taken off their hands?How do you know?
Horvitz · 1999 · today’s reading
Stop choosing sides
Mixed initiative means you and the computer share the work. Either one can take the lead, and the computer steps in only when that is likely to help.
The idea
- Keep direct manipulation. You can always do it yourself.
- Add automated help. The computer offers to do parts of the task.
- Decide when to help. Weigh how sure it is against what a wrong guess would cost you.
His example: Lookout
- An add-on for Outlook email. Built at Microsoft Research.
- It reads incoming mail. It guesses whether you want to put a meeting on your calendar.
- Then it does nothing, asks, or acts, depending on how sure it is.
Horvitz, E. (1999). Principles of mixed-initiative user interfaces. Proceedings of CHI ’99, 159–166.
Horvitz · 1999
What “utility” means
Utility is a score for how good or bad an outcome is for you. Positive helps you; negative costs you. It’s like deciding whether to carry an umbrella: how likely is rain, and how bad is getting wet compared with carrying it for nothing?
Lookout’s four outcomes
| You wanted a meeting | You didn’t | |
|---|---|---|
| It adds the event | +6 Saved you the work | −5 You have to delete it |
| It stays quiet | 0 You do it yourself | 0 Nothing happens |
The numbers are made up for this example; Horvitz’s point is that a system should weigh them.
Expected utility: weigh each outcome by its chance
Horvitz · 1999
Decision theory, in one picture
Decision theory is a way to choose when you can’t be sure what will happen: list your options, how likely each outcome is, and how good each outcome would be, then pick the option with the highest expected utility.
The four ingredients, for Lookout
- Actions: stay quiet, ask, or add the event.
- What might be true: you want a meeting, or you don’t.
- Probability: how sure it is, from reading the email.
- Utility: how good each outcome is for you.
Horvitz · 1999
Act, ask, or stay quiet?
Same idea with a third option, asking. Each choice has a payoff (its utility):
- Stay quiet: nothing gained, nothing lost.
- Act: a big win if you wanted it, annoying if you didn’t.
- Ask: a smaller win, at the small cost of interrupting you.
Horvitz calls this choosing the action with the highest expected utility: the payoff of each choice, weighted by how likely you are to want help. The two crossing points are his thresholds.
Horvitz · 1999
Twelve principles, in plain words
Be worth it
- Add real value beyond what you could easily do yourself.
- Keep learning by watching what you do.
Know what you don’t know
- Be unsure about goals; don’t assume.
- Weigh costs and benefits before acting.
- Ask when a question would settle it.
- Help less precisely when unsure, e.g. open the right week, not the exact hour.
Respect the person
- Mind attention: don’t interrupt at a bad moment.
- Easy to call up and to dismiss.
- Make bad guesses cheap to ignore or undo.
- Behave politely, as people expect.
Work together
- Let the person refine what the agent produced.
- Remember recent interaction so you don’t repeat yourself.
Numbers follow the paper’s list of “critical factors.”
Try it: a Lookout-style assistant
Mock-up · not a real AITask. Open each email. The assistant waits while you read, guesses whether you want a meeting on your calendar, then stays quiet, asks, or drafts the event, depending on its two thresholds. You can always press “Add to calendar” yourself.
Inbox
Message
Assistant
What it noticed
Your calendar
The TEPI task, mixed initiative
Mock-up · scripted suggestionsTask. Do the task yourself with the lasso and the slider. The assistant watches, waits until you stop dragging, and offers help only when it is sure enough. Accept, ignore or dismiss it.
After 1999
Mixed initiative today
- Code suggestions in grey text. Press Tab to accept; keep typing to ignore. A bad guess costs nothing.
- Smart Compose in email. Offers the end of your sentence while you type.
- “Allow this action?” AI agents that pause and ask before doing something risky.
- Clarifying questions. A chatbot that asks what you meant before answering.
- Driver assistance. The car keeps its lane, and hands control back to you when unsure.
- Clippy, the counter-example. Interrupted at the wrong time with unwanted help. See what it looked like ↗
Today
AI is rebuilding these paradigms from scratch
Three moves
- The command line came back. ChatGPT (2022) and assistants like Claude: you type a request in plain English instead of code, and it prints text back.
- With an agent inside. Plain English is ambiguous, so the system has to guess what you meant. That guessing is Maes’s agent.
- But not personal in Maes’s sense. Her agents learned only from you. Today’s assistants learn from huge amounts of other people’s data first, and know you only through a short memory or instructions.
- Agents learned direct manipulation. AI agents now click buttons and fill forms in apps built for people.
- Direct manipulation is being rebuilt around AI. Accept a suggestion with Tab, edit the document it wrote, undo one step.
The question
- Is the future pure Maes, or mixed initiative?
- Mixed initiative assumes direct manipulation won. Agents are relearning how to let you grab things.
- Who decides where each task sits, and who controls what you see?
This page was built by talking to one of these assistants, in a chat window.
IEEE VIS 2026 · VISions talks
The visualization community is having this debate now
Screenshot: ieeevis.org, VISions: Zhu-Tian Chen; Ana Crisan.
Two invited talks
- Zhu-Tian Chen: “Will Visualization Help Humans Act on Data or Help AI Act on Humans?” AI now does more of the analysis, and people will delegate those steps (Maes). He argues visualization should guard the last step, what a person finally sees, as a layer the AI can’t control that asks for attention only when it matters (Horvitz).
- Ana Crisan: “Ways of Thinking, Ways of Doing, and the Future of Visualization Research.” If AI can build designs and gather evidence, what is left for researchers? She points to openness, reproducibility and human agency.
Discussion
Questions on mixed initiative
In general
- Does mixed initiative settle the debate, or just move it?The argument becomes: who sets the thresholds?
- Who should set the thresholds: the user, the designer, or the system?What goes wrong with each choice?
- When is asking worse than acting or staying quiet?Think about the cost of an interruption.
- How should a system know when it is a good moment to interrupt?What signals could it use?
- Which of Horvitz’s principles do today’s AI assistants follow, and which do they break?Think of the chat and agent tools you use.
In your project
- Which paradigm is your project built on: direct, agent, or a mix?Point to one feature that shows it.
- Where does your project already let software decide for the user?A model, a default, a recommendation.
- What does a wrong guess cost in your project, and who pays for it?Put rough numbers on it, like the utility table.
Wrap-up
Map the room
Add each person’s project and place it between doing it yourself and handing it to an agent.
Nothing is saved: this list clears when the page is reloaded.