showstack

Start here

The senses nobody counts

Sight and hearing get all the attention because they are the ones a show buys equipment for. They are not the ones doing the work of locating you.

The useful split

Two illusions, and they fail separately

Mel Slater's distinction is the most practically useful idea in this whole area, and it applies to a set, a room and an installation exactly as much as to a headset.

Owning a body

The rubber hand, and why it matters here

Hide someone's hand, put a false one where they can see it, and stroke both at exactly the same moment. Within a minute or two most people report that the false hand feels like theirs — and will flinch when it is threatened.

The mechanism is straightforward once stated. The brain is continuously deciding which body it is in, based on which sensory streams agree. Vision says the touch is happening there; touch says a touch is happening now; the two are synchronous; the simplest explanation is that there is where the hand is. Break the synchrony by even a modest delay and the effect collapses.

For anything with an avatar, a puppet, a prosthetic or a tracked object, this is the working principle: ownership follows synchrony. A virtual hand that moves with yours becomes yours within seconds. Add latency, or make the movement approximate, and it stays a graphic. The tolerance here is tight — the same order as the audiovisual binding window, and for the same reason.

Causing things

The sense of agency, and why latency destroys it

Separately from owning a body, there is a sense of having caused something — and it too is inferred rather than known directly. The brain compares what it predicted an action would do with what happened, and if they match closely enough in time and content, it concludes you did it.

The timing tolerance is remarkably short. Push it out and the feeling of authorship fades even though the causal chain is unchanged: you know you pressed the button, and it stops feeling like you made the thing happen. There is a measurable companion effect — an action and its outcome are perceived as closer together in time when you caused them, sometimes called intentional binding — and it disappears as the delay grows.

This is why interactive work lives or dies on latency in a way that video playback does not. A hundred milliseconds in a linear show is nothing. A hundred milliseconds between a visitor's gesture and the response is the difference between "I did that" and "something happened".

Breaking it

What actually ends presence

Presence is experienced as binary — you are either in the place or you are in a room with equipment in it — while its causes are continuous. A break in presence is usually one cue crossing a threshold, and the same short list is responsible almost every time.

Latency and mismatch. The world lagging behind your head, or moving when you did not move it. The special case is a conflict your body cannot resolve: eyes reporting motion while the vestibular system reports stillness. Camera and performer position reach a render engine over FreeD, PSN and RTTrPM, and the latency of that path is the thing being described here. That is not confusing, it is nauseating — a hard physiological limit rather than a quality setting.

A visible seam. The edge of a projection, the join in an LED wall, a masking gap, a light spilling from a doorway, a technician crossing upstage. Attention is drawn by contrast, so any seam becomes the most interesting thing in the frame.

Sound from the wrong place. Sound localisation is fast, accurate and involuntary. A voice that comes from a PA rather than a body is one of the strongest presence-breaking cues there is, which is exactly what object audio — a DS100, an L-ISA system — exists to fix.

A world that does not notice you. The plausibility failure — the thing you cannot touch, the character who looks through you, the state that visibly resets.

The real world getting in. A phone, a draught from the wrong direction, a queue visible past the set, the smell of the foyer. Presence is a whole-body agreement, and it only takes one dissenter.

How much delay before the world stops being yours?

Motion-to-photon is the time between your head moving and the light for that new viewpoint reaching your eye. Everything on this page has a threshold somewhere along it.

Building it

What actually helps

Get the head right first. Tracking quality and motion-to-photon latency buy more presence per pound than resolution does, every time. A lower-fidelity world that responds correctly beats a beautiful one that lags.

Give the world a way to notice people. One genuine response — a light that follows, a sound that acknowledges, a door that opens because you approached — does more for plausibility than a great deal of visual detail.

Design the threshold. The transition into the space is when presence is established or lost. A corridor, a dimming, a change of temperature, a moment of held silence — a deliberate boundary gives the body time to accept a new place.

Use the uncounted senses. Airflow, temperature, floor texture, a subtle low-frequency floor vibration. These are cheap relative to visual fidelity and they land on channels nothing else is competing for.

Hide the seams or make them intentional. If a visitor can see the edge, either mask it or make the edge part of the fiction. What you cannot do is hope they will not look.

Choose comfort deliberately. Locomotion that moves the view without moving the body is the primary cause of sickness, and the mitigations — teleporting instead of gliding, narrowing the field of view during motion, giving a stable reference in view — cost some elegance and are worth it.

Why this page is here

A set and a headset are the same problem

It would be easy to file all of this under virtual reality. That would be a mistake, because everything on it applies to a black box, an arena and a promenade piece.

A set convinces because the sightlines hold from every seat, the sound comes from where the thing is, nothing visible contradicts the fiction, and the world reacts. A headset convinces for exactly the same reasons, using exactly the same senses, with tighter tolerances because it has taken responsibility for more of the signal.

And both of them are working on a body that has been quietly running the prediction machinery the previous pages describe — comparing what it expected against what arrived, and concluding, or not, that it is somewhere.