showstack

Hearing, part one

How two ears become a direction

There is no organ for direction. There are two ears about 20 cm apart, and everything else is arithmetic done on the difference between them.

Time. A sound from the left reaches the left ear first — up to roughly 700 microseconds earlier for a sound directly to one side. That interaural time difference is the dominant cue below about 1.5 kHz, where a wavelength is long enough that the phase difference is unambiguous.

Level. Above that, the head itself becomes an obstacle and casts an acoustic shadow, so the far ear hears less. That interaural level difference takes over as the main cue for high frequencies.

Shape. Both of those are symmetrical front to back, which leaves a whole cone of positions that produce identical differences — the cone of confusion. What resolves it is the pinna: the folds of your outer ear filter incoming sound differently depending on where it came from, notching particular frequencies. Your brain learned your own ears' filtering in childhood, which is why generic binaural audio never quite works on everybody and why individualised head-related transfer functions are a research industry.

And when all of that fails, you move your head. A tiny movement changes the differences in a way that is different for a source in front and one behind, and the ambiguity collapses. Head movement is the single most powerful localisation cue there is, which is exactly why a fixed headphone image feels artificial and a head-tracked one does not.

Hearing, part two

How one ear becomes a pitch — and why a mix is a competition

The cochlea is a tapered tube, and where along it a frequency causes the greatest movement depends on that frequency: high notes near the entrance, low notes deep inside. That is tonotopy, and it means pitch is partly just which hair cells moved — a place code, the same principle a cochlear implant exploits.

Below roughly 4 to 5 kHz there is a second mechanism running alongside it: nerve fibres fire in step with the waveform, so the timing of the spikes carries the frequency directly. Two codes, overlapping, which is part of why pitch perception is so much better than the mechanical tuning of the cochlea alone would allow.

The consequence that matters on a show is masking. A loud sound raises the threshold for anything close to it in frequency — the neighbours are still arriving, and they stop being heard. Masking is asymmetric, spreading further upward in frequency than downward, and it is why a mix is not a set of independent channels but a competition for the same detectors. Carving a hole for a vocal is not a stylistic choice; it is the only way to make room in a mechanism that is already full.

Seeing

Three overlapping detectors, and a colour that is not in the light

There is no such thing as a red photon. Wavelength is a physical property; colour is a construction, and it is built from exactly three numbers.

The retina has three cone types with peak sensitivities at roughly 420, 534 and 564 nm — conventionally short, medium and long. They are very broadly tuned and overlap heavily, particularly the M and L cones, which sit remarkably close together. No single cone can tell you a wavelength: a middling response could be a weak light at its peak or a strong light off to one side. What identifies a colour is the ratio between all three.

That is exactly why metamerism works and why three numbers can encode a colour at all. If vision sampled the spectrum finely, RGB would be hopeless. It works because we only ever had three samples.

Downstream, those three signals are immediately recombined into opponent channels: light-versus-dark, red-versus-green, and blue-versus-yellow. Which is why you cannot imagine a reddish green — the channel that would have to carry it can only point one way at a time — and why staring at a colour and looking away gives you an afterimage in its complement.

And the parts of that which are not universal

Colour vision deficiency is common. Roughly 8% of men and about 0.5% of women, overwhelmingly red-green, usually because the M and L cone pigments are shifted closer together than typical. For those people red and green are not confusable in some abstract sense — they are genuinely closer together.

Which makes one design rule non-negotiable and constantly broken: never encode meaning in red versus green alone. Ready and not-ready, armed and safe, pass and fail. Add shape, position, text or brightness. This applies to a control surface, a status page, a cue light and a signage system, and it costs nothing at design time.

Rods are a different sense. Roughly 120 million of them against about 6 million cones, far more sensitive, no colour information at all, and almost none in the fovea — so in a dark room the centre of your gaze is the blind spot for dim things, and you genuinely see a faint star better by looking slightly away from it. It is also why a deeply dark scene reads as monochrome no matter what colour the light is, and why the Purkinje shift pulls sensitivity toward blue at low levels.

Seeing depth

Two eyes, and the eight cues that do most of the work

A retina is flat. Depth is not in the image — it is reconstructed, and stereo vision is only one of the ingredients and not the largest.

Binocular disparity is the famous one. Your eyes are about 63 mm apart, so each sees a slightly different image, and the horizontal difference between where a point lands in the two is the disparity. It is the only cue that gives genuinely absolute depth, and its useful range is short — a few metres. Beyond about ten metres the disparity between two eyes 63 mm apart is too small to signify, which is why stereo 3D of a distant landscape looks like a flat backdrop.

Convergence adds a little more: how far your eyes have to turn inward to fuse an object. Also short-range, and it is the cue that gets into trouble on a screen.

Everything else is monocular, works with one eye shut, and is doing most of the heavy lifting: occlusion (what covers what — the strongest depth cue there is), motion parallax (near things sweep past faster when you move), perspective, familiar size, texture gradient, shading and shadow, and aerial perspective (distant things are hazier and bluer).

Which is why a well-lit set with good shadows reads as deep on a flat screen, and why adding stereo to badly composed depth cues adds very little. Stereo is a top-up, not the mechanism.

Why 3D gives people headaches, in one sentence

In the real world, the distance your eyes converge to and the distance they focus to are always the same number. On a stereoscopic screen they are not: your eyes converge at whatever depth the disparity implies, and focus on the screen, always. That is the vergence–accommodation conflict, it is a genuine physiological mismatch rather than a preference, and it is the single largest cause of discomfort in stereo 3D and in headsets.

The mitigations are all versions of use less of it: keep the depth budget small, keep the action near the screen plane, avoid strong negative parallax for long stretches, and never ask the eyes to diverge — which happens the moment on-screen separation exceeds the distance between somebody's eyes, and which they physically cannot do.

And the budget is a percentage of screen width, not a distance, because what matters is the angle the eyes are being asked to make. The same shot that is comfortable on a laptop can be unwatchable on a cinema screen.

How it is delivered, and how it is captured

Every stereo display is solving one problem: get a different image into each eye. The methods differ only in how they separate them, and each one costs something.

Polarised — the two images carry opposite polarisation and passive glasses filter them. Cheap glasses, comfortable, and it halves light and usually resolution. The cinema standard.

Active shutter — the display alternates left and right frames and the glasses blank each eye in turn. Full resolution per eye, needs powered synchronised glasses, and is where flicker sensitivity shows up.

Anaglyph — red and cyan filters. Works on anything, wrecks colour, and survives because it needs no special screen at all.

Autostereoscopic — a lenticular sheet or parallax barrier sends different columns of pixels in different directions, so no glasses are needed. It costs resolution and it only works from particular positions, which is why it suits a kiosk and not a theatre.

A headset is the honest version: a separate display per eye, no separation trick needed, and the full presence problem instead.

Capture is the mirror image. Two cameras on a rig with an interaxial separation — often smaller than human eyes for close work, larger for distant subjects to create depth that would not otherwise exist — and a convergence setting that decides which distance sits on the screen plane. Increase interaxial and everything gets deeper; move convergence and the whole scene slides through the screen. Those two controls are the entire grammar, and getting them wrong is not fixable later.

And the newer one

Volumetric capture, which is a different thing entirely

Stereo records one viewpoint, twice. You cannot walk around it, because there is nothing behind the subject to see. Volumetric capture records the subject itself, so a viewpoint can be generated afterwards from anywhere.

The usual method is a rig of many synchronised cameras — often dozens, sometimes with depth sensors — surrounding a performer. Each frame is reconstructed into geometry with texture: a per-frame mesh, or increasingly a neural representation such as a radiance field or a set of Gaussians that encodes how the scene looks from any direction. The output is not a video, it is a moving object you can put in a scene, light, occlude and walk around.

The costs are real. Every camera must be synchronised to the frame — the shared clock problem again — and calibrated to a common coordinate system, which is the registration problem again. Data rates are enormous. Hair, fine fabric and transparency reconstruct badly. And the capture volume is a hard boundary: step outside it and the performer stops existing.

What it buys, in the terms this site cares about, is plausibility. A stereo recording of a person is a picture that happens to have depth; a volumetric one responds correctly when the viewer moves, which is the sensorimotor contingency place illusion is built from. That difference is worth far more than resolution.

Hearing, part three

The machinery, and why the damage is permanent

The two sections above treat the ear as something that already produces a direction and a pitch. It is worth saying how, because the mechanism explains both what it is astonishingly good at and what breaks it.

Sound arrives as a pressure wave in air and has to end up as nerve impulses in fluid, and air and fluid are wildly mismatched: pushed straight at a fluid surface, almost all of the energy reflects and only around a thousandth gets in. That is a loss of about 30 dB, and it would make ordinary speech inaudible. The middle ear exists to recover it. Three tiny bones — the ossicles, the smallest in the body — act as a mechanical transformer, concentrating the force from the large eardrum onto the much smaller oval window and adding a small lever advantage. It is an impedance matching device, doing the same job for sound that a transformer does for a circuit, and for exactly the same reason.

Inside the cochlea is the part that ought to be more famous. It is a coiled tube with a membrane running along it that is stiff and narrow at one end and floppy and wide at the other, so different frequencies peak at different places along its length. High frequencies at the entrance, low at the far end. That is a mechanical spectrum analyser, built out of geometry, running with no power and no latency — the ear does not compute a Fourier transform, it is one. Which is why pitch is called place coding, and why the frequency resolution of hearing is set by how sharply the membrane peaks rather than by anything neural.

Sitting on that membrane are the hair cells, around 16,000 of them, converting movement into nerve signals. There are two things to know about them and both matter on a show. The outer ones actively amplify quiet sounds, which is where the enormous dynamic range of hearing comes from. And humans do not regrow them. Noise damage is mechanical destruction of a finite, non-renewing population, which is why noise exposure is a dose over time rather than a level, and why hearing protection is not a comfort measure.

Seeing, part two

The eye as an optical instrument, and the two degrees that are sharp

Most people picture the lens doing the focusing. It does not do most of it. The cornea — the transparent front surface — provides roughly two thirds of the eye’s refractive power, because that is where the biggest change in refractive index happens, from air into tissue. The lens supplies the rest and, crucially, the adjustable part: it changes shape to focus near or far, and it stiffens with age, which is why reading glasses eventually happen to nearly everybody and has nothing to do with the retina.

Behind that sit two detector populations doing two different jobs. Around 6 million cones, which need reasonable light, give colour and fine detail, and are packed almost entirely into a tiny central pit called the fovea. Around 120 million rods, twenty times as many, work down to nearly single photons, give no colour at all, and are spread across the rest of the retina and absent from the middle of it.

The consequence is the single most under-appreciated fact about vision: the sharp part of your visual field is about two degrees wide. That is roughly a thumbnail at arm’s length. Everything else is peripheral and dramatically coarser — the impression of a wide, detailed world is a reconstruction assembled from several fixations a second, and you never see the gaps. It is also why where you point attention decides what is actually seen, and why peripheral content can be far lower resolution than anyone expects to get away with.

Then there is a hole. Where the optic nerve leaves there are no receptors at all, a blind spot several degrees across in each eye, and nobody perceives it. The visual system fills it from the surroundings without flagging that it did, which is the same machinery every illusion on this site borrows.

Seeing, part three

Dark adaptation, and the shift that changes what colour means

Two effects of the rod-and-cone split matter directly on a stage, and both are about low light.

Dark adaptation is slow and it happens twice. Cones adapt within about five minutes and then stop improving. Rods keep going for twenty to thirty minutes and end up vastly more sensitive. The practical figures: after a blackout an audience regains useful vision in a couple of minutes, but full dark adaptation takes half an hour and is destroyed in seconds by one bright cue. Which is why a followspot through a dark scene costs the entire house its adaptation, and why running lights are dim and red — red light barely stimulates rods, so it preserves the adaptation it lets you work by.

The Purkinje shift is the one that surprises lighting people. Rods peak around 500 nm and cones around 555 nm, so as light falls and vision hands over from cones to rods, the eye’s sensitivity peak moves toward blue. A red and a blue that matched in brightness at full will not match at 5% — the blue will look markedly brighter. Nothing changed about the fixtures. The receiver changed.

That gap between what a meter says and what an eye reports is the whole reason photometry has three regimes with different names: photopic at daylight levels where cones dominate, scotopic in near darkness where rods do, and mesopic in between, which is where almost every theatrical low-level cue actually sits and where neither curve is right.

Seeing, part four

One arcminute, and every viewing distance that follows from it

Acuity has a number, and it is the origin of a surprising number of rules that get quoted without it. Standard vision — 20/20 — resolves detail about one arcminute across: a sixtieth of a degree. At ten metres that is 2.9 mm.

Turn it round and you get the distance at which a pixel pitch stops being resolvable, which is about 3.4 metres per millimetre of pitch. A 3.9 mm wall is genuinely pixel-free at around 13 m. The familiar rule of thumb — pitch in millimetres equals minimum viewing distance in metres — gives 3.9 m, and the gap between those two numbers is the difference between invisible and acceptable. Both are legitimate targets; they are not the same target, and quoting one while meaning the other is how walls get specified badly.

Two caveats worth carrying. 20/20 is a norm rather than a ceiling — plenty of people resolve half an arcminute — so designing exactly at the threshold designs for the average eye and fails the sharpest ones in the room. And this figure only describes the fovea. Two degrees off-axis it is already much worse, which is why a wall that resolves badly straight on can be perfectly acceptable in peripheral vision.

Temperature

Receptors that report change, and cannot tell chilli from heat

There is no thermometer in your skin. There are separate warm and cold receptors, and they respond far more strongly to change and the rate of change than to the absolute level. Walk into a cool room and it feels cold; twenty minutes later it feels normal, and the room has not moved.

The receptors are ion channels, and this is where it gets useful. The main cold channel, TRPM8, is opened by low temperature — and also by menthol. The main heat channel, TRPV1, is opened by high temperature — and also by capsaicin, the compound in chilli. The receptor has no way of distinguishing the two triggers, so the brain receives an identical signal and reports the only thing it can: that is hot, that is cold.

Chilli is not metaphorically hot. It is opening the same channel that heat opens, and by the time the signal reaches you the distinction no longer exists. That is the cleanest demonstration anywhere that perception is downstream of transduction — which is the common-currency argument in one mouthful of food.

For a room this means comfort is mostly about gradient and airflow rather than a set point. Moving air reads as cold at a temperature still air does not, radiant heat from a lighting rig reads differently from warm air, and an audience arriving from outside is adapted to somewhere else entirely.

Touch

Why bass is felt, and where a tactile transducer lives

Skin has several mechanoreceptor types, each tuned to a different band of mechanical frequency — slow sustained pressure at one end, rapid vibration at the other.

The one worth knowing by name is the Pacinian corpuscle, which is most sensitive to vibration in roughly the 200 to 300 Hz region. That is squarely in the low end of music, and it is why bass is not simply heard — it is genuinely detected by a second sensory system with its own detectors, threshold and adaptation.

This is the mechanism a tactile transducer under a seat is using, and the reason a haptic vest can deliver a mix to somebody who cannot hear it. It is also why very low frequency stops being a pitch and becomes a sensation in the chest: below the hearing range there is nothing for the cochlea to do and plenty for the body to feel.

Touch, part two

Four detectors, and what a haptic device is actually addressing

The section above named the Pacinian corpuscle. There are four types worth separating, because a haptic effect that ignores which one it is talking to reads as a buzz rather than as a sensation.

Merkel cells respond to sustained pressure and fine spatial detail, and they adapt slowly — they are how you feel a texture or an edge held still. Meissner corpuscles handle low-frequency flutter around 5 to 50 Hz and adapt fast, which is how you detect something starting to slip out of your grip. Ruffini endings report skin stretch and, with them, the direction of a force. And Pacinian corpuscles sit deepest, are the most sensitive of all, and peak around 200 to 300 Hz — which is why they are the ones a tactile transducer is aimed at.

Two consequences for anybody building haptics. Spatial resolution varies enormously across the body: two points a couple of millimetres apart are distinguishable on a fingertip and need several centimetres on a back. A vest with an array of actuators has far less spatial vocabulary than the number of actuators suggests, and putting them closer together past a point buys nothing.

And the actuator has to match the receptor. An eccentric rotating mass spins a weight, so it cannot change amplitude and frequency independently and takes tens of milliseconds to spin up — fine for a phone buzz, useless for anything that has to land on a beat. A linear resonant actuator is faster and cleaner but only really works at its own resonance. A voice coil is a small loudspeaker without a cone, driven by an audio signal, and it is the only one of the three you can send a waveform to. Which is why haptic design on a show is largely an audio engineering problem, complete with a latency budget: touch and sound have to arrive inside roughly the same window as sight and sound, or the effect separates into two events.

Smell

The only sense that does not stop at the switchboard

Smell is the sense most likely to be dismissed as a novelty and the one with the most direct line to memory and emotion, and there is a structural reason for that rather than a poetic one.

Humans have roughly 400 different olfactory receptor types. Nothing like enough for the number of distinguishable odours, so the coding is combinatorial: a molecule activates a pattern across many receptors, and the identity is the pattern. It is the same trick colour vision plays with three cone types, with a far larger alphabet — which is why smells blend into new smells rather than being heard as chords the way sounds are.

The structural fact is the routing. Every other sense passes through the thalamus first, which is the brain’s relay and, loosely, its switchboard. Olfaction does not. It reaches the olfactory bulb and from there goes more or less directly to the amygdala and hippocampus — emotion and memory. That is not a metaphor about smell being evocative; it is a wiring diagram, and it is the best available explanation for why a smell can return a memory whole in a way a photograph does not.

Then the part that makes it hard to use. Olfactory adaptation is fast — a constant odour fades from awareness within a few minutes, sometimes less — so a scent that fills a room is gone from the audience’s experience long before it is gone from the room. And there is no off switch: a scent released into a space stays in the air handling, on the soft furnishings and in the next audience’s experience of the first scene. Use once, at a threshold, at the moment it should land, and plan how the room is cleared before you plan the effect.

The rest of them

Briefly, and they matter more than their airtime