The brain as a signal system
Transduce, encode, carry, decode, act — with a latency budget on every stage. That is the block diagram of a show system, and it is also the block diagram of a nervous system. Once you draw them next to each other, controlling a game with brain activity and giving somebody back a sense they lost stop being science fiction and become engineering, with honest limits.
The reframe
Every sense is a transducer, and they all output the same thing
A microphone turns pressure into voltage. A retina turns photons into trains of electrical impulses. A cochlea turns pressure into trains of electrical impulses. A mechanoreceptor in the skin turns deformation into trains of electrical impulses.
Notice what happened. Three completely different physics went in, and the same currency came out. The cortex never receives light, sound or pressure. It receives spikes — and spikes from an eye look, in themselves, much like spikes from an ear.
So what makes a signal a sight rather than a sound is largely where it arrives and what it is correlated with, not what produced it. That is the fact everything else on this page depends on. If the currency is common, then a signal from an unusual source, delivered consistently to a channel that still works, can come to mean what the missing sense used to mean.
Reading
What EEG actually measures, and what a brain-computer interface really does
Put electrodes on a scalp and you can measure voltage differences of a few tens of microvolts. Those come from the summed electrical activity of very large populations of neurons — millions at a time — smeared by the skull and scalp on the way out. EEG has superb time resolution and poor spatial resolution: you know when something happened much better than where.
That is the constraint that shapes every practical brain-computer interface. You cannot read a thought. What you can do is detect a small number of distinguishable, reliably reproducible patterns — and then build an interface out of that handful of choices. Three paradigms carry most of the working systems.
- SSVEP
- Steady-state visual evoked potential. Show several targets each flickering at a different rate. Attend to one, and the visual cortex response picks up that rate — detectable in a second or two with very little training. The highest information rate of the non-invasive methods, and it works because it is frequency detection, not mind reading.
- P300
- A characteristic response roughly 300 ms after a rare or meaningful stimulus in a stream. Flash rows and columns of a grid; the one the user is waiting for produces the response. This is how the classic EEG spellers work.
- Motor imagery
- Imagining a movement changes rhythms over the motor cortex, even with no movement. It requires real training on both sides — the user learns, and the classifier learns them — and it delivers a low rate. It is also the most intuitive to use once learned.
- Most "mind control" demos are muscles. A jaw clench, a blink, an eyebrow raise produce electrical signals orders of magnitude larger than cortical activity, and they sit right where a headset's electrodes are. If a demo responds instantly and reliably, suspect EMG before celebrating.
- A consumer headset is not clinical EEG. Fewer electrodes, dry contacts, worse placement, far more noise. It can be genuinely fun and it is not measuring what a research rig measures.
- "Attention" and "meditation" scores are proxies. They are derived band-power heuristics, not measurements of a mental state, and they should be treated as an interesting control input rather than as a fact about the person.
- Invasive is a different regime entirely. Electrode arrays placed in cortex — the research systems behind cursor control, robotic arm control and speech decoding — get orders of magnitude more signal, at the cost of neurosurgery. The gap between what is possible in a lab with an implant and what is possible with a headset at a trade show is enormous.
Writing
Giving a sense back, or delivering it through a different door
The reverse direction is the older and, so far, the more successful one.
The cochlear implant is the clearest case in all of medicine. When the hair cells of the inner ear are gone, the auditory nerve behind them is often intact. So the implant bypasses the broken transducer entirely: a processor splits incoming sound into bands and stimulates an electrode array threaded into the cochlea, placing each band where the cochlea would naturally have represented that frequency. Hundreds of thousands of people use them.
And the detail that matters most is the one that gets left out: it is not restored hearing. A healthy cochlea has thousands of hair cells; an implant has on the order of a couple of dozen electrodes, whose fields overlap. What arrives is a drastically reduced version of the signal. It works because the brain learns to read it — over months — and because speech turns out to be robust enough to survive that reduction. Music, which needs fine pitch resolution, fares much worse. That is a codec problem, and it is describable in exactly those words.
Retinal implants attempt the same trick for vision and have had a harder time, because vision needs far more channels. Existing systems deliver a coarse grid of light spots rather than an image. One of them, Argus II, is also the field's standing cautionary tale: the company stopped supporting the device, leaving implanted users with hardware nobody maintains. A sensory prosthesis is not a product you can discontinue like a phone.
Sensory substitution takes the other route: leave the damaged sense alone and deliver its information through a channel that still works. Paul Bach-y-Rita began this in the 1960s with a camera driving a grid of vibrating points on the back. Later systems moved the grid to the tongue — dense with nerve endings, wet, conductive, so tiny stimulating currents work well — where a camera image becomes a pattern of stimulation that trained users interpret as spatial layout. Others convert images to soundscapes, or colour to audible pitch, which is what lets a person with total colour blindness experience colour as tone.
None of these deliver the resolution of the sense they stand in for. What they deliver is information about the world through an unusual door — and the striking, repeatedly observed result is that with enough use, people stop experiencing the device and start experiencing the world. The stimulation on the tongue stops feeling like tingling on the tongue and starts feeling like something being over there.
The condition
Why it only works if the person can move
The finding that turned sensory substitution from a curiosity into a field is that it requires action. Passively receiving the stimulation produces a sensation on the skin, or on the tongue, and it stays that way. Being able to move the camera — to turn your head, to reach out and change what arrives — is what converts it into perception of an external world.
That makes sense in signal terms. A sense is a mapping between what you do and what comes back. Without the loop, there is nothing for the brain to correlate against, and no reason for it to infer that the signal is about anything out there.
It is also a warning about every "just feed it into the brain" pitch. The bandwidth is the easy part. The closed loop, the consistency of the mapping, and the months of use are the hard parts, and they are not optional.
What a channel count actually removes
A healthy cochlea has thousands of hair cells; an implant has a couple of dozen electrodes whose fields overlap. Drag the channel count and watch a spectrum collapse into what actually gets delivered.
The scale
What these channels actually carry
Rough figures, chosen to be comparable rather than precise. They are the reason the field is difficult and the reason the successes are impressive.
Bars are illustrative proportions on a compressed scale, not measurements. The point is the gap, not the pixels.
The synthesis
The same block diagram, twice
Set the two chains side by side and they are the same shape, stage for stage. That is not a metaphor — it is why the vocabulary transfers.
A show system
A nervous system
Why this belongs on a show-technology site
Three concrete reasons
Because the audience is the last device in the signal chain. Everything else here is about getting a signal accurately to a person; this is the specification of the person. The thresholds on the perception page are that device's data sheet, and the mechanisms here are why the data sheet reads the way it does.
Because accessibility is a substitution problem, and it is a design problem. Haptic vests that turn a mix into patterns on the body for deaf audience members, hearing loops built to IEC 60118-4, audio description, captioning, sign interpretation, seat-level tactile transducers: every one of those is the same idea — deliver the information through a channel the person has. Treating that as an engineering layer of the show, budgeted and designed, produces something far better than treating it as a compliance item bolted on in week eleven.
Because biometric input into shows is real and is usually oversold. Heart rate, skin conductance, breathing and EEG can all drive content, and all of them are low-bandwidth and noisy. Designed well — as a slow, aggregate influence on a system that looks good regardless — they add something genuine. Designed as a direct control path, they produce a show that depends on the least reliable signal in the building.
- Bandwidth is never the whole story. A cochlear implant carries a fraction of one percent of the channels an ear has and restores conversation, because speech is redundant and brains adapt. Ask what the signal has to carry before asking how much of it you can carry.
- Latency applies here too. A prosthesis or a haptic system with inconsistent delay is harder to learn than one that is slower but steady — the same lesson as the determinism page, in a different domain.
- Adaptation takes months, not minutes. Any demonstration that shows someone using a new sense competently within an afternoon is measuring something simpler than it appears to be.
- A device somebody depends on cannot be discontinued. The Argus II story is the one to remember before shipping anything a person will rely on.
Where to go next
This page is one half of a pair
The perception page is the thresholds — what a person can detect, and under what conditions. This one is the mechanism underneath those thresholds, and what happens when you interfere with it deliberately.
Everything else on this site is the chain in between: how a signal survives a wire, how it survives a network, how it survives the air, and how several of them agree with each other. It ends here, at the only receiver that matters.
This page is written for show technologists from public research and clinical literature, and it simplifies aggressively. If something here is out of date or overstated — particularly the numbers — open an issue. Corrections from people who do this work are worth more than any amount of rewriting from outside it.
- Can you really read a brain?
- How does a cochlear implant work?
- Can a lost sense be replaced?
Where to go next
These pages are a chain rather than a library — the next one along usually answers the question this one raised.
Everything on this page, in the index
Each of these has its own entry with sources, ports, gotchas and an open gap where a field is missing.