Making a state audible:
closed-loop neurofeedback with NeuroEcho

1.What is measured, and why with light

A person's state is not an impression: it is written into physiological quantities that the autonomic nervous system regulates continuously. When the body mobilises, the heart speeds up, breathing becomes faster and less regular, the facial muscles contract, the gaze moves more. When it recovers, the heart rate slows down and oscillates with breathing, the breath lengthens and steadies, the face relaxes. The brain, for its part, shifts its energy between rhythms: alpha dominates relaxed wakefulness with the eyes closed, and the beta to theta ratio follows attentional engagement.

Biofeedback rests on an established principle: a physiological quantity made perceptible in real time becomes learnable. The body does not control its heart rate voluntarily, but it learns to return to the state that moves a perceptual feedback in the desired direction. The quality of this learning depends on three conditions: the feedback is continuous, it is contingent (it responds to the state and to nothing else), and it is non-intrusive, so that it does not mobilise the very attention it seeks to calm.

NeuroEcho meets these three conditions through two choices. The feedback is musical: music that opens up is received with the eyes closed, without reading numbers, and the ear perceives a change of timbre over durations where the eye only sees a noisy curve. The measurement is optical: an ordinary camera measures pulse, respiration, eyes and face without any sensor on the skin. A conventional cardiac biofeedback system requires an ear clip or a chest strap; NeuroEcho only needs a phone placed in front of the user. The optional EEG headband extends the measurement to the electrical activity of the cortex.

2.The optical measurement chain

2.1 Capture and clock

The camera is opened at 1280 × 720 and 30 frames per second, with exposure and white balance locked whenever the device allows it: an automatic exposure that corrects the image at every heartbeat erases the very signal being sought. Each frame is timestamped by the sensor clock (requestVideoFrameCallback) and not by the program clock: a frame processed late keeps its true capture time. The physiological signals are then resampled onto a regular grid, a prerequisite for any correct spectral analysis.

2.2 The face in 478 points

An on-device neural network (MediaPipe Face Landmarker) reconstructs the facial surface at every frame as 478 three-dimensional points, ten of them for the two irises, and estimates 52 expression coefficients modelled on the action units of Ekman and Friesen's FACS. Inference runs on the phone's graphics processor; no image leaves the device. The landmarks serve three purposes: delimiting the skin regions for the pulse, measuring head and gaze movements, and reading expressions.

2.3 Feature blocks

The slow features (pulse, respiration, blinks, gaze, tension) are computed over sliding windows and published once per second, as timestamped blocks that each carry their own quality index. The fast features of dynamic mode (§ 13) are published every 100 ms. This separation of time scales is essential: a pulse is estimated over twelve seconds, a micro-expression lasts a tenth of a second.

3.The pulse from skin colour

At each systole, the volume of blood in the dermal capillaries increases. Haemoglobin strongly absorbs green light: the skin returns slightly less green light at each heartbeat. The relative variation is in the order of 0.1 to 1 %, invisible to the eye but measurable by averaging thousands of pixels. This is the principle of remote photoplethysmography, demonstrated by Verkruysse in 2008 with a simple camera and ambient light.

3.1 Three skin regions

The forehead and both cheeks are delimited by polygons of facial landmarks, shrunk by 12 to 15 % towards their centre to exclude the eyebrows, the hair and the sides of the nose. These three regions are the richest in superficial capillaries and the most stable against expressions. For each frame, the mean of the red, green and blue channels is computed over all pixels of the three regions.

3.2 Plane orthogonal to skin (POS)

The raw signal of each channel mixes three components: the pulse, the variation of illumination intensity and the specular reflection caused by motion. The POS method of Wang and colleagues (2017) separates them with a geometric argument. After temporal normalisation, intensity variations move along the skin-tone direction; the signal is therefore projected onto a plane orthogonal to that direction, where only the blood-related hue variations remain:

3.3 From signal to heart rate

The irregularly timestamped colour samples are resampled at 30 Hz over a 12-second window. The signal h goes through a 0.7 to 3.5 Hz band-pass filter, i.e. 42 to 210 beats per minute, then a 4096-point periodogram. The peak frequency gives the pulse; the ratio between the energy around the peak and the energy in the rest of the band gives its sharpness:

The variation of the pulse from one window to the next also provides a cardiac instability index, which feeds the stability index (§ 8).

4.Respiration, eyes and face

4.1 Respiration without a sensor

Breathing moves the face: the rib cage lifts the shoulders, neck and head by a few tenths of a millimetre at each inhalation. Breathing also modulates the pulse and the cutaneous blood volume. NeuroEcho uses two candidate signals over a 32-second window: the vertical position of the nose, divided by the interocular distance to cancel out the distance to the camera, and the green channel of the skin regions. Both are resampled at 10 Hz and filtered between 0.1 and 0.5 Hz, i.e. 6 to 30 breaths per minute. The candidate kept is the one whose spectrum is the most concentrated:

Regularity is not a by-product: it is a physiological quantity in its own right. A slow, regular breath drives respiratory sinus arrhythmia, the oscillation of heart rate in phase with breathing, which reflects vagal nerve activity.

4.2 The eyes

The closure of each eye is read from the expression coefficients. A blink is a closure that crosses 0.55 and then falls back below 0.40 (hysteresis) within 500 ms. Over a 60-second window, this yields the blink rate, the median blink duration, and PERCLOS, the proportion of time during which closure exceeds 0.80. PERCLOS has been the reference drowsiness indicator since the work of Dinges and Grace (1998); blink duration lengthens with fatigue even before blink rate changes.

4.3 The gaze

The position of the irises, relative to the corners of the eyes, gives the gaze direction. Its instability is the norm of the horizontal and vertical standard deviations over 5 seconds. Gaze loss is the proportion of frames, over 10 seconds, in which the face is not detected or is turned by more than 0.35 radian. A gaze that holds on the central point is the behavioural signature of sustained attention; a wandering gaze, that of dispersion.

4.4 Facial tension

Tension is the mean of eight expression coefficients that express contraction: brow lowering, eye squinting, lip pressing and nose wrinkling, left and right. These action units (AU4, AU6-7, AU9, AU24 in FACS terms) accompany strained concentration, effort and discomfort. Their release is one of the first visible signs of relaxation.

4.5 Eyes closed

Several goals are practised with the eyes closed. The eye-dependent measures (blinks, PERCLOS, gaze) are then removed from the computation rather than set to zero: attention and fatigue are no longer computed, and the remaining indices renormalise their weights over the available quantities (§ 8).

5.Signal quality

An optical measurement is never good by default. Light, framing, motion and camera frame rate vary from one second to the next. NeuroEcho therefore computes, for each block, a global signal quality index (SQI, from 0 to 100) from six components:

Without a clear face (component below 0.3), the SQI is capped at 10. Each feature also carries its own quality: the pulse requires 55, respiration 40, the others 50. Below an SQI of 45, the state is frozen at its last reliable value and the music stops adapting, without stopping. A poor signal is never interpreted: this is the condition of feedback contingency. The user then hears stable music, and the screen shows the correction to make (light, framing, stillness).

6.The electroencephalographic path

The Muse 2 headband carries four dry electrodes, two frontal (AF7, AF8) and two temporal (TP9, TP10), referenced to the middle of the forehead, sampled at 256 Hz and transmitted over Bluetooth Low Energy in packets of twelve 12-bit samples. NeuroEcho talks to the headband directly from the browser, decodes the packets and checks their sequence number: every loss is counted, and gaps are never interpolated in the analysis.

6.1 Filtering and segmentation

Each channel goes through a 1 to 45 Hz band-pass filter and a 50 Hz notch (60 Hz on American grids). The analysis covers a 4-second window split into 1-second segments. A segment is rejected as an artefact if it contains:

The power spectrum is the average of the spectra of the clean segments (Welch's method). The quality of a channel is the proportion of clean segments, weighted by contact quality; below 45 out of 100, the channel is set aside. A detached electrode never blocks the rest: measurement continues on the valid channels.

6.2 The quantities

Beta and gamma powers, the most exposed to muscle activity, have their quality reduced in proportion to the muscle tension index. The "Alpha relaxation" goal follows relative alpha power, "Concentration" the beta to theta ratio, a quantity studied for thirty years in attention neurofeedback.

7.The personal baseline

A pulse of 72 beats per minute says nothing on its own: it is low for one person, high for another, and varies from day to day in the same person. NeuroEcho therefore never compares a measurement with a population norm. Each session starts with a 60-second calibration, whose first 12 seconds are discarded (the time for the person to settle and for the measurement windows to fill). Over the blocks of sufficient quality, at least 20, the median and the median absolute deviation (MAD) of each quantity are computed, two robust estimators insensitive to outliers:

A floor MAD_min is set for each quantity (2 bpm for the pulse, 1 breath per minute, 2 blinks per minute, 0.015 for relative alpha). Without this floor, a person who is very still during calibration produces an almost zero MAD, and the slightest later variation passes for an extreme deviation. The floor sets the smallest physiologically meaningful variation.

8.From measurements to state indices

The normalised quantities combine into six indices, each in [0, 1], 0.5 denoting the baseline state. All combinations are weighted means that ignore unavailable quantities and renormalise the remaining weights; the share of weights actually covered is kept as the index coverage.

The indices are recomputed every 3 seconds and smoothed by an exponential moving average with a 9-second time constant: fast enough to follow a change of state, slow enough to ignore an isolated gesture. When the EEG headband and the camera work together, a fusion engine combines the sources index by index, each source weighted by its confidence, computed from its SQI and the share of available quantities. A source whose signal becomes insufficient immediately stops contributing: a detached headband never prolongs an old value while the camera remains reliable.

9.From state to music

9.1 Progress towards the goal

Each goal defines a combination of indices, its target. Progress TP is its weighted mean, in [0, 1]:

9.2 The reward

Progress is only used if its confidence, the product of state confidence and target coverage, reaches 0.4. It is then smoothed into a reward r by a 20-second exponential moving average, and an audio decision is taken every 20 seconds. This slow rhythm is deliberate: feedback that reacts to every fluctuation teaches the body to chase noise; feedback that follows the trend teaches it to return to a state.

9.3 The sound mapping

The reward goes through an S-curve, then sets five mixing parameters:

At low reward, the music is dark, narrow and grainy; at high reward, it is bright, wide and harmonic. The melody follows a hysteresis: it appears when the reward stays above 0.55 for 15 seconds, and disappears below 0.50. It rewards a sustained state, never a peak. Its volume then grows from 0.10 to 0.55 with the reward.

9.4 One loop, two time constants

The delay between a decision and its application to the sound is below 10 milliseconds; the delay between a change of state and its full musical translation is in the order of half a minute. The first constant guarantees contingency; the second tunes learning to the time scale of autonomic regulation, the scale at which the baroreflex, respiration and vagal tone reorganise.

10.The sound material

The feedback music is composed in four superimposed layers, in D major, at 60 beats per minute, on 64-second loops: a low sustained base, harmonics that enrich the spectrum, a melody, and a grainy texture that carries discomfort. The four layers are played in sample-accurate sync, so that none of them drifts over a thirty-minute session. They are delivered losslessly (FLAC): AAC and MP3 encoding add silence at the start and end of a file, which breaks the continuity of a loop.

The 60 beats per minute tempo places the musical pulse at 1 Hz, in the range of the resting heart rate; the nervous system tends to synchronise with a perceived pulse, and a pulse close to rest invites the body to settle there. Stereo width is set by mid-side processing, space by a convolution reverb, and a final limiter protects the ear. A second bus, independent of the feedback, plays the NeuroSound® sequences: this is the subject of the next section.

11.Learning what works for each person

11.1 Judging a sound programme

During a session, NeuroEcho tries several NeuroSound® sound programmes. Each is judged by the slope of progress while it plays, estimated with the Theil-Sen estimator, the median of the slopes between all pairs of points: robust, it ignores up to 29 % of outliers. But a calm session calms down on its own over time; a programme played at the end of a session inherits a slope it did not produce. The slope is therefore corrected by a drift prior, the mean spontaneous slope measured over the person's previous sessions for the same goal:

11.2 A bandit across sessions

Choosing the order of programmes is an exploration and exploitation problem: play the one that has worked best, or try a less known one whose potential remains to be measured. NeuroEcho solves it with an upper confidence bound bandit (UCB1):

Only segments of at least one minute, measured over at least 60 % of their duration, feed the learning. Over the sessions, the "What works for you" section reveals the person's musical signature.

12.Guided breathing

The "Calm breathing" goal displays a guide at 7 breaths per minute, inhalation over 40 % of the cycle (3.4 s), exhalation over 60 % (5.1 s). This rhythm relies on the resonance frequency of the cardiovascular system described by Vaschillo and Lehrer: around 0.1 Hz, i.e. 4.5 to 7 breaths per minute, the heart-rate oscillation induced by breathing comes into phase with that of the baroreflex, and its amplitude becomes maximal. The lengthened exhalation reinforces the vagal part of the cycle.

The guide is set at 7 breaths per minute, i.e. 0.117 Hz, and not at 6: the respiratory analysis band starts at 0.1 Hz, and a rhythm placed exactly on its edge is measured less accurately. Half of this goal's progress is carried by the respiration index: the music opens when the measured breath slows down and steadies relative to the baseline. The guide proposes, the measurement verifies.

13.Dynamic mode and micro-interruptions

Dynamic mode reverses the logic of the goal. The music no longer follows a progress value: it stays stable, and brief interruptions highlight the moments when the state changes quickly. An interruption is a salient event: the nervous system responds with an orienting reflex, described by Sokolov, which suspends ongoing activity and updates the model of the environment. Placed at the moment of a transition, the interruption signals to the body its own change, without prescribing a direction.

13.1 Detecting a transition

Each fast feature is compared with its own dynamics over the last 60 seconds, through sliding median and MAD. The deviations combine into a quadratic score:

The camera does not see the brain; it sees the fast behavioural markers of a change of state: the onset of a micro-expression, a gaze saccade, a sigh (ample variation of face position), the start of a blink. On the EEG side, a blink, a muscle contraction or a change in the set of valid channels blocks triggering: these events produce spectacular deviations without saying anything about cortical activity.

13.2 The interruption

The feedback music drops to zero in 15 ms, stays silent for 180 ms, and comes back in 15 ms. The ramps remove the click of a hard cut; 180 ms is long enough to be perceived unambiguously, short enough not to break the musical flow. The interruption applies to the feedback, after the reverb, and spares the NeuroSound® bus by default. A 4-second refractory period, a cap of 8 interruptions per minute and 30 seconds of initial learning prevent any burst.

13.3 The control

The "Control" setting produces the same number of interruptions, subject to the same rules, but placed at random (Poisson process, 4 per minute). It makes every user a potential experimental subject: comparing a series of dynamic sessions with a series of control sessions isolates the effect of the timing of the interruption from that of the interruption itself (§ 14, H4). On 10 minutes of synthetic EEG containing 12 transitions, the detector finds 10 to 11 of them with 0 to 0.2 false alarms per minute.

14.Hypotheses and research programme

NeuroEcho measures, on a phone, quantities that psychophysiology usually observes in the laboratory with contact sensors: pulse, respiration, eye tracking, expression, brain rhythms. Each session produces a timestamped time series, associated with a goal, sound programmes and the exact chronology of audio decisions. We set out seven hypotheses that can be tested with this instrument.

The meaning of a closed loop

Adaptive music is not a playlist that changes with mood: it is a mirror. By making audible, second by second, a quantity the body regulates without being aware of it, NeuroEcho gives this regulation an object of perception, and therefore a handle. Sonification turns a measurement into an experience; the closed loop turns the experience into learning. The next step is to let the music itself evolve with the person: to compose, from their signature (H5) and the drift of their baseline (H7), a sound material that belongs to them, and to measure whether music tuned in this way speeds up learning compared with shared music. NeuroEcho provides the stimulus, the measurement and the record of the experience at once.

References

1. Verkruysse, W., Svaasand, L. O., Nelson, J. S. (2008). Remote plethysmographic imaging using ambient light. Optics Express, 16(26), 21434-21445.
2. Poh, M.-Z., McDuff, D. J., Picard, R. W. (2010). Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Optics Express, 18(10), 10762-10774.
3. Wang, W., den Brinker, A. C., Stuijk, S., de Haan, G. (2017). Algorithmic principles of remote PPG. IEEE Transactions on Biomedical Engineering, 64(7), 1479-1491. POS method.
4. Kartynnik, Y., Ablavatski, A., Grishchenko, I., Grundmann, M. (2019). Real-time facial surface geometry from monocular video on mobile GPUs. CVPR Workshops.
5. Lugaresi, C., et al. (2019). MediaPipe: A framework for building perception pipelines. arXiv:1906.08172.
6. Ekman, P., Friesen, W. V. (1978). Facial Action Coding System. Palo Alto: Consulting Psychologists Press.
7. Dinges, D. F., Grace, R. (1998). PERCLOS: A valid psychophysiological measure of alertness as assessed by psychomotor vigilance. Federal Highway Administration, FHWA-MCRT-98-006.
8. Vaschillo, E., Lehrer, P., Rishe, N., Konstantinov, M. (2002). Heart rate variability biofeedback as a method for assessing baroreflex function. Applied Psychophysiology and Biofeedback, 27(1), 1-27.
9. Lehrer, P. M., Gevirtz, R. (2014). Heart rate variability biofeedback: how and why does it work? Frontiers in Psychology, 5:756.
10. Welch, P. D. (1967). The use of fast Fourier transform for the estimation of power spectra. IEEE Transactions on Audio and Electroacoustics, 15(2), 70-73.
11. Klimesch, W. (1999). EEG alpha and theta oscillations reflect cognitive and memory performance. Brain Research Reviews, 29(2-3), 169-195.
12. Davidson, R. J. (1992). Anterior cerebral asymmetry and the nature of emotion. Brain and Cognition, 20(1), 125-151.
13. Arns, M., Conners, C. K., Kraemer, H. C. (2013). A decade of EEG theta/beta ratio research in ADHD: a meta-analysis. Journal of Attention Disorders, 17(5), 374-383.
14. Krigolson, O. E., Williams, C. C., Norton, A., Hassall, C. D., Colino, F. L. (2017). Choosing MUSE: validation of a low-cost, portable EEG system for ERP research. Frontiers in Neuroscience, 11:109.
15. Sitaram, R., et al. (2017). Closed-loop brain training: the science of neurofeedback. Nature Reviews Neuroscience, 18(2), 86-100.
16. Hampel, F. R. (1974). The influence curve and its role in robust estimation. Journal of the American Statistical Association, 69(346), 383-393.
17. Theil, H. (1950). A rank-invariant method of linear and polynomial regression analysis. Indagationes Mathematicae, 12. Sen, P. K. (1968). Estimates of the regression coefficient based on Kendall's tau. Journal of the American Statistical Association, 63(324), 1379-1389.
18. Auer, P., Cesa-Bianchi, N., Fischer, P. (2002). Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47, 235-256. UCB1 algorithm.
19. Sokolov, E. N. (1963). Perception and the Conditioned Reflex. Oxford: Pergamon Press. Orienting reflex.
20. Hermann, T., Hunt, A., Neuhoff, J. G. (eds.) (2011). The Sonification Handbook. Berlin: Logos.