Skip to content

Stream Audio

Neon +2.8.31 +1.7

Using the receive_audio_frame method, you can receive audio frames, which you can use it to listen live, record, or perform real-time analysis like speech-to-text or sound analysis.

The data returned is an instance of AudioFrame.

AudioFrame(
    av_frame=<av.AudioFrame pts=None, 1024 samples at 8000Hz, mono, fltp at 0x1189e0ac0>,
    timestamp_unix_seconds=1758211800.8221593,
    resampler=<av.audio.resampler.AudioResampler object at 0x1189e04c0>
)
AudioFrame

AudioFrame

Bases: NamedTuple

An audio frame with timestamp information.

This class represents an audio frame from the audio stream with associated timestamp information. The Class inherits AudioFrame from py.av library.

Note

Audio in Neon is streamed as fltp mono 8K, this class takes the decoded packets as av.AudioFrames.

Methods:

Attributes:

av_frame instance-attribute 

av_frame: AudioFrame

The audio frame.

datetime property 

datetime: datetime

Get timestamp as a datetime object.

resampler instance-attribute 

resampler: AudioResampler

A reference to a shared AudioResampler instance.

timestamp_unix_ns property 

timestamp_unix_ns: int

Get timestamp in nanoseconds since Unix epoch.

timestamp_unix_seconds instance-attribute 

timestamp_unix_seconds: float

Timestamp in seconds since Unix epoch.

to_ndarray 

to_ndarray(*args: Any, **kwargs: Any) -> NDArray

Convert the audio frame to a NumPy array.

Source code in src/pupil_labs/realtime_api/streaming/audio.py 
58
59
60
def to_ndarray(self, *args: Any, **kwargs: Any) -> npt.NDArray:
    """Convert the audio frame to a NumPy array."""
    return self.av_frame.to_ndarray(*args, **kwargs)

to_resampled_ndarray 

to_resampled_ndarray(*args: Any, **kwargs: Any) -> Iterator[NDArray]

Convert the audio frame to a resampled s16 NumPy array

Source code in src/pupil_labs/realtime_api/streaming/audio.py 
62
63
64
65
def to_resampled_ndarray(self, *args: Any, **kwargs: Any) -> Iterator[npt.NDArray]:
    """Convert the audio frame to a resampled s16 NumPy array"""
    for frame in self.resampler.resample(self.av_frame):
        yield frame.to_ndarray(*args, **kwargs)

By default, audio is streamed in mono using the AAC codec. The stream is downsampled from the original 48 kHz source to a sampling rate of 8 kHz to save bandwidth, and uses a 32-bit floating-point planar (fltp) format.

The audio stream does not have it's own RTSP stream but is multiplexed with video, so we create a virtual sensor component using the Scene Camera stream.

Working with Audio Data

You can easily receive audio frames and convert them to NumPy arrays using the to_ndarray method and feed these to any audio library of your choice like librosa for analysis.

Check the whole example code here
stream_audio.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
from pupil_labs.realtime_api.simple import discover_one_device

# Look for devices. Returns as soon as it has found the first device.
print("Looking for the next best device...")
device = discover_one_device(max_search_duration_seconds=10)
if device is None:
    print("No device found.")
    raise SystemExit(-1)
print(f"Found device: {device}")
device.streaming_start("audio")  # optional, if not called, stream is started on-demand
try:
    while True:
        audio_frame = device.receive_audio_frame(timeout_seconds=5)
        if audio_frame:
            print(audio_frame)
except KeyboardInterrupt:
    pass
finally:
    print("Stopping...")
    # device.streaming_stop()  # optional, if not called, stream is stopped on close
    device.close()  # explicitly stop auto-update

Playing Audio

Audio Playback in realtime can be tricky, here we use SoundDevice. This library digest NumPy arrays and allows to play them back quickly. The only caveat is that it does not accept 32 bit planar audio format, thus, we have to resample it.

For commodity, we included a PyAv AudioResampler object to the AudioFrame class, it lazy loads, and calling to_resampled_ndarray will convert convert the av.AudioFrame to a NumPy array in signed 16-bit integer format, which is supported by SoundDevice for playback.

To simplify development and ensure audio does not block your application, the examples include an AudioPlayer class. It handles audio buffering and playback in a background thread, using a circular buffer to guarantee smooth playback without glitches or silence.

AudioPlayer

AudioPlayer 

AudioPlayer(samplerate: int, channels: int, dtype: str = 'int16')

Bases: Thread

A threaded, low-latency audio player using a shared RingBuffer.

Methods:

  • add_data –

    Directly write data to the shared RingBuffer.

  • close –

    Signal the thread to stop and clean up resources.

  • get_buffer_size –

    Get the current number of samples in the buffer for debugging.

  • run –

    Run the main entrypoint for the thread.

Source code in examples/audio_player.py 
120
121
122
123
124
125
126
127
128
129
130
131
132
def __init__(self, samplerate: int, channels: int, dtype: str = "int16"):
    super().__init__(daemon=True)
    self.samplerate = samplerate
    self.channels = channels
    self.dtype = dtype

    self._stop_event = threading.Event()
    self._buffer = RingBuffer(
        capacity=1024,
        dtype=np.int16,
        channels=channels,
    )
    self.stream = None

add_data 

add_data(data: NDArray[int16]) -> None

Directly write data to the shared RingBuffer.

Source code in examples/audio_player.py 
161
162
163
def add_data(self, data: npt.NDArray[np.int16]) -> None:
    """Directly write data to the shared RingBuffer."""
    self._buffer.write(data)

close 

close() -> None

Signal the thread to stop and clean up resources.

Source code in examples/audio_player.py 
169
170
171
172
173
174
def close(self) -> None:
    """Signal the thread to stop and clean up resources."""
    logging.debug("Closing audio player...")
    self._stop_event.set()
    self.join()  # Wait for the thread to finish
    logging.info("Audio player closed.")

get_buffer_size 

get_buffer_size() -> int

Get the current number of samples in the buffer for debugging.

Source code in examples/audio_player.py 
165
166
167
def get_buffer_size(self) -> int:
    """Get the current number of samples in the buffer for debugging."""
    return self._buffer.size

run 

run() -> None

Run the main entrypoint for the thread.

Source code in examples/audio_player.py 
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
def run(self) -> None:
    """Run the main entrypoint for the thread."""
    try:
        self.stream = sd.OutputStream(
            samplerate=self.samplerate,
            channels=self.channels,
            dtype=self.dtype,
            callback=self._callback,
            blocksize=0,  # Let the device choose the optimal size for low latency
            latency="low",
        )
        with self.stream:
            logging.debug("Audio stream started.")
            self._stop_event.wait()  # Wait until the close() method is called
    except Exception:
        logging.exception("Error in audio thread.")
    finally:
        logging.debug("Audio stream closed.")
Check the whole example code here
stream_audio_and_play.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
import os
import sys

# Allow for absloute imports, to import audio_player from the parent directory.
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))

from audio_player import AudioPlayer

from pupil_labs.realtime_api.simple import discover_one_device


def main():
    # Look for devices. Returns as soon as it has found the first device.
    print("Looking for the next best device...")
    device = discover_one_device(max_search_duration_seconds=10)
    if device is None:
        print("No device found.")
        raise SystemExit(-1)
    print(f"Found device: {device}")
    device.streaming_start(
        "audio"
    )  # optional, if not called, stream is started on-demand
    player: AudioPlayer | None = None
    try:
        while True:
            audio_frame = device.receive_audio_frame(timeout_seconds=5)
            if audio_frame:
                if not player:
                    # Initialize the player with the correct parameters from first frame
                    player = AudioPlayer(
                        samplerate=audio_frame.av_frame.sample_rate,
                        channels=audio_frame.av_frame.layout.nb_channels,
                        dtype="int16",
                    )
                    # Start the player process
                    player.start()
                    print(
                        f"Audio stream parameters: "
                        f"Sample Rate: {audio_frame.av_frame.sample_rate}, "
                        f"Channels: {audio_frame.av_frame.layout.nb_channels}, "
                        f"Layout: {audio_frame.av_frame.layout.name}"
                    )
                    print("Started audio playback.")
                else:
                    player.add_data(next(audio_frame.to_resampled_ndarray()).T)

    except KeyboardInterrupt:
        pass
    finally:
        print("Stopping...")
        # device.streaming_stop()  # optional, if not called, stream is stopped on close
        device.close()  # explicitly stop auto-update
        if player:
            player.close()


if __name__ == "__main__":
    main()

Playing Video and Audio

Here you can find an example that shows how to play both video with gaze overlay and audio using OpenCV and SoundDevice.

Check the whole example code here
stream_video_gaze_and_audio.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
import os
import sys

import cv2
import numpy as np

# Allow for absloute imports, to import audio_player from the parent directory.
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
from audio_player import AudioPlayer

# Workaround for https://github.com/opencv/opencv/issues/21952
cv2.imshow("cv/av bug", np.zeros(1))
cv2.destroyAllWindows()

from pupil_labs.realtime_api.simple import discover_one_device  # noqa: E402

# NOTE: Audio playback is done in a separate process with SoundDevice and a circular
# buffer to avoid blocking the main thread and ensure smooth playback.
# An AudioPlayer class is provided in the realtime_api package for this purpose.


def main():
    # Look for devices. Returns as soon as it has found the first device.
    print("Looking for the next best device...")
    device = discover_one_device(max_search_duration_seconds=10)
    if device is None:
        print("No device found.")
        raise SystemExit(-1)

    print(f"Connecting to {device}...")
    player = AudioPlayer(samplerate=8000, channels=1, dtype="int16")
    try:
        player.start()
        while True:
            matched = device.receive_matched_scene_video_frame_and_audio(
                timeout_seconds=5
            )
            if matched is None:
                continue

            frame, audio, gaze = matched

            # We add all audio frames to the player's queue
            for audio_frame in audio:
                player.add_data(next(audio_frame.to_resampled_ndarray()).T)

            buffer_fill_ms = 1000 * player.get_buffer_size() / player.samplerate

            # We display the number of audio frames received with the video frame
            time_diff_ms = [
                frame.timestamp_unix_seconds - af.timestamp_unix_seconds for af in audio
            ]
            if audio:
                cv2.putText(
                    frame.bgr_pixels,
                    (
                        f"Audio frames: {len(audio)} / "
                        f"Buffer: {buffer_fill_ms:.0f} ms / "
                        f"Mean diff. audio-scene: {np.mean(time_diff_ms) * 1000:.0f} ms"
                    ),
                    (10, 30),
                    cv2.FONT_HERSHEY_SIMPLEX,
                    1,
                    (0, 255, 0),
                    2,
                )
            else:
                cv2.putText(
                    frame.bgr_pixels,
                    f"No audio / Buffer: {buffer_fill_ms:.0f} ms",
                    (10, 30),
                    cv2.FONT_HERSHEY_SIMPLEX,
                    1,
                    (0, 0, 255),
                    2,
                )
            if gaze:
                cv2.circle(
                    frame.bgr_pixels,
                    (int(gaze.x), int(gaze.y)),
                    radius=80,
                    color=(0, 0, 255),
                    thickness=15,
                )
            cv2.imshow("Scene camera and audio", frame.bgr_pixels)
            if cv2.waitKey(1) & 0xFF == 27:
                break
    except KeyboardInterrupt:
        pass
    finally:
        print("Stopping...")
        player.close()
        device.close()  # explicitly stop auto-update


if __name__ == "__main__":
    main()

Bonus

On the Async API examples you can also find how to use the audio and plot the spectrum using librosa library.