Sonar: Real-Time On-Device Speech Intelligence & Diarization
Engineered an edge speech intelligence system running entirely on a Raspberry Pi 5. Combines USB audio ingestion, Silero VAD v5, dual-stream ECAPA2 diarization with AOSC adaptation, and faster-whisper INT8 ASR.
Hardware architecture showing USB ALSA capture, NumPy SIMD decimation, Silero VAD, and parallel CTranslate2 + ECAPA2 execution.
End-to-End Latency
2,433.5 ms
49.8% FasterSpeaker Accuracy
93.3%
+11.9%CPU Utilization
68% avg
Thermals <62°CRAM Footprint
1.15 GB
-66.2%Sonar is an edge speech intelligence and speaker diarization engine deployed completely on-device on a Raspberry Pi 5 without cloud reliance. Captures 48kHz audio via ALSA, decimates 3:1 to 16kHz mono, and filters speech with Silero VAD v5. Operates a dual-stream architecture: Stream A extracts 512-D ECAPA2 speaker embeddings with exponential moving average profile adaptation, while Stream B transcribes live speech with faster-whisper small.en quantized to INT8 via CTranslate2. In verified hardware trials on Broadcom BCM2712, Sonar processed 49.2 seconds of conversational speech across 15 utterances with 2,433.5ms end-to-end latency and 93.3% speaker attribution accuracy.
Cloud-dependent speech diarization suffers from unacceptable network latency, privacy violations in confidential environments, and recurring API costs. Deploying dual-stream diarization and transcription on edge ARM hardware requires aggressive memory optimization and CPU parallelism.
End-to-End System Dataflow
Expandable Vector ArchitectureSystem Execution Pipeline
Hardware Ingestion & SIMD Decimation
Captures raw 48kHz audio directly from the USB microphone via Linux ALSA, applying 3:1 SIMD decimation down to 16kHz 16-bit PCM mono.
Silero VAD v5 Speech Gating
Evaluates 512-sample frame windows through Silero VAD v5, gating non-speech silence and passing active speech bursts to the inference queue.
Dual-Stream Diarization & Transcription
Stream A extracts 512-D ECAPA2 speaker embeddings with AOSC clustering, while Stream B transcribes speech tokens in parallel using faster-whisper small.en INT8.
SCDiar Alignment & Telemetry
Aligns word timestamp boundaries with active speaker clusters, publishing real-time attributed transcripts over WebSockets to the web dashboard.
Core Architectural Components Breakdown
ALSA Audio Ingest & Decimator
Audio Acquisition & ResamplingDirect ALSA hardware access with vectorized NumPy 3:1 decimation filter.
Silero VAD v5
Voice Activity DetectionLightweight deep neural network evaluating 512-sample frame probabilities with 300ms hangover.
SpeechBrain ECAPA2
Speaker Embedding ExtractionTime-delay neural network extracting 512-dimensional speaker acoustic identity vectors.
AOSC Profile Adaptation
Dynamic Speaker TrackingAdaptive Online Speaker Clustering with exponential moving average (EMA, alpha=0.15) updating centroid profiles.
faster-whisper small.en (INT8)
Streaming Automatic Speech RecognitionQuantized INT8 Transformer running on CTranslate2 utilizing 4 ARM Cortex-A76 cores.
SCDiar Token Aligner
Temporal Cross-Modal SynchronizationSynchronizes word start/end timestamps with active speaker cluster intervals.