whereismyai
Whisper Diarization
Whisper Diarization is an audio processing pipeline that combines OpenAI Whisper with speaker diarization to identify speakers.
This project combines Whisper ASR capabilities with Voice Activity Detection and Speaker Embedding to identify the speaker for each sentence in the transcription generated by Whisper. First, the vocals are extracted from the audio to increase speaker embedding accuracy. Then, the transcription is generated using Whisper, and timestamps are corrected and aligned using ctc-forced-aligner. The audio is passed into MarbleNet for VAD and segmentation, and TitaNet extracts speaker embeddings. Finally, the results are associated with timestamps and realigned using punctuation models. It requires Python 3.10 or 3.9, FFMPEG, and Cython as prerequisites. It uses OpenAI Whisper, Faster Whisper, Nvidia NeMo, and Facebook Demucs.
Whisper Diarization at a glance
Key features
- Extract vocals from audio to increase accuracy
- Generate transcriptions using OpenAI Whisper
- Correct and align timestamps using ctc-forced-aligner
- Perform VAD and segmentation using MarbleNet
- Extract speaker embeddings using TitaNet
- Realign results using punctuation models
Use it for
- Transcribe audio files with speaker identification
- Process audio with parallel model execution
- Isolate speakers in audio recordings
Alternatives to Whisper Diarization
- WhisperLive — It provides real-time speech-to-text using multiple backends like Faster Whisper and TensorRT.
- WhisperX — It enables 70x faster, batched transcription with high-accuracy alignment.
- whisper.cpp — It provides high-performance, offline speech recognition on diverse hardware.
- Whisper ASR Webservice — Provides a REST API for multiple ASR engines including WhisperX and Faster Whisper.
- WhisperLiveKit — It delivers ultra-low-latency streaming audio processing with multiple advanced backends.
- faster-whisper — Transcribes audio faster with lower memory usage than standard implementations.
- Whisper Timestamped — Generates accurate word-level timestamps without extra neural networks.
- browser-whisper — Transcribes audio and video locally with zero servers involved.
- MOSS Transcribe — Jointly performs speech transcription and speaker diarization in one pass.
- AssemblyAI — Industry-leading speech models deliver high accuracy and low latency for voice applications.
- Faster Whisper GUI — Provides a visual interface for faster-whisper, whisperX, and Demucs models.
- sherpa-onnx — Runs complex audio AI tasks offline on local devices and embedded hardware.
All Whisper Diarization alternatives