Human listener
Keep the message
Speech remains recognizable and understandable within a strict amplitude budget.
01 · The idea
Make transcription expensive to trust, not expensive to attack
NoWhisper turns a slow, sample-specific waveform attack into a reusable generator. It learns the attack once, then produces a bounded perturbation for unseen speech without per-clip optimization.
Whisper ASR
Lose the transcript
Substitutions and insertions push the decoded text away from the original speech.
- Course
- Deep Learning & Applied AI
- Data
- LibriSpeech · 16 kHz mono
- Stack
- PyTorch · Transformers · Gradio
- Attack
- Untargeted · waveform-level
02 · Method
From iterative gradients to one forward pass
Both paths optimize the same idea: increase Whisper’s reference loss while keeping the waveform bounded.
A
White-box reference
Optimize each clip
Audio→Adam→Project δ→Attack
Gradients pass through Whisper’s differentiable frontend for many update steps.
StrongSlow per waveform
B
Amortized attack
Generate for every clip
Audio→Gθ→Bound δ→Attack
The trained network predicts a complete perturbation in a single evaluation.
ReusableOne forward pass
Technical note · bounded attack equations
x_adv = clip(x + Π[-ε, ε](δ), -1, 1)
x_gen = clip(x + ε tanh(Gθ(x)), -1, 1)
The projection and tanh parameterization enforce the same L∞ amplitude budget in both pipelines.
03 · Model
A compact residual U-Net for raw waveforms
Skip connections preserve local speech structure while dilated residual blocks expand temporal context.
Encoder
8 ch
16 ch
32 ch
64 ch
Bottleneck
128 ch
wide context
Decoder
64 ch
32 ch
16 ch
8 ch
Base8 ch
Depth4
Kernel15
Blocks2 / stage
The surprising result
The strongest generator skipped teacher imitation and attacked Whisper’s reference loss directly.
Architecture and training objective
Blocks: Conv1D, GroupNorm, SiLU, dilation, channel projection, and residual paths.
Objective: adversarial reference loss plus waveform regularizers and optional waveform or spectral teacher losses.
J(θ) = -w_adv L_ref(x_gen, y) + R_gen + optional teacher losses
04 · Evidence
Trained on Small. Effective on larger Whisper models.
The same ε = 0.0075 generator was evaluated on 100 held-out clips per model without retraining.
Full transfer and perturbation-budget results
| Model | WER clean → adversarial | CER clean → adversarial | Success | SNR |
|---|---|---|---|---|
| Small | 0.047 → 3.825 | 0.020 → 3.028 | 99% | 20.79 dB |
| Medium | 0.033 → 2.242 | 0.014 → 1.860 | 98% | 20.79 dB |
| Large-v3 | 0.023 → 2.817 | 0.009 → 2.248 | 96% | 20.79 dB |
At ε = 0.005 the best 20-epoch run reached 2.165 WER at 23.37 dB. Strong masking raised SNR to 23.64 dB but reduced adversarial WER to 0.064.
05 · Run locally
The frontend and a ready checkpoint ship with the repository
Clone, install the matching PyTorch group, and launch the Gradio interface. No generator training is required.
git clone https://github.com/Reewd/NoWhisper.git
cd NoWhisper
uv sync --group cpu
uv run --group cpu nowhisper-ui
Checkpoint included
models/generators/resunet1d/latest.pt
Loaded lazily when generator mode is first used.
- Python 3.13 is required.
- For NVIDIA GPUs, replace
cpuwithcu126orcu130. - Whisper Small weights may download on first use.
NoWhisper
local · ready
Input audioUpload or microphone
LanguageAuto-detect
Clean transcription
Editable reference text appears here before the attack.
Waveform + perturbation
Delta spectrogram
WER3.825
SNR20.79 dB
StatusSuccess
06 · Limits & resources
A strong proof of concept with an audible trade-off
Not imperceptible
At roughly 20–23 dB SNR, the speech is understandable but the perturbation can still be audible.
Masking can collapse the attack
The strongest reported masking preserved quality but barely moved WER.
Open evaluation space
Other ASR families, languages, and over-the-air playback remain future tests.
Future directions
Learned perceptual objectives, multilingual evaluation, physical playback tests, and adversarial training are the clearest next steps.