#Designing speech models: components, interfaces and architecture recipes

Study guide · 15 October 2026 HKT · primary-source survey and engineering synthesis

A useful speech architecture combines components that answer complementary questions: what acoustic structure is present, which person matters, and how the desired waveform should be reconstructed. Adding a stronger encoder does not help automatically. Its information must reach a compatible reconstruction or selection mechanism, under supervision that rewards the intended behavior.

This guide focuses on separation, enhancement and target-speaker extraction, with automatic speech recognition (ASR) as a downstream objective. It assumes basic familiarity with neural-network training, but introduces signal-processing terms as needed. “Effective” always means effective for a specified task, dataset, quality criterion and computational budget. Published benchmark results motivate candidates; they do not promise improvements after recombination.

#1. Begin with the task, then choose the information path

Enhancement usually preserves desired speech while reducing noise or reverberation. Separation returns multiple source estimates, often without identifying their speakers. Target-speaker extraction returns one specified person's speech, using a cue such as enrollment speech, a face track or a direction. The target may be the dry source or its reverberant image at a reference microphone; that choice changes the learning problem. The authors' target-speech-extraction tutorial provides a useful task-level introduction.

A separator can produce two excellent waveforms and still choose the wrong person. Conversely, a localizer can identify the right direction while a reconstruction module damages the words. Keep three capabilities distinct: localization, speaker association, and waveform reconstruction.

The common signal flow is:

flowchart LR
    A[Recorded audio] --> R[Waveform or spectral representation]
    R --> M[Temporal and spectral modeling]
    Q[Enrollment speech or video] --> C[Target cue encoder]
    C --> F[Conditioning or fusion]
    M --> F
    F --> H[Mask, filter or waveform decoder]
    H --> O[Target speech]
    O --> E[Listening and downstream ASR evaluation]

This diagram describes direct conditioning. A second architecture reconstructs anonymous candidates first and applies the cue only to select among them. The two designs distribute risk differently: direct conditioning can help recover a weak target, whereas later selection cannot recover speech missing from every candidate.

#2. Representations determine what later components can do

#Waveforms and learned analysis–synthesis

A learned encoder converts short waveform segments into feature channels; a decoder converts modified features back into samples. Conv-TasNet predicts masks in this learned domain using dilated temporal convolutions. The masks are not frequency-bin gains: the learned channels need not correspond to physical frequencies. Its original experiments concern single-channel two- and three-speaker separation. Conv-TasNet paper

The practical attraction is joint optimization of representation and reconstruction. The cost is that another model's latent features are not interchangeable merely because their channel counts match. Treat an encoder and decoder as a trained pair unless a new adapter is explicitly learned and tested. A short analysis kernel also does not establish low end-to-end latency when the separator uses future context.

#Short-time Fourier transforms

The short-time Fourier transform (STFT) describes short overlapping audio windows with complex coefficients: magnitude describes local strength, and phase describes position within an oscillation. It provides physically meaningful frequency bins and retains cross-microphone phase relationships. That makes it a natural interface for masks, complex filters and beamforming.

For example, the inspected TIGER configuration uses 16 kHz audio, a 640-sample window and a 160-sample hop: 40 ms windows every 10 ms, with 321 one-sided frequency bins. Those values define the representation, not just the tensor shape. Resampling audio or changing the Fourier window changes what a frequency index means. TIGER paper, Appendix B, pinned configuration

#Pretrained representations are evidence, not ready-made waveforms

WavLM learns speech representations through masked prediction and denoising. Its official extraction example expects 16 kHz waveform input and applies normalization according to checkpoint configuration. These features can support a downstream model, but the encoder is not itself a waveform separator or a universal speaker-identity embedding. WavLM paper, official extraction documentation

There is direct evidence for combining WavLM with enhancement systems: Song and colleagues found quality and recognition gains especially with limited fine-tuning resources; at higher resources, only word error rate improved substantially. This supports testing pretrained features under the actual data regime, not adding them by default. Exploring WavLM on Speech Enhancement

Discrete-token reconstruction is another family, but requires its own compatible tokenizer and waveform generator. TSELM combines quantized WavLM layers, target conditioning and a HiFi-GAN decoder. This is a published system design, not evidence that arbitrary tokens can replace continuous separator features. Official TSELM implementation

#3. Model the dependencies the representation leaves unresolved

Speech contains local events, sustained harmonic structure and long-range identity cues. A backbone needs enough context to distinguish these, without paying for every possible interaction.

Temporal convolutions expand context through stacked or dilated filters. Dual-path networks split long sequences into chunks, then model relationships within and between chunks. DPRNN demonstrates this organization with recurrent networks. “Chunked” does not automatically mean streaming: inter-chunk processing can still access future chunks. DPRNN paper

Spectral and temporal modeling can also be separated explicitly. TF-GridNet alternates full-band processing within a frame, temporal processing within sub-bands and cross-frame attention, predicting real and imaginary spectral components. Its evaluations include anechoic, noisy and reverberant separation. TIGER instead compresses frequency bands and combines multiscale processing with full-frequency-frame attention, seeking a different quality–computation balance. Neither result establishes a universally better backbone. TF-GridNet paper, TIGER official implementation

The design question is therefore specific: does the current failure require finer local detail, longer context, cross-frequency information or better target identity? Adding global attention to a system that consistently follows the wrong speaker may reinforce a wrong decision rather than correct its cause. This is an engineering interpretation to test, not a result claimed by either paper.

#4. A compact map of component roles

The table identifies representative mechanisms, not a ranking. Each row's source defines the published scope.

AV-HuBERT was developed for audio–visual representation learning and recognition. A visual-only extraction configuration must actually exclude unavailable audio, rather than silently using a multimodal teacher's inference inputs. Dolphin illustrates a different use of pretraining: its compact DP-LipCoder is distilled from AV-HuBERT and incorporated into an audio–visual separator. AV-HuBERT paper, Dolphin paper

Names need verification. Here Dolphin means Li, Gao and Hu's audio–visual model, not an unspecified multichannel adaptation. IPDnet means Wang, Yang and Li's direct-path phase estimator; the checked repository distinguishes fixed- and variable-array implementations. A locally named “IPDNet2” or visually conditioned derivative cannot inherit the original paper's claims without its own provenance. The original public model is not a lip-conditioned target extractor. IPDnet repository

#5. Conditioning and reconstruction are separate choices

A target embedding can be concatenated with mixture features, used to scale and shift them, or supplied through cross-attention. These are engineering design options: concatenation permits learned interactions; modulation biases existing channels; cross-attention matches query information to a sequence. No fusion operation guarantees that the model uses the cue correctly. Test a changed target cue, an irrelevant cue and a missing cue while holding audio fixed.

For reconstruction, distinguish four operations:

• Real-valued spectral masking changes magnitudes while retaining mixture phase when the mask is nonnegative. It is a constrained output mechanism.
• Complex masking or direct complex mapping can change magnitude and phase. Mapping predicts target coefficients directly; a mask multiplies observed coefficients.
• Deep filtering combines neighboring complex coefficients across time rather than multiplying only the current bin. DeepFilterNet uses this after spectral-envelope enhancement to exploit local speech structure. Future taps, if used, contribute lookahead. DeepFilterNet paper
• Beamforming combines synchronized microphone channels. In the documented Souden MVDR implementation, the inputs include multichannel complex spectra, target/noise spatial covariance matrices and a reference channel; the output is a single-channel complex spectrum. TorchAudio 2.6 MVDR tutorial

A spatial covariance matrix summarizes cross-channel relationships at a frequency. Masks can estimate these statistics, but a target speech mask and a complex reconstruction mask have different roles. MVDR seeks noise reduction subject to a target-preservation constraint; Wiener filtering instead embodies a reconstruction-error trade-off. TF-GridNet supplies a published example of a two-network system with a multi-frame Wiener filter between networks. That evidence concerns its tested system, not arbitrary encoder–filter substitutions. TF-GridNet paper

The interpretation is that stronger output freedom can repair errors a simple mask cannot represent, while also creating more ways to distort speech. Choose the head together with the reference definition, loss and preservation checks.

#6. Compatibility is a contract, not a matching dimension

Before joining modules, write down the following interface facts. These are recommended integration checks, not claims that every published system already satisfies them.

The incompatibility can be concrete. Public IPDnet uses 16 kHz audio with a 512-sample STFT window and 256-sample hop; its reported localization output occurs every 12 frames. TIGER's checked grid is 640/160. A spatial hidden tensor cannot be inserted into a TIGER frame simply by matching array dimensions. A proposed adapter must define timestamps and frequency meaning; interpolation of a direction estimate is not equivalent to interpolation of a complex phase representation. IPDnet paper, configurations, TIGER configuration

Training assumptions are also version-specific. Dolphin's paper and currently inspected repository configuration differ in some optimization settings. The pinned model code loads and freezes the video encoder in its training initialization. A reproduction should specify whether it follows the paper or that code revision, and pin the weight revision rather than relying on a moving “main.” Dolphin configuration, model initialization

#7. Four architecture recipes, with explicit evidence strength

Published below means the cited combination was evaluated in its source setting. Engineering-compatible proposal means a concrete signal path can be specified, but its implementation and benefit remain unverified here. An untested hypothesis is a proposed mechanism of improvement, not a performance claim. Costs below describe operations and likely bottlenecks, not measured timings for a new system.

#Recipe A: Enrollment-conditioned extraction for a known voice

Task and flow. Mono mixture → learned audio features → target-conditioned separator → matched decoder. A separate enrollment utterance supplies the target representation. This is the published SpeakerBeam family; the official tutorial implementation includes an 8 kHz Libri2Mix recipe. It is a suitable starting point when enrollment speech is available but video is not. SpeakerBeam implementation and paper references

Interface and training proposal. Reproduce the chosen recipe before changing components. Mixture, reference and enrollment must use its sampling rate and amplitude policy; enrollment must identify the target without being the same aligned mixture segment. Preserve the recipe's learned adaptation pathway. If substituting a pretrained speaker encoder, initially freeze it and train an adapter and separator; that substitution is a new proposal, not the published SpeakerBeam result.

Mechanism, cost and failure. A stable voice cue could reduce identity ambiguity without a visual frontend. An enrollment embedding can be cached; separator context and decoder dominate recurring work. Short, noisy or mismatched enrollment may encode channel conditions instead of identity. A weak target can still be erased.

Minimum validation. Compare correct, wrong and absent-target enrollment; include clean target-only audio and same-speaker enrollment recorded on a different device. Measure both reconstruction and identity errors. Optional WavLM features are a separate hypothesis requiring a no-feature control and a measured compute budget.

#Recipe B: Direct visual conditioning when reliable lip motion is available

Task and evidence. Dolphin is a published combination: synchronized mouth video → pretrained DP-LipCoder; mono audio → learned waveform encoder; visual conditioning and global–local modeling → target reconstruction. Its reported tests use LRS2, LRS3 and VoxCeleb2 mixtures. This supports the complete system in those settings, not arbitrary replacement of either encoder. Dolphin paper

Interface and training. The reported setup uses 16 kHz audio, 25 frames/s and 88×88 grayscale lip inputs. The inspected code freezes its loaded video encoder while training the remaining pathway. Retain crop normalization, valid-frame treatment and audio–video alignment. Multiple microphone channels are not accepted simply by extending the mono input dimension. Pinned configuration, model code

Mechanism, cost and failure. Visual articulation offers target-related evidence during acoustic overlap. Face tracking, cropping and visual encoding add cost; global context prevents assuming strict causality. Occlusion, off-axis faces, timing errors or a wrong face track can contaminate reconstruction.

Minimum validation. Reproduce an in-domain checkpoint result, then test synchronized versus shifted video, wrong-person video, missing frames and target-only preservation. A matched audio-only control tests whether visual information helps rather than merely being consumed.

#Recipe C: Separate first, identify later when audio candidates are already useful

Published precedent. DAVE uses an audio-only TIGER-M backbone and a separate attribution stage. Its public attribution combines voiceprint, StableSyncNet, SyncNet and lip-keypoint evidence; TalkNet is not that published selector. The repository's standalone CLI returns two 16 kHz streams, while attribution and selective enhancement are reference pipelines dependent on additional assets. DAVE official repository

Proposed flow. A fixed TIGER-family separator → anonymous waveforms → candidate/face scoring with TalkNet → calibrated target selector → unchanged selected waveform. This is an engineering-compatible proposal, not an official DAVE reproduction. TalkNet was trained for active-speaker detection; applying it to separated candidates changes its input distribution. Its checked demo uses 16 kHz audio, 13-dimensional mel-frequency cepstral coefficients at 10 ms steps, and a 25 frames/s face-crop pipeline. Use that preprocessing, not Dolphin's lip crops. TalkNet paper, pinned demo

Training, mechanism and cost. Initially freeze both models and train only calibration/selection on labeled development data. The hypothesis is that video can improve assignment without changing candidate samples. Track candidate identity across windows before smoothing decisions. Scoring grows with candidate–face pairs; long-context scores add delay. Wrong selection remains possible, and no selector repairs an absent or damaged candidate.

Minimum validation. Compare predicted assignment with separately labeled oracle assignment, random assignment and audio-only identity scoring. Test stream permutations, batch independence, missing video and switching artifacts. Confirm selected samples are unchanged. DAVE's challenge-specific “non-degradation” argument for untouched reference partitions is not a guarantee of preserved real-world words. DAVE paper, §§3.2–3.3

#Recipe D: Target-conditioned multichannel filtering

Task and proposed flow. Synchronized array audio → multichannel STFT → target/noise mask estimator conditioned on enrollment or video → spatial covariance estimates → MVDR → inverse STFT. The beamforming interface is documented; the particular cue-conditioned estimator is an engineering proposal. A directly tested alternative is TF-GridNet's multichannel network–Wiener-filter–network system. Do not label the proposed MVDR recipe as that published system. TorchAudio MVDR tutorial, TF-GridNet

Interfaces and training proposal. Choose one sampling rate and STFT grid for filtering; preserve channel order, calibration, reference microphone and source timing. Begin with separately verified mask and filter stages, then compare frozen versus joint optimization through a numerically checked filter. IPDnet direction or phase evidence is an optional hypothesis: its grid, array variant and target association require adapters and controls.

Mechanism, cost and failure. Spatial filtering can exploit channel relationships unavailable to mono processing. Covariance accumulation and matrix solves add cost; the averaging interval trades responsiveness against statistical reliability. Poor masks, short low-energy regions, moving speakers, near-coincident directions or array mismatch can damage the target. Whole-recording statistics are not causal.

Minimum validation. Check known-delay signals and channel permutations; compare reference-informed masks with learned masks as diagnostics; test finite outputs in quiet windows. Compare the identical estimator with and without spatial filtering and the same recipe with and without spatial conditioning. Add a residual postfilter only as another separately controlled hypothesis.

#8. Losses and training should follow the contract

A common solution to anonymous-output ordering is permutation-invariant training (PIT): select the output–reference assignment with minimum training loss. This solves ordering ambiguity during optimization; it does not supply the requested person's identity. Target-conditioned extraction instead has a cue-defined reference. Original PIT paper, utterance-level PIT

Scale-invariant signal-to-distortion ratio (SI-SDR) evaluates reconstruction up to a global scale. It is useful for separation but cannot alone enforce output level. Pair it with amplitude-sensitive reconstruction and explicit target-only/absent-region checks when preservation matters. Perceptual quality, intelligibility and recognition should remain separate outcomes. SI-SDR definition and metric analysis

ASR-aware training is possible through a frozen recognizer with a differentiable frontend. DAVE reports teacher-forced cross-entropy on transcripts; despite its “character error rate (CER) loss” label, this is not direct differentiation of discrete character edit distance. Its reported gains are tied to its data and recognizer. Retain waveform supervision and evaluate with another recognizer before claiming general transcription benefit. DAVE paper, §3.1

The following is a proposed training progression, not a universal published schedule: verify pretrained inference; train a small adapter or head; compare selective unfreezing at a matched budget; then consider joint optimization if the frozen interface is demonstrably limiting. Freezing weights must also handle normalization buffers and train/evaluation modes. Preserve optimizer state for a true continuation, and record intentional resets.

Data design should include target-only speech, absent targets, overlap, varied levels and realistic cue failures. Reserve speakers, sessions and acoustic conditions appropriate to the generalization claim. The value of a loss or modality should be tested against an unchanged-training control, not just the initialization.

#9. Choose architectures through questions and small discriminating tests

Before choosing a recipe, answer:

1. Is the main failure noise, overlap reconstruction, wrong-speaker selection or recognition?
2. Which reliable cue will actually exist at inference: enrollment, face track, geometry or none?
3. Does each component output the information the next one needs, at the right time and in the right domain?
4. Would a reference-informed selector or mask diagnose missing capacity, or merely hide the real deployment problem?
5. What future context does the whole pipeline require, including video and covariance estimation?
6. Which preservation failures would make an average improvement unacceptable?
7. What evidence would make you reject the added component?

A minimal comparison should lock data, checkpoint selection, recognition normalization and output assignment; report overlap and target-only quality, target absence, wrong-speaker rate, lower-tail failures, ASR errors and paired listening. Measure acquisition/context delay separately from warmed processing time. Count preprocessing, all encoders, tracking, filtering and transfers. These are proposed decision criteria; no new combination has been benchmarked for this guide.

Primary papers and official code were checked on 15 October 2026. Repository interfaces above are pinned where inspected; paper evidence includes TIGER v2, Dolphin v1, DAVE v1, WavLM v5, AV-HuBERT v2 and the cited IPDnet paper. These are representative sources rather than an exhaustive survey. No published ranking is transferred across datasets or local adaptations.

#10. A brief illustration from the BYD experience

The project provides two examples of these general distinctions. First, repairing covariance arithmetic removed extreme outputs without making saturated learned masks useful. Numerical correctness and separation quality required different checks. Second, a learned residual selector improved a locally fine-tuned DAVE/TalkNet pipeline, but matched feature controls did not establish robust incremental spatial benefit. Correct attribution of a pipeline gain required more than its final score.

The final selected SAVE v2 residual obtained 19.1373% test CER versus 23.1752% for matched temporal TalkNet; the earlier 18.9984% result belongs to another candidate. These are local adapted systems, not results for official DAVE or TalkNet alone. The approximately 17% objective remained unmet. Physical array mapping and audio–video alignment remain unverified, and local listening cannot establish general device performance.

Internal provenance: SAV-BandNet — experimental record and analysis, original doc_302b58d8ce, 15 October 2026, §§5–7 and “15 October: bounded DAVE-preserving spatial–visual exploration,” project snapshot doc_5071d94dc2; Experimental Results, original doc_99259c8241, 15 October 2026, SAVE v2 section, snapshot doc_5c81a9cddc. The earlier detailed retrospective remains in this document's version history.