#Related Work: positioning SAV-BandNet

Updated 8 October 2026. Focused research synthesis for the current gate-bypassed Wiener21 model.

#1. The problem and the defensible research question

We aim to extract a visible target speaker from synchronized six-channel Mandarin audio and mouth video under overlap, noise and reverberation. The current training target is the full reverberant source at microphone 0. Consequently, our present task is extraction in reverberation; explicit dereverberation is not an established capability.

The literature already provides visual target selection, multichannel spatial features, neural beamforming and activity-aware extraction objectives. Combining audio, video and spatial information is therefore insufficient as a novelty claim. The useful question for SAV-BandNet is more specific:

Does learned, visually conditioned target-phase evidence improve a band-wise extractor beyond ordinary visual conditioning and direction-derived spatial features, while preserving target-only speech and suppressing target-absent output?

This is a research hypothesis. The current implementation and development results motivate testing it; they do not yet answer it. Model specifications remain in SAV-BandNet — current model proposal (doc_776d1f0e0f), and local measurements remain in the experimental record (doc_302b58d8ce).

#2. Closest prior work: multimodal multichannel target extraction

#Spatial and visual fusion already exists, including for Mandarin

Gu et al., Multi-modal Multi-channel Target Speech Separation (2020) is a central comparator. It combines multichannel audio, target direction, lip motion and enrolled speaker characteristics through multimodal fusion, including factorized attention. Its noisy, reverberant Mandarin mixtures use simulated room responses. The paper also studies missing lip information and direction-estimation errors. Its direction-dependent feature compares observed phase with target phase derived from direction and array geometry. Primary paper, §§III–V

This establishes that Mandarin coverage, multimodal fusion and robustness to imperfect target cues all have precedent. SAV-BandNet's narrower distinction is its use of continuous target-phase predictions and features from an adapted visual IPDnet2, rather than requiring a decoded target direction to construct that conditioning. Whether this representation improves extraction is unresolved. A matched direction-feature baseline is needed; comparing only against a mono AV separator would leave the central question unanswered.

Li et al., Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation and Recognition (2023) integrates visual information into mask-based MVDR, dereverberation and recognition. It examines DNN-WPE, spectral mapping and joint WPD configurations on simulated and replayed LRS2 mixtures. Its audio features include direction information obtained by tracking a face with a wide-angle camera. Primary paper, §§II–VII

Thus, an end-to-end AV spatial-filtering pipeline is also established. Our mouth-conditioned learned spatial representation differs from camera-derived direction, but this is a design distinction, not demonstrated superiority. This work also sets a stronger dereverberation reference than our current full-reverberant-target objective.

#Spatial reconstruction and numerical stability have substantial precedent

Xu et al., Neural Spatio-Temporal Beamformer (2020) extends mask-based MVDR to exploit both inter-microphone and inter-frame correlation, using complex masks and joint training. It connects waveform reconstruction with downstream recognition, rather than assuming that a separation score fully describes useful speech quality. Primary paper

Zhang et al., Multi-channel Multi-frame ADL-MVDR (2021) discusses unstable matrix operations during joint training and replaces inverse/eigenvector computations with recurrent estimators. Its evaluation uses a Mandarin AV corpus, and visual information supplies target direction. This is relevant both to our multichannel reconstruction design and to our original MVDR failures. Primary paper, §§II–IV

SAV-BandNet instead retains an analytic Wiener computation with direct double-precision covariance accumulation and diagonal loading. Our observed covariance cancellation, mask saturation and subsequent Wiener preference are local implementation and optimization findings. They do not establish that MVDR generally fails, that Wiener is universally better, or that differentiable spatial filtering is new.

#3. What visual, acoustic and localization models contribute

#Visual conditioning identifies the requested speaker

Looking to Listen established speaker-independent separation using visual information. VisualVoice combines lip motion with face–voice consistency, showing that temporal speech evidence and speaker identity are distinct useful cues. These methods establish the visual-attribution foundation of our task. Ephrat et al., 2018; Gao and Grauman, 2021

Dolphin supplies a closer modern AV baseline through discrete lip representations and global–local fusion. Its evaluations include constructed mixtures from LRS2, LRS3 and VoxCeleb2, with additional challenging acoustic conditions. These results support its representation and fusion design within those protocols. They do not by themselves establish transfer to our physical circular array or natural Mandarin conversations. Primary paper, method and experimental appendices

The remaining question is not whether video can help separation. It is whether our spatial representation adds information beyond a strong visual-conditioned extractor under matched acoustics and target selection. Our retained six-channel Dolphin pipeline is a separate system and should remain an engineering comparator, not be described as SAV-BandNet's result.

#Efficient band processing and learned spatial processing are established

TIGER provides efficient frequency-band processing and temporal/frequency context modelling. Its EchoSet experiments also investigate more realistic simulated acoustics. SAV-BandNet inherits its acoustic backbone; band partitioning and efficient separation are not new contributions of our model. Primary paper

SpatialNet learns multichannel spatial structure for separation, denoising and dereverberation. Its evaluations include varying array geometries and real replay data. It therefore challenges any blanket claim that existing learned separators ignore spatial structure or have no evidence of array generalization. Its coverage still does not verify our microphone mapping, device response or visual-target extraction. Primary paper, §§III–IV

These works motivate two controls: a strong multichannel acoustic model, and the same SAV acoustic backbone with simpler conditioning. Gains against a weaker mono baseline would not isolate the value of the proposed fusion.

#Localization features need an extraction-specific test

IPDnet2 estimates direct-path inter-channel phase differences using an oSpatialNet backbone and frequency/time pooling. The paper explicitly studies where compression preserves useful localization information. Its task is localization, not visible-target waveform reconstruction. Primary paper, §§2–3

Our adaptation adds visual target conditioning, and the extractor consumes its continuous phase predictions and intermediate features. The extractor preserves a 10 ms acoustic grid around fusion on a 100 ms timeline. The engineering rationale is to reuse slower spatial context without discarding fine acoustic detail.

That rationale remains a hypothesis about extraction. Better azimuth error does not establish better masks or waveforms. Coarse-versus-fine fusion and raw-versus-learned spatial evidence need matched comparisons. FiLM itself is an existing conditioning mechanism, so bounded feature modulation should be attributed rather than presented as a new operator. Perez et al., FiLM

#4. Preservation and unreliable cues are part of the task

USEV treats target-present and target-absent intervals explicitly. Its differentiated objective separates activity scenarios and uses scale-sensitive reconstruction for active speech and output-energy penalties for inactive regions. This directly precedes our concern that average separation gains may conceal target attenuation or leakage. Primary paper, §III-B

Our balanced regional amplitude loss and absence penalty belong to this established line of work. Their useful contribution would be demonstrated behavior under our room, overlap and target conditions, rather than the general idea of activity-aware training.

Missing video, wrong-person video and audiovisual time offsets also require separate tests. RAVSS studies missing visual information in a multiple-speaker setting; its available cues differ from our single supplied target mouth crop. Such results establish partial robustness evidence, not universal protection against visual failure. Primary paper

For SAV-BandNet, validity information and gate bypass do not guarantee graceful failure. A frozen visual/spatial encoder can still provide misleading conditioning. Claims of robustness should therefore report overlap extraction, target-only amplitude-sensitive SNR and level error, and target-absent leakage alongside video-corruption tests.

#5. The remaining application gap is joint validation

MISP 2023 provides a directly relevant natural Mandarin AV target-extraction setting. It connects enhancement to recognition, but natural near-field recordings should not be assumed to be exact isolated clean references. Our dev100 CER protocol is a local evaluation and cannot be ranked directly against challenge results. Challenge paper

RealMAN provides real-recorded multichannel Mandarin acoustics and localization annotations. Its loudspeaker playback and camera setup do not supply natural speaking-mouth video for lip-conditioned extraction. EchoSet supplies controlled room simulation but not native synchronized mouth video. These resources answer complementary acoustic questions. RealMAN; TIGER/EchoSet

Our synthetic circular-six data connects synchronized mouth donors with controlled source images and nominal device geometry. It permits diagnostic evaluation but does not prove physical-array transfer. The important validation gap for this project is the combination of target identity, room mismatch, actual channel mapping and preservation in real Mandarin use. This is a gap in our evidence, not a claim that previous literature lacks Mandarin or reverberant multichannel systems.

#6. Where SAV-BandNet can contribute, and what would establish it

These comparisons are proposed research tests, not completed experiments. No “first” claim is established by this review. A different combination of existing components becomes a substantive contribution only when a specific mechanism or practical advantage is demonstrated.

#What our current evidence supports

The consolidated experimental record reports full-500-validation overlap SI-SNR improving from 6.0403 to 6.3498 dB between updates 5,000 and 7,000. Target-only SNR improves from 15.6709 to 16.1501 dB, while absence suppression declines from 27.5881 to 26.4942 dB. These are repeatedly used development data and show an optimization trade-off, not superiority over the closest published systems.

Localization adaptation improves matched synthetic overlap MAE from 17.0731° to 10.6147°. Some test scenes had previous exposure, and this result does not isolate an extraction gain from spatial conditioning. Gate bypass, regional amplitude balancing and TIGER unfreezing changed together, so their individual effects remain unresolved. Real-array gains, SAV-specific CER gains and deployment latency remain unestablished.

#Working positioning for a paper

Existing audiovisual extraction methods already combine target identity with spatial cues, while neural beamforming provides structured multichannel reconstruction. SAV-BandNet investigates whether a learned, visually conditioned phase representation can guide a band-wise mask estimator through coarse temporal conditioning while retaining fine acoustic resolution. Its intended contribution is the measured value of this representation and integration under reverberant Mandarin conditions, evaluated jointly with target preservation. The present results support continued investigation; a causal advantage and real-world benefit still require matched baselines and external validation.

#7. Scope and remaining literature checks

This revision prioritizes the closest mechanisms rather than preserving a broad catalogue of loosely related methods. Core method/evaluation passages were checked for Gu et al., Li et al., ADL-MVDR, SpatialNet, IPDnet2, Dolphin and USEV. Some supporting entries use primary abstracts or earlier checked resource descriptions. No external system was independently reproduced for this review.

A further close candidate is Multimodal and multichannel speech separation using location-guided speech feature mapping network (2025). Its publisher search summary indicates lip-coordinate guidance and complex multichannel features, but full text was unavailable in this pass. It must be checked before narrowing a novelty claim around learned spatial/visual fusion. Publisher record

The next literature priority is to resolve the nearest architectural overlap and baseline availability, rather than collect more generic AVSS papers. Explicit source-specific dereverberation remains a separate possible extension; it should not be retroactively attributed to the current Wiener21 extractor.