Improving the Robustness of Audio-Visual Target Speaker Extraction With AV-HuBERT Based Lip Features
The target speaker extraction task aims to extract clean speech of the target person from a segment of mixed speech. In recent years, audio-visual speech enhancement (AVSE) has been increasingly applied, and the use of visual information from the target speaker has important application value in noisy environments. However, existing AVSE methods often face the problem of insufficient robustness of visual features, especially when parts of the content are missing or the video quality is poor. This will significantly reduce the effectiveness of the extracted visual features, further affecting the extraction performance. To address this issue, this paper first introduces a power compression strategy to enhance the effective components of the speech signal and avoid overreliance on visual information. Then, an end-to-end training approach is adopted to optimize the feature extraction process, initially alleviating the problem of insufficient robustness of lip movement features. To further improve performance, this paper uses the self-supervised AV-HuBERT model to extract features of lip movement. Through its multimodal self-supervised learning strategy, it can capture more discriminative dynamic features of lip movements and also achieve deep consistency between audio and video features. Experimental results show that the proposed method achieves stable improvements in key metrics such as PESQ, STOI, and SI-SDR, verifying the importance of visual feature extraction in the AVSE task and providing ideas for target speaker extraction in complex scenarios.
