What if there is only one speaker in mix audio, and he/she is not the enrollment person

[SherryYu33]

First thank you for your great work!
I've tried your pretrained model, it can't solve the problem in the title, it just export the original audio.

In non-causal case, maybe i can use spk_model to clasify the whole audio, and export a zero tensor.
But in causal case, what should i do?
I've found that ecapa_tdnn can export frame-level embbeding, but i'm not sure if it's discriminative.

[mrjunjieli]

1. Ideally, if you use a speech sample from speaker A as enrollment, and there is no speech from speaker A in the mixture, the output should be a zero signal. However, in this project, we use SI-SNR as the loss function, which prevents us from outputting a zero signal. You could replace the loss function with MAE or MSE.

2. Additionally, you should incorporate this type of data into the training set to avoid the mismatch problem.

[SherryYu33]

I've tried to use MAE before, like |s-s_hat|+|S_real-S_hat_real|+|S_imag-S_hat_imag|, and add 20% such case. It made my model perform worse in common 2-speaker case.
Maybe zero tensor has strong impact in training, i don't know.

[mrjunjieli]

i guess it is hard to cover different scenarios with one model.

[KyleZhang1118]

I've tried to use MAE before, like |s-s_hat|+|S_real-S_hat_real|+|S_imag-S_hat_imag|, and add 20% such case. It made my model perform worse in common 2-speaker case. Maybe zero tensor has strong impact in training, i don't know.

You can refer to these works:
[1] Delcroix, M., Kinoshita, K., Ochiai, T., Zmolikova, K., Sato, H., Nakatani, T. (2022) Listen only to me! How well can target speech extraction handle false alarms? Proc. Interspeech 2022, 216-220, doi: 10.21437/Interspeech.2022-11252
[2] Zhang, K., Borsdorf, M., Pan, Z., Li, H., Wei, Y., Wang, Y. (2023) Speaker Extraction with Detection of Presence and Absence of Target Speakers. Proc. Interspeech 2023, 3714-3718, doi: 10.21437/Interspeech.2023-655

[SherryYu33]

I've tried to use MAE before, like |s-s_hat|+|S_real-S_hat_real|+|S_imag-S_hat_imag|, and add 20% such case. It made my model perform worse in common 2-speaker case. Maybe zero tensor has strong impact in training, i don't know.

You can refer to these works: [1] Delcroix, M., Kinoshita, K., Ochiai, T., Zmolikova, K., Sato, H., Nakatani, T. (2022) Listen only to me! How well can target speech extraction handle false alarms? Proc. Interspeech 2022, 216-220, doi: 10.21437/Interspeech.2022-11252 [2] Zhang, K., Borsdorf, M., Pan, Z., Li, H., Wei, Y., Wang, Y. (2023) Speaker Extraction with Detection of Presence and Absence of Target Speakers. Proc. Interspeech 2023, 3714-3718, doi: 10.21437/Interspeech.2023-655

thank you!

[SherryYu33]

I've tried to use MAE before, like |s-s_hat|+|S_real-S_hat_real|+|S_imag-S_hat_imag|, and add 20% such case. It made my model perform worse in common 2-speaker case. Maybe zero tensor has strong impact in training, i don't know.

You can refer to these works: [1] Delcroix, M., Kinoshita, K., Ochiai, T., Zmolikova, K., Sato, H., Nakatani, T. (2022) Listen only to me! How well can target speech extraction handle false alarms? Proc. Interspeech 2022, 216-220, doi: 10.21437/Interspeech.2022-11252 [2] Zhang, K., Borsdorf, M., Pan, Z., Li, H., Wei, Y., Wang, Y. (2023) Speaker Extraction with Detection of Presence and Absence of Target Speakers. Proc. Interspeech 2023, 3714-3718, doi: 10.21437/Interspeech.2023-655

I've read the 2nd paper, but i found other questions, what if the 1st half of audio is not target speaker, and 2nd half is the target speaker? Or 1st half of audio is not target speaker, and 2nd half is 2-speaker? etc. It seems that the Detection Module is utterance-level, not frame level.

[KyleZhang1118]

I've tried to use MAE before, like |s-s_hat|+|S_real-S_hat_real|+|S_imag-S_hat_imag|, and add 20% such case. It made my model perform worse in common 2-speaker case. Maybe zero tensor has strong impact in training, i don't know.

You can refer to these works: [1] Delcroix, M., Kinoshita, K., Ochiai, T., Zmolikova, K., Sato, H., Nakatani, T. (2022) Listen only to me! How well can target speech extraction handle false alarms? Proc. Interspeech 2022, 216-220, doi: 10.21437/Interspeech.2022-11252 [2] Zhang, K., Borsdorf, M., Pan, Z., Li, H., Wei, Y., Wang, Y. (2023) Speaker Extraction with Detection of Presence and Absence of Target Speakers. Proc. Interspeech 2023, 3714-3718, doi: 10.21437/Interspeech.2023-655

I've read the 2nd paper, but i found other questions, what if the 1st half of audio is not target speaker, and 2nd half is the target speaker? Or 1st half of audio is not target speaker, and 2nd half is 2-speaker? etc. It seems that the Detection Module is utterance-level, not frame level.

Yes, you are right. Basically, I recommend that you try different loss function combinations to handle the scenario when the target speaker is missing. Even for the scenario where the target speaker is missing in the entire sentence, the above method can still work to a certain extent, as reported in the 2nd paper. If you want to use additional modules to improve the accuracy of judgment, you can remove the pooling layer in the detection module to obtain the frame-level judgment result, but the specific structure needs further exploration. In this case, you essentially try to train a system combining the TSE and target speaker diarization model.

[SherryYu33]

I've tried to use MAE before, like |s-s_hat|+|S_real-S_hat_real|+|S_imag-S_hat_imag|, and add 20% such case. It made my model perform worse in common 2-speaker case. Maybe zero tensor has strong impact in training, i don't know.

You can refer to these works: [1] Delcroix, M., Kinoshita, K., Ochiai, T., Zmolikova, K., Sato, H., Nakatani, T. (2022) Listen only to me! How well can target speech extraction handle false alarms? Proc. Interspeech 2022, 216-220, doi: 10.21437/Interspeech.2022-11252 [2] Zhang, K., Borsdorf, M., Pan, Z., Li, H., Wei, Y., Wang, Y. (2023) Speaker Extraction with Detection of Presence and Absence of Target Speakers. Proc. Interspeech 2023, 3714-3718, doi: 10.21437/Interspeech.2023-655

I've read the 2nd paper, but i found other questions, what if the 1st half of audio is not target speaker, and 2nd half is the target speaker? Or 1st half of audio is not target speaker, and 2nd half is 2-speaker? etc. It seems that the Detection Module is utterance-level, not frame level.

Yes, you are right. Basically, I recommend that you try different loss function combinations to handle the scenario when the target speaker is missing. Even for the scenario where the target speaker is missing in the entire sentence, the above method can still work to a certain extent, as reported in the 2nd paper. If you want to use additional modules to improve the accuracy of judgment, you can remove the pooling layer in the detection module to obtain the frame-level judgment result, but the specific structure needs further exploration. In this case, you essentially try to train a system combining the TSE and target speaker diarization model.

thank you for your advice!

[xczhusuda]

我以前试过用 MAE，比如 |s-s_hat|+|S_real-S_hat_real|+|S_imag-S_hat_imag|，并添加 20% 的此类大小写。它使我的模型在常见的 2 扬声器外壳中表现更差。也许零张量对训练有很大影响，我不知道。

您可以参考这些作品：[1] Delcroix， M.， Kinoshita， K.， Ochiai， T.， Zmolikova， K.， Sato， H.， Nakatani， T. （2022） 只听我说！Target 语音提取处理误报的能力如何？Interspeech 2022， 216-220， doi： 10.21437/Interspeech.2022-11252 [2] Zhang， K.， Borsdorf， M.， Pan， Z.， Li， H.， Wei， Y.， Wang， Y. （2023） 通过检测目标说话人的存在和不存在来提取说话人。交互过程 2023， 3714-3718， doi： 10.21437/Interspeech.2023-655

我已经阅读了第 2 篇论文，但我发现了其他问题，如果音频的前半部分不是目标扬声器，而下半部分是目标扬声器怎么办？或者音频的前半部分不是目标扬声器，而后半部分是 2 扬声器？等。检测模块似乎是话语级的，而不是帧级的。

是的，你是对的。基本上，我建议您尝试不同的损失函数组合来处理目标说话人缺失的情况。即使对于整个句子中缺少目标说话人的情况，上述方法仍然可以在一定程度上起作用，正如第 2 篇论文所报道的那样。如果想使用额外的模块来提高判断的准确性，可以去掉检测模块中的池化层，得到帧级的判断结果，但具体结构有待进一步探索。在这种情况下，您实际上是尝试训练一个结合了 TSE 和目标说话人分类模型的系统。

感谢您的建议！

请问下您是否做过帧级相关实验？效果如何呢？

[SherryYu33]

我以前试过用 MAE，比如 |s-s_hat|+|S_real-S_hat_real|+|S_imag-S_hat_imag|，并添加 20% 的此类大小写。它使我的模型在常见的 2 扬声器外壳中表现更差。也许零张量对训练有很大影响，我不知道。

您可以参考这些作品：[1] Delcroix， M.， Kinoshita， K.， Ochiai， T.， Zmolikova， K.， Sato， H.， Nakatani， T. （2022） 只听我说！Target 语音提取处理误报的能力如何？Interspeech 2022， 216-220， doi： 10.21437/Interspeech.2022-11252 [2] Zhang， K.， Borsdorf， M.， Pan， Z.， Li， H.， Wei， Y.， Wang， Y. （2023） 通过检测目标说话人的存在和不存在来提取说话人。交互过程 2023， 3714-3718， doi： 10.21437/Interspeech.2023-655

我已经阅读了第 2 篇论文，但我发现了其他问题，如果音频的前半部分不是目标扬声器，而下半部分是目标扬声器怎么办？或者音频的前半部分不是目标扬声器，而后半部分是 2 扬声器？等。检测模块似乎是话语级的，而不是帧级的。

是的，你是对的。基本上，我建议您尝试不同的损失函数组合来处理目标说话人缺失的情况。即使对于整个句子中缺少目标说话人的情况，上述方法仍然可以在一定程度上起作用，正如第 2 篇论文所报道的那样。如果想使用额外的模块来提高判断的准确性，可以去掉检测模块中的池化层，得到帧级的判断结果，但具体结构有待进一步探索。在这种情况下，您实际上是尝试训练一个结合了 TSE 和目标说话人分类模型的系统。

感谢您的建议！

请问下您是否做过帧级相关实验？效果如何呢？

只能说有时候行有时候不行，取决于余弦相似度

[xduzhangjiayu]

I've tried to use MAE before, like |s-s_hat|+|S_real-S_hat_real|+|S_imag-S_hat_imag|, and add 20% such case. It made my model perform worse in common 2-speaker case. Maybe zero tensor has strong impact in training, i don't know.

我试过全部用SI-SNR loss, 效果还ok, mix场景性能也不会掉太多。反而用MSE loss会出现nan...