Annotation Quality

[dakshvar22]

Hi,

Thank you for sharing this dataset. I've read the paper introducing the dataset and it seems to suggest that the dataset was collected and annotated by domain experts. I went through the dataset manually and observed a lot of wrong annotations. For example, intent label CHAT_WITH_AN_AGENT in Powerplay11 dataset is highly overlapping with other intents which express some user problem, for e.g. - WINNINGS, etc.
Could you please clarify what was the extent of manual supervision applied during the collection of the dataset? Moreover, any inter-annotator agreement scheme that may have been used would be useful to know.

Regards,
Daksh

[goru001]

Thanks @dakshvar22 for reaching out.

The train set for all 3 datasets was prepared by domain experts. We had given strict guidelines to discard/merge intents which might have overlapping training data. We also have an automatic layer on top of this which flags if two intents are highly overlapping on our platform. These datasets were filtered through both these layers. Having said that, there can be generic and specific intents in a bot, about which we've talked about in our paper, where it is expected that bot will be able to figure out if the user query is supposed to be answered from generic or specific node. CHAT_WITH_AN_AGENT is an example of a generic node.

The test set was manually labelled by 3 different annotators and conflict was resolved by domain experts. We've reported inter-annotator agreement in the paper. Quoting directly from the paper:

Inter-annotator agreement was 75.8%, 80.0% and 73.4% for SOFMattress, Curekart and Powerplay11 respectively and conflicts were resolved by domain experts. One major reason for low inter-annotator agreement was unclear criteria for defining an intent which sometimes lead to overlapping intents of different levels of granularity, even after we had made sure to manually merge any conflicting or highly similar intents in the training data.

Having said that, we've also noticed in our analysis of the results that there are few ground truth labels which are incorrect. We've been working on it and expect to release corrections by next week.

Let me know if this answered your question or if there's anything which you think I missed out.