Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
ceee927
create base metric class
Feb 17, 2026
46b7456
return pre commit config versions
Feb 17, 2026
8c96457
#233 add new classes
Feb 23, 2026
950964c
#232 fix class
Feb 23, 2026
33624b2
#232 fix class
Feb 24, 2026
1053b57
Merge branch 'improvement/232-create-abstract-class-for-metric' of ht…
Feb 24, 2026
27f7888
#232 add aggregation
Feb 24, 2026
c38331a
#232 fix methods
Feb 24, 2026
901dd5a
#232 fix nits
Feb 24, 2026
c4efd79
improvement/233-create-classes-for-metrics add classes
Mar 3, 2026
800f8c0
improvement/232-create-abstract-class-for-metric fixes
Mar 3, 2026
2014e22
improvement/232-create-abstract-class-for-metric fixes
Mar 3, 2026
363551c
improvement/232-create-abstract-class-for-metric remove agent evaluation
Mar 3, 2026
9ab36e2
Merge branch 'improvement/232-create-abstract-class-for-metric' of ht…
Mar 3, 2026
a253f74
improvement/233-create-classes-for-metrics edit classes
Mar 3, 2026
777e00e
improvement/233-create-classes-for-metrics remove old files
Mar 3, 2026
5a55b43
improvement/233-create-classes-for-metrics remove extra files
Mar 3, 2026
c5bb100
improvement/233-create-classes-for-metrics remove changes
Mar 3, 2026
a1fac9f
improvement/233-create-classes-for-metrics remove changes
Mar 3, 2026
4ed5727
move file
Mar 3, 2026
e6f0ef3
move file
Mar 3, 2026
431ae40
refactoring
Mar 3, 2026
6f74153
add script
Mar 10, 2026
773822d
fixes
Mar 10, 2026
5e1b3a6
fixes
Mar 10, 2026
558b588
made fixes
Mar 10, 2026
d78bda3
new structure
Mar 10, 2026
9146421
remove class from other pr
Mar 10, 2026
fe8ea9c
changes after new structure
Mar 10, 2026
d749156
remove main
Mar 10, 2026
8f333e7
changes
Mar 10, 2026
6e6e8b2
changes
Mar 10, 2026
1c802ff
changes
Mar 10, 2026
8335005
simplyfying
Mar 17, 2026
f006930
remove utility class
Mar 17, 2026
e0f3c56
fix get_recommendation_rounds
Mar 17, 2026
4157fdc
fixes
Mar 17, 2026
3b068b4
resolve issues
Mar 24, 2026
d7efc12
Merge branch 'main' of https://github.com/iai-group/UserSimCRS into i…
Mar 24, 2026
8944621
fixes
Mar 24, 2026
335d3bd
Merge branch 'improvement/234-create-main-evaluation-script' of https…
Mar 24, 2026
98564af
Merge branch 'improvement/233-create-classes-for-metrics' of https://…
Mar 24, 2026
8be81e4
234-create-main-evaluation-script add eval script
Mar 24, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions config/default/config_evaluation.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
dialogues: data/datasets/moviebot/annotated_dialogues.json
metrics:
- satisfaction
- success_rate
- successful_recommendation_round_ratio
- reward_per_dialogue_length
output: data/evaluation/moviebot_non_quality_results.json

quality_llm_interface:
llm_interface_class_path: "usersimcrs.llm_interfaces.ollama_interface.OllamaLLMInterface"
llm_interface_args:
configuration_path: config/llm_interface/config_ollama_default.yaml
default_response: ""
quality_aspects:
- REC_RELEVANCE
- COM_STYLE
- FLUENCY
- CONV_FLOW
- OVERALL_SAT

user_nlu_config: config/default/config_default.yaml
agent_nlu_config: config/default/config_default.yaml

recommendation_intent_labels:
- REVEAL
- REVEAL.SIMILAR
- REVEAL.NONE
- REVEAL.REVISE
accept_intent_labels:
- NOTE.ACCEPT
reject_intent_labels:
- NOTE.DISLIKE
168 changes: 0 additions & 168 deletions scripts/evaluation/quality_evaluation.py

This file was deleted.

63 changes: 0 additions & 63 deletions scripts/evaluation/satisfaction_evaluation.py

This file was deleted.

Loading
Loading