OS-Atlas/eval/eval_scripts/android_control /evaluation_unify.py评测问题

[DQYZHWK]

grounding指标只统计了action type为 click的，其实long_press 也有坐标预测。
为了使你的数据有意义，我应该和你对齐，只使用click作为grounding指标统计是吗

[zhshj0110]

Excuse me.
Can you reproduce the performance of high-level android_control based on the given checkpoint of OS-Atlas-Pro-7B?

[DQYZHWK]

想问下high level 中到底用没有到previous actions 信息？ 你们的论文貌似说没有用到，但另外有篇文章UGround说用到了

[zhshj0110]

I suppose the previous actions were used.
Accoding to the providing ac_test.jsonl and the author's response at #47

[DQYZHWK]

根据os-atlas table 9 我认为是用到了

[SYSU-AIGC]

@DQYZHWK @zhshj0110 @xufangzhi

Sorry to bother you, we cannot reproduce the performance reported in the paper, even when following evaluation_unify.py exactly.

Besides, we notice that there are some formatting errors in the outputs, like,
"{"episode_id": 7548, "step": 1, "instruction": "Look for running shoes for women with size 8 in size filter in JD app ", "screenshot": "7548/screenshot_1.png", "action": {"action_type": "None", "action_inputs": null}, "reason": "format error: thoughts:\nClick on the plus icon of size brand\nactions:\nCLICK [[931,done\nactions:\nCLICK [[, done><|im_end|>"}"

Can you reproduce the performance or kindly give any hint on this? Thank you