关于AC推理代码中a11_tree字段

[CTZhou-byte]

请问下面prompt中a11_tree应该提供什么信息。此外，low-level的测评是在哪改动？
’‘You are a GUI task expert. I will provide you with a high-level instruction, an action history, and a screenshot with its corresponding accessibility tree.\nHigh-level instruction: {high_level_instruction}\nAction history: {action_history}\nAccessibility tree: {a11y_tree}\nPlease generate the low-level thought and action for the next step.
'''

[CTZhou-byte]

我解析并生成了A11_tree字段的数据，但是我看提供的json是用的绝对坐标，但是模型预测似乎是相对坐标。这样参考A11_tree会不会有误差？

[heroding77]

我解析并生成了A11_tree字段的数据，但是我看提供的json是用的绝对坐标，但是模型预测似乎是相对坐标。这样参考A11_tree会不会有误差？

Hi @CTZhou-byte , thanks for reaching out.

This is the expected behavior. The model is trained on the a11y tree and, as a result, predicts absolute coordinates from it.

So, if your inference input includes the a11y tree, the output will correctly be the absolute coordinates from that tree

[heroding77]

请问下面prompt中a11_tree应该提供什么信息。此外，low-level的测评是在哪改动？ ’‘You are a GUI task expert. I will provide you with a high-level instruction, an action history, and a screenshot with its corresponding accessibility tree.\nHigh-level instruction: {high_level_instruction}\nAction history: {action_history}\nAccessibility tree: {a11y_tree}\nPlease generate the low-level thought and action for the next step. '''

For the AC evaluation, the a11y tree we use is parsed directly from the raw AC dataset.

For the low-level evaluation, you simply need to add the low-level instructions to your input.