Collect new 2024 code condests dataset

[jimbou]

Generate a new dataset from the Code Contests 2024 submissions with the following specifications:

• Unique ID: Each item must have a unique ID in the format task_id_submission_id.
• Tests Passed: Include a field indicating the percentage of tests passed by the submission.
• Failure Details: For incorrect submissions, include a way to highlight the failed tests or provide the differences from the correct version.
• Diverse Coverage:Ensure the dataset includes different submissions for each problem, covering a range of correctness levels (e.g., from 0% to 100% of tests passed).

[mechtaev]

@jimbou, great summary, thanks!

@yihandai3826, when constructing the dataset, please think about these two issues:

Consideration 1: if we want to focus on pairs (buggy submission, fixed submission) or arbitrarily chosen submissions. That is, if we consider the history of submissions of a given user for a given problem, it will look approximately like this:

Fail, Fail, Fail, Fail, Pass

Then, we take the submissions corresponding to the last pair of [Fail, Pass]

This decision will have the following implications:

1. The positive implication is that the incorrect version will have a more clearly defined and likely more subtle bug. It will be easier to understand what the bug is by just checking the diff between the buggy and the fix submissions, because the difference is likely to be not very significant. We will also be able to add an additional research question in our evaluation about how well HoarePrompt is able to detect that a bug it finds is fixed in the correct version.

2. The negative implication of this choice is that the dataset will have a bias towards small, subtle bugs like corner cases rather than submissions with completely wrong algorithms.

Consideration 2: if we should select problems with buggy submission for which we do not have any accepted submissions. For such problems, we will not be able to objectively evaluate HoarePrompt's ability to generate an error-exposing failing tests (which you implemented), because we cannot say if the test makes sense without also running it on an accepted submission.

I do not know what would be the best answer to these questions, but the decisions you take will significantly impact what is going to be written in the evaluation section of the paper, so please think about it carefully.