Hi, thanks for your wonderful work!
In the examples/summer folder, we only found example code snippets for individual components. However, we could not find an end-to-end implementation that covers the full workflow—starting from the dataset, running the Summarize and Retrieve steps to generate the final SUMMER QA prompts, and then parsing the model’s QA outputs to compute accuracy and other evaluation metrics.
Would it be possible for you to share the code for this end-to-end pipeline (or provide a reference implementation/script)? This would greatly help us reproduce your work more faithfully. Thank you very much for your time and help.
Hi, thanks for your wonderful work!
In the
examples/summerfolder, we only found example code snippets for individual components. However, we could not find an end-to-end implementation that covers the full workflow—starting from the dataset, running the Summarize and Retrieve steps to generate the final SUMMER QA prompts, and then parsing the model’s QA outputs to compute accuracy and other evaluation metrics.Would it be possible for you to share the code for this end-to-end pipeline (or provide a reference implementation/script)? This would greatly help us reproduce your work more faithfully. Thank you very much for your time and help.