Add compilation performance regression tests

[baszalmstra]

• claim this issue (assign yourself or comment below)
• setup repository on your local machine and make sure all tests pass (cargo test)
• read our contributing guidelines
• add a framework (I've been looking at criterion but you're free to use whatever) to benchmark compilation times and add regression tests
• start a Pull Request. Set description to closes #57. If this is your first PR, welcome 🎉 😄

This could be combined with #58

[baszalmstra]

Some initial runtime performance benchmarks have been added here: #104

[ChrisPWill]

Howdy, I'd like to have a go at tackling this. It may take a little time since I'll need to learn a little about the libraries in question, and don't have a huge amount of time at the moment, but it doesn't seem super difficult.

Do we have a definition of done for this issue? Is it a case of setting up the framework and adding "some" regression tests?

[Wodann]

Hi, thank you for helping out! 🙂

The most uncertain part is the Continuous Integration. As we don't have dedicated hardware, our regression tests will suffer from variable workloads. That will probably result in a lot of false positives/negatives.

Until we find enough funding or a sponsor, we will probably need to manually run these regression tests periodically (and consequently pinpoint regressions in history) and ask suspect PRs to do the same. So no need to worry about that aspect.

W.r.t. specific tests, you could:

• reuse Mun code in existing test cases (there are a lot of them, so that should provide some volume)
• reuse Mun code in the Rust examples

Metrics that we'll want to benchmark are:

• first-time compilation
• incremental compilation
• edit-to-hot-reload latency (this benchmark will need to be in the mun_runtime crate)

Overall, that might be quite some work, so feel free to only implement a subset of these in your first PR.

[ChrisPWill]

Great, thanks for the clarification and tips! That's a shame re: CI, but I guess that can be fixed down the track. It all sounds like good fun and I'm excited to be involved, will try and open a WIP PR some time soon to keep the discussion going.

[Wodann]

Hi @ChrisPWill. How has your work on this been going? Do you need any help?