Add WIT equivalent of WITX 0.10 spec

[Robbe-Haegeman]

The following PR introduces the WIT equivalent of the WITX 0.10 specification, aiming to stick as close to the WITX 0.10 version as possible.
I hope this makes the change easier to review by allowing direct comparison to the current spec.

Closes #56

This was partially done through a quick-and-dirty transpiler I wrote using the [witnext](https://crates.io/crates/witnext) and [wit-encoder](https://crates.io/crates/wit-encoder) crates.
An overview of the changes required to move from WITX to WIT, including automatic mapping vs. manual changes, can be found in the README in the WIT directory.

Regressions when compared to the WITX version:

• size: no longer backed by a usize, but a u32 instead.
• options-set-guest-buffer: not possible in WIT to provide guest memory to the host (i.e. it does not count toward the guest's memory budget anymore).
  • Workaround uses a list<u8>, but is marked as deprecated.
  • Serves as a stopgap till [caller-supplied buffers](https://wasi.dev/roadmap) arrive in future WASI 0.3.x release.
• For symmetric-state-encrypt, symmetric-state-encrypt-detached, symmetric-state-decrypt, and symmetric-state-decrypt-detached; the ability to encrypt in-place had to be removed
• version: no longer stored in one u64 but instead as a variant .
• _batch: these make use of out-buffer and in-buffer which are not defined in the WITX files.
  • This was replaced with list<u8>s instead.
  • Some of the types of the parameters also did not line up with the docs/non-batched versions, which was also changed for the WIT adaptation.

The interface is currently not tested on an implementation.
The syntax itself was quickly verified using wasm-tools component wit.
I'm more than happy to try and adapt the wasi-crypto example implementation or try and link to another crypto library, but wanted to discuss this first.
This is also the main reason why this is in draft.

If this PR lands, I propose to make this WIT specification the new source of truth.
To keep this PR manageable, I suggest handling modernizations in follow-up PRs/Zulip discussions.
Examples of this would include:

• moving functions which are linked to specific handles to methods on resources
• removing the use of close functions, using destructors instead
• making use of streams (either wasi/io streams or the ones from wasi 0.3)
• ...

I'm completely open to feedback and happy to make adjustments where needed!
My goal is to help push wasi-crypto further in the standardization process for which I hope this is a good first step :)

[jedisct1]

This is great! Thank you so much for taking this on!

One concern I have is that losing memory-hard key derivation support and in-place encryption/decryption would be significant regressions.

Beyond performance, this will have a major impact on memory usage, requiring twice as much memory as before. That may make it difficult, or even impossible, for some existing applications to move to the new interface as-is.

Given that, I think it may be better to wait for WASI 0.3 restoring caller/guest-supplied buffers before making this the new source of truth. It would also be really valuable to have at least two implementations before we treat the WIT version as authoritative.

Implementing it is often the best way to find out whether the specification is actually implementable and ergonomic, especially outside of Rust.

Maybe this could live in a separate branch for now? That would let us keep exploring the WIT/component-model version while allowing the documentation to stay aligned.

Thanks again! This feels like a really useful step forward.

[Robbe-Haegeman]

My initial suggestion was mostly to get some WIT version out the door to serve as a source of truth (which I would not want to do without it being tested on at least one implementation).

I agree with your observation that some of the regressions would be significant, though I would argue that these shortcomings are not uncommon in current production libraries.
These could be brought back in later releases, once the dependencies stabilize, preventing them from blocking the standardization process.

I'm ok though with moving it to a separate branch for now. That way I have the freedom to make the WIT API more idiomatic, without making review more difficult. Seems especially useful while I'm creating the implementation.
Are you willing to merge the current version to a branch or would you prefer to wait till at least one (or both) of the implementations is/are finished? I'm fine with it either way :)

Since having two implementations is only part of phase 3, I would prefer to focus on a wasmtime implementation currently and use that to bring the interface to a standard we're happy with.
Afterwards, we could have a look at other implementations.

Nevertheless, here is a short rundown of the runtimes I considered:

• wasmtime is a given. The WITX version already supports it and it is the reference implementation for the component model.
• wasmedge? As far as I can tell from this issue only has partial support for the component model.
• jco is the main other option which I see used for WASI proposals. Here the best option seems to be to use web-crypto under the hood. However, I assume that there are cases where it does not provide/expose everything we would need. Another option would be to forego web support entirely and instead only focus on Node.js, which would allow us to use its crypto API. I'm guessing that is probably easier to support.

A lot of the other runtimes seem to either embed wasmtime or have less widespread use, but I could have definitely missed some!

Thanks for the quick response btw!

[Robbe-Haegeman]

Hey!

I've finished the wasmtime implementation I mentioned in my previous comment.
It can currently be found in the same org as where this PR lives (idlab-discover/wasmtime-wasi-crypto), but I'm happy to move it if needed.

Similar to how I worked with the witx -> wit transpile, I focused on keeping it as close to the original (wasi-crypto-host-functions) as possible.
This also means that I kept the same things as unimplemented as that project (e.g., the secrets manager, batch impls,...).
Some more dormant bugs, like the ability to call update on signature algorithms that do not allow re-feeding after signature production e.g. pure Ed25519, were also addressed (caught by tests)

In the process of making a host implementation, a few changes to the wit spec were made:

1. The *_pull functions now take ownership of the handles. This is to allow for the "The handle is automatically closed after all the data has been consumed." statements to remain true without introducing additional state variables (think of an is_valid bool) to copy the same behavior. There is a case to be made to change these types of functions to use streams anyway, but again, that is imo a discussion to be had after the direct port.
2. A mistake slipped into the wit spec for signature-state-sign which now correctly returns a signature instead of the array-output previously.
3. symmetric-state-decrypt now takes an out-len parameter again (like it used to in the witx spec) since the implementation otherwise has no way of knowing what tag length to expect for algorithms that support multiple (e.g., AES-GCM supports tag lengths of 4–16 bytes).
4. A world.wit has been added for easy bindings.

As for the host implementation itself:
The repo is structured as a workspace with 3 main parts:

1. wasmtime-wasi-crypto: contains the actual host implementation. Most of the code is the same as the witx version, but all the direct bindings were moved to a bindings module.
2. wasmtime-host: contains both a reference integration of the wasmtime-wasi-crypto project into wasmtime and the test runner code for the next part.
3. test-components: contain a separate rust project for each interface. They are compiled to wasm32-wasip2, then each test within a project is run using separate threads, and lastly the results are combined using libtest-mimic

The need for a workspace + subprojects mostly comes from wit not really supporting something similar to what the witx version was able to do with the ctx item.

Again, I'm open to feedback :)