Replace full `NoteMetadata` with a fixed-size header in `NoteSyncRecord`

[SantiagoPittella]

With the recent addition of note attachments, NoteMetadata is no longer bounded. It can be up to 8-16 KB per note. Since NoteSyncRecord is embedded in sync responses that return summaries of many notes, this makes response sizes unbounded.

Replace the full NoteMetadata in NoteSyncRecord with a fixed-size note metadata header (64 bytes: metadata header + attachment commitment word). Users who need full note details including attachments can use the GetNotesById endpoint.

[bobbinth]

I think there are several related issues here:

First, we have several slightly different protobuf messages: NoteSyncRecord, NoteInclusionInBlockProof, CommittedNote. I'd love to find a way to normalize these. For example, maybe we need CommittedNote and CommittedNoteInfo (replacing NoteSyncRecord), and both should use NoteInclusionInBlockProof.

Second, NoteMetadata now contains all attachment data. It is not clear what's the best solution is here but we need to refactor this somehow so that we can:

• Have a small, fixed-size object that contains sender, tag, note_type, and some information about the attachments.
• A way to get full attachments for a note (either as a part of full metadata object - or maybe somehow separately).

Maybe, the Note message could be refactored to look something like:

message Note {
    NoteMetadata metadata = 1;
    optional bytes details = 2;
    optional NoteAttachment attachment = 3;
}

It would be good to align these changes with miden-base though. cc @PhilippGackstatter in case he has some ideas.

---

Thinking about the future, we will probably transition to having multiple attachments per note. Then, the Note message would look something like:

message Note {
    NoteMetadata metadata = 1;
    optional bytes details = 2;
    repeated NoteAttachment attachments = 3;
}

[PhilippGackstatter]

Have a small, fixed-size object that contains sender, tag, note_type, and some information about the attachments.

This would be NoteMetadataHeader. It is fixed-size and consists of the full metadata + attachment info. That exists in miden-base as a private type, but we can make it public if it helps. If we make it public, then we will probably need something like NoteMetadata::from_header(NoteMetadataHeader, NoteAttachment), since these are returned by different endpoints.

message Note {
NoteMetadata metadata = 1;
optional bytes details = 2;
optional NoteAttachment attachment = 3;
}

We can double-check if the metadata needs to be included here, since the full metadata should be provided by NoteMetadataHeader already.

An alternative would be to just include NoteMetadata in full here (which already contains the full NoteAttachment), if that makes it easier at the cost of a few bytes of overhead. I'd probably try including just the NoteAttachment here and adding the from_header constructor.

[bobbinth]

Yes, I think using NoteMetadataHeader in NoteSyncRecord (and in other similar messages, if relevant) is probably the right way to go now.

Once we enable support for multiple attachments per note, things may look a bit different - but we can cross this bridge when we get there. @PhilippGackstatter - do we already have an issue for multiple note attachments? (I remember we discussed them somewhere, but not sure where it was).

[PhilippGackstatter]

We discussed it internally, but opened an issue now: 0xMiden/protocol#2555.

[SantiagoPittella]

The needed changes to start to work on this are already merged in the protocol repository. Currently, to start addressing the node part we still need to update the node to the latest changes of the protocol, including the miden-vm and crypto updates.

[SantiagoPittella]

@drahnr Currently, I'm updating this repository to use miden-vm 0.22, miden-crypto 0.19 and the latest next branch from the protocol repository in order to be able to use the changes that I need from the protocol to address this, while doing this I noticed that StorageError is no longer exported from miden-crypto, and we were using that in the rocksdb crate.

Should I:

1. Use LargeSmtError instead? It wraps StorageError via the Storage(#[from] StorageError) variant, and is publicly re-exported.
2. Make a PR re-exporting it again. It would need to be added to the re-export list in miden-crypto's smt/mod.rs.

[drahnr]

@SantiagoPittella v0.19.x should always include the StorageError, I have yet to carry over 0xMiden/crypto#850 to next, is this your issue?

[SantiagoPittella]

Yes, since protocol is is already using 0.22 of the crypto repository.