Allow trailing characters when deserializing

[MorganR]

Apologies if I'm simply doing this wrong.

I'm trying to implement a sort of streaming deserializer, but running into a limitations with the public Deserializer API that seem to make this impossible.

What I'm doing at a high level:

I want to deserialize several small structs out of a long data stream. Imagine there is 16kB of data, which contains several 1kB structs, and I can throw away a lot of data so that I end up with individual structs that are only 128 bytes each. If I have the full 16kB in memory, then I can use serde-json-core::from_slice to parse this perfectly fine. However, this requires me to have a large, mostly unused buffer.

Instead, I want to have a 4kB buffer that I can read into, parsing out one struct at a time, and throwing away data as I go. This would avoid allocating the full 16kB. This all seems fairly doable if I could do something like this:

// Fill the buffer.
let chunk: &[u8] = buffer.read_chunk().await?;
// Deserialize the next value from the buffer.
let (value, n): (T, usize) = serde_json_core::from_slice(chunk)?;
// Tell the buffer it can throw away `n` bytes.
buffer.consume(n);

However, this doesn't work because from_slice returns Err(Error::TrailingCharacters) if chunk contains more data than a single value.

I can call serde::de::Deserialize() like below, but I can't see any way to get the value of n (the number of bytes read when deserializing the value).

let mut deserializer = serde_json_core::de::Deserializer::new(chunk, None);
let value: T = de::Deserialize::deserialize(&mut deserializer)?;
let n = 0 // ???

What I need:

Either:

1. Allow trailing characters in serde_json_core::from_slice, making use of the existing return parameter to indicate the number of bytes read.
2. Expose a public index() method on the Deserializer to get the value of self.index

Would either of those be amenable? Or have I missed some existing way to do this?

[zargony]

I've run into basically the same issue. I'm parsing a large JSON response from a https request. It's one large JSON object with hundreds of keys. Since it doesn't fit into memory, it's being read chunk by chunk, manually parsing begin {, separators : and , and ending }. Keys and values could be parsed by serde_json_core, if there would be a way to ignore trailing characters and get the number of bytes consumed.

serde_json's StreamDeserializer offers a byte_offset() method which basically would return self.index in serde_json_core. This sounds like a simple and powerful addition and it wouldn't change behavior or break any existing interface in serde_json_core.