Skip to content

Add streaming download support #699

@BigLep

Description

@BigLep

The SDK currently supports streaming uploads but not streaming downloads. The download path accumulates the full response in memory (via the accumulating reader in synapse-core), which means pieces larger than available process memory can't be downloaded.

Background

There used to be a "Retriever" architecture that returned a `fetch` `Response` you could stream from, but that was removed and replaced with the current accumulating approach. This needs to be revisited.

Proposed API

The main complexity is CommP validation during download. Two approaches have been discussed:

  1. Error on last chunk — the stream errors if CommP validation fails at the end (preferred)
  2. Post-stream validation callback — a promise that resolves after stream completion to indicate whether CommP matched (used in FilBeam)

Consensus is that erroring on the last chunk is more idiomatic with WebAPI Streams, with the ability to opt out of validation for cases where it isn't needed.

Related

A merkle inclusion proof API or header for byte range piece retrievals from Curio is also a TODO (client and server) if we're going to encourage or demonstrating byte range retrievals.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    📌 Triage

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions