gRPC: Implement FollowTip (UTxO RPC SyncService) in cardano-rpc

[carbolymer]

Summary

Implement the FollowTip RPC method from the UTxO RPC SyncService spec, providing a server-streaming gRPC endpoint that delivers blocks as they are applied or rolled back on the chain.
This is the gRPC equivalent of the Ouroboros ChainSync mini-protocol and the most architecturally significant method in the UTxO RPC spec.

Motivation

Replacing ChainSync for remote clients

Chain-following clients (Kupo, Scrolls, Oura, custom indexers) currently connect to the cardano-node Unix socket and run the ChainSync mini-protocol directly.
This requires:

• Local filesystem access to the node socket
• A Haskell (or compatible) implementation of the Ouroboros N2C handshake and ChainSync state machine
• Version negotiation with the node

FollowTip exposes the same block stream over gRPC, which:

• Works over TCP (no filesystem coupling)
• Uses standard gRPC client libraries available in every language
• Handles versioning at the proto/gRPC level rather than the Ouroboros protocol level

Kupo's primary data path

Kupo's entire indexing pipeline runs on a single ChainSync connection.
It uses MsgFindIntersect to establish a starting point, then streams blocks via MsgRequestNext with adaptive pipelining.
FollowTip maps directly onto this: intersect for the starting point, apply for roll-forward, undo for roll-backward.

Background

What the spec requires

FollowTipRequest takes:

• repeated BlockRef intersect - candidate block references for finding a common point (analogous to Ouroboros MsgFindIntersect)
• Optional FieldMask - to select which parsed fields to materialise in the response

FollowTipResponse is a server-streaming response where each message carries:

• oneof action:
  • apply (AnyChainBlock) - a new block to apply (= Ouroboros RollForward)
  • undo (AnyChainBlock) - a block to undo (= Ouroboros RollBackward). Notably, the full block data is included, so the client does not need to maintain its own block cache for rollback processing
  • reset (BlockRef) - server-initiated reset to a point. No direct Ouroboros analogue; used when no intersect matched or on reinitialisation
• tip (BlockRef) - current chain tip for lag measurement

Key differences from Ouroboros ChainSync

Aspect	ChainSync	FollowTip
Transport	Unix socket (N2C)	TCP/gRPC
Rollback data	Only the rollback point	Full block included in undo
No-intersect	MsgIntersectNotFound	reset action
Pipelining	Client-driven (MsgRequestNextPipelined)	Server-driven (gRPC stream backpressure)
Block format	CBOR	Proto (AnyChainBlock with native_bytes and/or parsed cardano.Block)

Why this depends on node kernel access

The existing N2C path is insufficient for a production-quality FollowTip:

• N2C ChainSync is per-connection: each gRPC client would require a separate N2C socket connection to the node, each running its own ChainSync follower. This does not scale.
• No shared follower: the node's ChainSync server creates a fresh follower per N2C connection. There is no way to multiplex multiple gRPC clients onto a single follower.
• In-process access: ADR-019's NodeAccess record can expose a ChainDB.newFollower callback, letting cardano-rpc create followers directly against the consensus ChainDB. Multiple gRPC clients can each get their own follower without socket overhead.

The in-process path also enables zero-copy block delivery: ChainDB.getBlockComponent GetRawBlock returns the raw CBOR directly from the ImmutableDB/VolatileDB without serialisation.

Proposed approach

NodeAccess extension

Extend the NodeAccess record (ADR-019) with:

• A follower creation callback (wrapping ChainDB.newFollower)
• Follower operations: read next (with rollback support), find intersection

The exact API depends on how ChainDB.Follower is surfaced through the consensus interface.

Follower lifecycle

Each FollowTip gRPC stream creates a new ChainDB follower.
On stream setup:

1. Create a follower via NodeAccess.
2. Find intersection using the client's repeated BlockRef intersect.
3. If no intersection found, send a reset action to genesis (or the oldest available point).

On each iteration:

1. Read the next change from the follower (blocking if at tip).
2. Translate to apply, undo, or reset action.
3. Send the FollowTipResponse on the gRPC stream.

On stream teardown (client disconnect or error):

1. Close the follower to free resources.

Block encoding

For the initial implementation, populate native_bytes (raw block CBOR) only.
Populating the parsed cardano.Block oneof requires mapping the full block structure to proto, which is a large effort and can follow separately.
Clients that need parsed data can deserialise the CBOR themselves.

Backpressure

gRPC server-streaming has built-in flow control via HTTP/2 window management.
The follower read loop should respect this: if the client is slow to consume, the server blocks on the gRPC send rather than buffering unboundedly.

Server wiring

Register FollowTip in the SyncService alongside FetchBlock and ReadTip.

Implementation notes

Scalability considerations

Each follower maintains a pointer into the ChainDB.
The node already supports multiple followers (one per N2C ChainSync connection), so this is not a new resource type.
However, gRPC makes it easier to open many connections from remote clients.
Consider:

• A configurable maximum number of concurrent followers
• Logging/metrics for active follower count
• Idle timeout for followers that stop consuming

Parsed block support (future)

The FollowTipResponse can carry a parsed cardano.Block in the AnyChainBlock.
This requires mapping the full Cardano block structure (header, body, witnesses, auxiliary data) to the proto model.
This is significant additional work and should be a follow-up issue, not a blocker for the initial FollowTip implementation.

FieldMask

FollowTipRequest includes a FieldMask.
As with other methods, this can be ignored initially (always return native_bytes).
Once parsed block support exists, FieldMask becomes valuable for letting clients request only headers, or only transaction hashes, reducing bandwidth.

Acceptance criteria

• NodeAccess (ADR-019) exposes follower creation and intersection-finding.
• followTipMethod handler implemented as a gRPC server-streaming endpoint.
• Intersection logic: client sends BlockRef list, server finds most recent common point.
• Stream delivers apply (with raw block CBOR), undo (with raw block CBOR), and reset actions.
• Each response includes the current tip.
• Follower is cleaned up on client disconnect.
• Integration test in cardano-testnet: connect via FollowTip, submit a transaction, verify the block containing it arrives as an apply action with valid CBOR.
• cardano-rpc/README.md updated to mark FollowTip as supported in the coverage table.

Out of scope

• Parsed cardano.Block population (native_bytes only for initial implementation).
• FieldMask support.
• Concurrent follower limits (can be added as a follow-up).
• DumpHistory (separate issue).

Dependencies

• ADR-019: Node Kernel Access for cardano-rpc and its implementation.
• gRPC: Implement FetchBlock (UTxO RPC SyncService) in cardano-rpc #1216 (FetchBlock) for the sync proto file and NodeAccess block-lookup foundation.

References

• UTxO RPC SyncService spec: https://utxorpc.org/sync/spec/
• Upstream proto: utxorpc/spec repo, proto/utxorpc/v1beta/sync/sync.proto
• Ouroboros ChainSync mini-protocol: network-protocols package
• Related: gRPC: Implement FetchBlock (UTxO RPC SyncService) in cardano-rpc #1216 (FetchBlock), gRPC: Implement ReadGenesis (UTxO RPC QueryService) in cardano-rpc #1217 (ReadGenesis)