How to start the DPS indexer

[kiselev-sv]

Hello!
I have read https://github.com/optakt/flow-dps/blob/master/README.md and built all components
flow-dps-client
flow-dps-indexer
flow-dps-live
flow-dps-server

What are the next steps, how can I run dps-live or indexer to start from scratch?

I tried to start it, but there are errors

./flow-dps-indexer  -l debug -d /var/flow/data/protocol -t /var/flow/data/execution -c /var/flow/bootstrap/root.checkpoint -i /var/flow/data/index
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/prometheus/tsdb/wal.newSegmentBufReader(...)
        /root/gopath/pkg/mod/github.com/prometheus/tsdb@v0.7.1/wal/wal.go:716
github.com/prometheus/tsdb/wal.NewSegmentsRangeReader(0xc000591808, 0x1, 0x1, 0xc00000e420, 0x1, 0x0, 0x0)
        /root/gopath/pkg/mod/github.com/prometheus/tsdb@v0.7.1/wal/wal.go:699 +0x770
github.com/prometheus/tsdb/wal.NewSegmentsReader(...)
        /root/gopath/pkg/mod/github.com/prometheus/tsdb@v0.7.1/wal/wal.go:671
main.run(0x0)
        /root/flow-dps/cmd/flow-dps-indexer/main.go:135 +0xe53
main.main()

Where can I find root.checkpoint?
Any information needed
Thank You

[Ullaakut]

Hi @kiselev-sv !

Thanks for opening this issue, sorry I did not see it later.

Indeed, trying to run the DPS with an empty hand-made root checkpoint will not work, this is expected.

However, it is your lucky day, since last Friday the Flow team made public all of the information required to run your own DPS instance!

See the sporks.json file in the Flow repository for the information for each spork. Besides that, the currently available documentation of the Flow DPS should be enough for you to figure out easily what goes in which flag.

Keep in mind that only the current network can be indexed using the flow-dps-live binary, and only historical networks can be indexed with flow-dps-indexer.

Once you have an index folder built out and complete, you can use the Flow DPS Rosetta API or the Flow DPS Access API to expose the data through a known API standard, or you can build your own binary that reads data from the index database.

Cheers!

[Ullaakut]

Also, in case you did not already find it, the documentation that explains how to index data is available here, in our Getting Started guide.

[kiselev-sv]

I tried to index mainnet-14 spork
so downloaded root.checkpoint to /var/flow/bootstrap
node-infos.pub.json root-protocol-state-snapshot.json to /var/flow/bootstrap/public-root-information
and flow_public_mainnet14_execution_state to /var/flow/data/execution

and tried to start indexer:
~/flow-dps-bin/1.4.8 # ./dps-indexer -l debug -d /var/flow/data/protocol -t /var/flow/data/execution -c /var/flow/bootstrap/root.checkpoint -i /var/flow/data/index
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/prometheus/tsdb/wal.newSegmentBufReader(...)
/home/runner/go/pkg/mod/github.com/prometheus/tsdb@v0.7.1/wal/wal.go:716
github.com/prometheus/tsdb/wal.NewSegmentsRangeReader({0xc0003e3980, 0x1, 0x1094960})
/home/runner/go/pkg/mod/github.com/prometheus/tsdb@v0.7.1/wal/wal.go:699 +0x22d
github.com/prometheus/tsdb/wal.NewSegmentsReader(...)
/home/runner/go/pkg/mod/github.com/prometheus/tsdb@v0.7.1/wal/wal.go:671
main.run()
/home/runner/work/flow-dps/flow-dps/cmd/flow-dps-indexer/main.go:135 +0xd4f
main.main()
/home/runner/work/flow-dps/flow-dps/cmd/flow-dps-indexer/main.go:46 +0x19

what I did wrong?

[Ullaakut]

Hi @kiselev-sv ! Thanks for coming back so quickly.

If you get this error, it's that the checkpoint you are passing is not a valid checkpoint. Are you sure you downloaded the mainnet-14 checkpoint? It should be approximately 29GB of binary data, if my memory serves me well.

Could you show me what commands you ran exactly to download and extract/move the files to the folders you mention above? Thanks.

Also in your message I see another discrepancy: You used /var/flow/data/protocol for the protocol data folder but did not mention putting anything there.

[kiselev-sv]

1. wget https://storage.googleapis.com/flow-genesis-bootstrap/mainnet-14-execution/public-root-information/root.checkpoint direct to /var/flow/bootstrap/ (about 29G)
2. wget https://storage.googleapis.com/flow-genesis-bootstrap/mainnet-14-execution/public-root-information/root-protocol-state-snapshot.json direct to /var/flow/bootstrap/public-root-information
3. wget https://storage.googleapis.com/flow-genesis-bootstrap/mainnet-14-execution/public-root-information/node-infos.pub.json to /var/flow/bootstrap/public-root-information
4. wget https://storage.googleapis.com/flow_public_mainnet14_execution_state to /var/flow/data/execution/

[bluesign]

@kiselev-sv on 4h you need to download all files from there individually.

[kiselev-sv]

Yes, thank you! Downloaded with gsutil

So now I have
./dps-live -u flow-block-data -i /var/flow/index -d /var/flow/data -c /var/flow/bootstrap/public-root-information/root.checkpoint -b /var/flow/bootstrap --seed-address access-001.mainnet14.nodes.onflow.org:9000 --seed-key 28a0d9edd0de3f15866dfe4aea1560c4504fe313fc6ca3f63a63e4f98d0e295144692a58ebe7f7894349198613f65b2d960abf99ec2625e247b1c78ba5bf2eae
{"level":"info","time":"2022-02-17T16:16:31Z","message":"Flow DPS Live Server starting"}
{"level":"info","node_role":"consensus_follower","node_id":"not set","time":"2022-02-17T16:16:31Z","message":"flow consensus_follower node starting up"}
{"level":"info","start":"2022-02-17T16:16:31Z","time":"2022-02-17T16:16:31Z","message":"Flow DPS Live Indexer starting"}
{"level":"info","component":"mapper_transitions","commit":"568f4ec740fe3b5de88034cb7b1fbddb41548b068f31aebc8ae9189e429c5749","time":"2022-02-17T16:16:31Z","message":"added empty tree to forest"}
{"level":"info","node_role":"consensus_follower","node_id":"not set","block_id":"23ac4033dee62797378d7897ccccd63aacac9124eb375c5c6d82008b07732995","height":19050753,"time":"2022-02-17T16:16:34Z","message":"last finalized block"}
{"level":"info","node_role":"consensus_follower","node_id":"not set","component":"protocol-state-id-cache","blockID":"23ac4033dee62797378d7897ccccd63aacac9124eb375c5c6d82008b07732995","time":"2022-02-17T16:16:34Z","message":"updating cached identities"}
{"level":"info","node_role":"consensus_follower","node_id":"not set","module":"id providers","time":"2022-02-17T16:16:34Z","message":"module initialization complete"}
{"level":"info","node_role":"consensus_follower","node_id":"not set","module":"network middleware","time":"2022-02-17T16:16:34Z","message":"module initialization complete"}
{"level":"info","node_role":"consensus_follower","node_id":"not set","module":"mutable follower state","time":"2022-02-17T16:16:34Z","message":"module initialization complete"}
{"level":"info","node_role":"consensus_follower","node_id":"not set","module":"sync core","time":"2022-02-17T16:16:34Z","message":"module initialization complete"}
{"level":"info","node_role":"consensus_follower","node_id":"not set","module":"committee","time":"2022-02-17T16:16:35Z","message":"module initialization complete"}
{"level":"info","node_role":"consensus_follower","node_id":"not set","module":"latest header","time":"2022-02-17T16:16:35Z","message":"module initialization complete"}
{"level":"info","node_role":"consensus_follower","node_id":"not set","time":"2022-02-17T16:16:35Z","message":"network will run on address: 0.0.0.0:0"}
{"level":"info","node_role":"consensus_follower","node_id":"not set","component":"unstaked network","time":"2022-02-17T16:16:35Z","message":"component initialization complete"}
{"level":"info","node_role":"consensus_follower","node_id":"not set","time":"2022-02-17T16:16:35Z","message":"Updating protocol state node addresses"}
{"level":"info","node_role":"consensus_follower","node_id":"not set","component":"unstaked network","time":"2022-02-17T16:16:35Z","message":"component startup complete"}
{"level":"info","node_role":"consensus_follower","node_id":"not set","component":"upstream connector","time":"2022-02-17T16:16:35Z","message":"component initialization complete"}
{"level":"error","node_role":"consensus_follower","node_id":"not set","bootstap_node":"access-0000000000000000000000000000000000000000000000000000000000000000@access-001.mainnet14.nodes.onflow.org:9000=0","error":"failed to dial QmUyQoWbG4Nb4FZwHybKPfdDw9DzJ7JC41s1cEjTf9rSXo:\n * [/ip4/23.236.58.237/tcp/9000] failed to negotiate security protocol: message did not have trailing newline","time":"2022-02-17T16:16:35Z","message":"failed to connect"}
{"level":"fatal","node_role":"consensus_follower","node_id":"not set","error":"1 error occurred:\n\t* failed to dial QmUyQoWbG4Nb4FZwHybKPfdDw9DzJ7JC41s1cEjTf9rSXo:\n * [/ip4/23.236.58.237/tcp/9000] failed to negotiate security protocol: message did not have trailing newline\n\n","time":"2022-02-17T16:16:35Z","message":"Failed to connect to a bootstrap node. Please ensure the network address and public key of the bootstrap access node are correct and that the node is running and reachable."}

Where can I find any seeds?
Is it a main problem?

[kiselev-sv]

All info at https://github.com/onflow/flow/blob/master/sporks.json

with latest root-protocol-state-snapshot.json for mainnet16 and dps 1.4.8

{"level":"error","error":"could not decode protocol snapshot: json: cannot unmarshal string into Go struct field Seal.LatestSeal.FinalState of type flow.StateCommitment","time":"2022-03-03T08:17:47Z","message":"could not initialize protocol state"}
Can anybody help?

[kiselev-sv]

Downloaded and extracted protocol, tried to start indexer on mainnet-15

root@ph243 /nodes-data/flow/flow-dps-bin/1.4.8 # ./dps-indexer -l debug -t /nodes-data/flow/data/execution -d /nodes-data/flow/data/protocol -c /nodes-data/flow/bootstrap/public-root-information/root.checkpoint
{"level":"info","start":"2022-03-06T03:53:42Z","time":"2022-03-06T03:53:42Z","message":"Flow DPS Indexer starting"}
{"level":"info","component":"mapper_transitions","commit":"568f4ec740fe3b5de88034cb7b1fbddb41548b068f31aebc8ae9189e429c5749","time":"2022-03-06T03:53:42Z","message":"added empty tree to forest"}
Killed

[1674969.073514] Out of memory: Kill process 10137 (dps-indexer) score 925 or sacrifice child
[1674969.073900] Killed process 10137 (dps-indexer) total-vm:196580572kB, anon-rss:182859944kB, file-rss:0kB, shmem-rss:0kB

I have
MemTotal: 197945536 kB
MemFree: 182456272 kB
MemAvailable: 184011180 kB
Buffers: 1337292 kB
Cached: 1376516 kB
SwapCached: 0 kB
Active: 11389932 kB
Inactive: 1199872 kB
Active(anon): 10000848 kB
Inactive(anon): 1176 kB
Active(file): 1389084 kB
Inactive(file): 1198696 kB
Unevictable: 269308 kB
Mlocked: 269308 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 47000 kB
Writeback: 0 kB
AnonPages: 10144964 kB
Mapped: 402216 kB
Shmem: 1736 kB
Slab: 1743600 kB
SReclaimable: 739352 kB
SUnreclaim: 1004248 kB
KernelStack: 24064 kB
PageTables: 255520 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 98972768 kB
Committed_AS: 15436376 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
Percpu: 112128 kB
:

[1674973.803671] oom_reaper: reaped process 10137 (dps-indexer), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

How much memory it needs?

All info at https://github.com/onflow/flow/blob/master/sporks.json

with latest root-protocol-state-snapshot.json for mainnet16 and dps 1.4.8

{"level":"error","error":"could not decode protocol snapshot: json: cannot unmarshal string into Go struct field Seal.LatestSeal.FinalState of type flow.StateCommitment","time":"2022-03-03T08:17:47Z","message":"could not initialize protocol state"} Can anybody help?

@kiselev-sv Have you gotten dps-live working with mainnet-16? this is as far as i've got https://github.com/dapperlabs/dps-rosetta-docker/pull/8

[kiselev-sv]

no, I am not

[admincheg]

@chaen-foundry as I see - new protocol state has new fields and formats, which not compatible with old version of flow-go. You can compare formats with mainnet-14, but after fixing live didn't start properly. Fresh run with empty database ends with segmentation fault (strange defer behavior, it closes resources before accessing to they) and next runs just silently shutdown. If you remove data and index directories - segmentation fault returns. If you comment defers in code of live module - first run doesn't produce SIGSEGV, but it's still not working.