Add internal command to watch shard relocations progresses

[seut]

It some cases, it's very interesting to know which shards are currently relocating and also the current progress of such operations. E.g.

• in rolling restarts as shard relocations will defer the restart
• when scaling up/down
• initial shard recovery on node start up
• ...

A common query used to see shards which are currently relocating or initializing is:

select node['name'], id, recovery['stage'], recovery['size'], routing_state, state, primary, table_name, relocating_node, size / 1024^3 as size_gb, partition_ident
        from sys.shards
        where routing_state in ('RELOCATING', 'INITIALIZING')
        order by id;

At least for relocating shards, it should be possible to know the current progress. Not sure if this is feasible on initial recovery as the final sizes or maximum sequence numbers may not be known upfront.

Implementing an internal command which directly gives a user the current state without the need to know such system queries could help a lot.

[mannreis]

I've started looking at this!

Could this be a reasonable view? I
The values on the recovery['size'] column seems quite static so I've just picked percentage. And for the default query I simply inverted the where clause to show all but routing_state = "STARTED".

cr> \shards
+---------------+------------+-------------+-----------+---------------------+
| routing_state | state      | shard_count |  num_docs |             size_gb |
+---------------+------------+-------------+-----------+---------------------+
| STARTED       | STARTED    |          10 |   2000000 | 0.10163962095975876 |
| UNASSIGNED    | UNASSIGNED |           4 |         0 | 0.0                 |
| RELOCATING    | STARTED    |           1 | 100000000 | 4.6805493999272585  |
+---------------+------------+-------------+-----------+---------------------+
+-----------+----+-------+--------+---------------+------------+---------+------------+------------------------+---------+-----------------+
| name      | id | stage |      % | routing_state | state      | primary | table_name | relocating_node        | size_kb | partition_ident |
+-----------+----+-------+--------+---------------+------------+---------+------------+------------------------+---------+-----------------+
| cratedb02 |  0 | DONE  |  100.0 | RELOCATING    | STARTED    | TRUE    | big_table  | 9OZST99-RDCT9aJlAlUg3A | 4907911 |                 |
| NULL      |  0 | NULL  | NULL   | UNASSIGNED    | UNASSIGNED | FALSE   | big_table  | NULL                   |       0 |                 |
| NULL      |  0 | NULL  | NULL   | UNASSIGNED    | UNASSIGNED | FALSE   | big_table  | NULL                   |       0 |                 |
| NULL      |  0 | NULL  | NULL   | UNASSIGNED    | UNASSIGNED | FALSE   | big_table  | NULL                   |       0 |                 |
| NULL      |  0 | NULL  | NULL   | UNASSIGNED    | UNASSIGNED | FALSE   | big_table  | NULL                   |       0 |                 |
+-----------+----+-------+--------+---------------+------------+---------+------------+------------------------+---------+-----------------+

Not sure if this is feasible on initial recovery as the final sizes or maximum sequence numbers may not be known upfront.

Once I increase the number_of_replicas for a given table the new shards are UNASSIGNED and initialized with NULL in most columns. Is this the scenario you stated here?

---

For all shards this could be the view:

cr> \shards all
+---------------+------------+-------------+-----------+---------------------+
| routing_state | state      | shard_count |  num_docs |             size_gb |
+---------------+------------+-------------+-----------+---------------------+
| STARTED       | STARTED    |          10 |   2000000 | 0.10163962095975876 |
| UNASSIGNED    | UNASSIGNED |           4 |         0 | 0.0                 |
| RELOCATING    | STARTED    |           1 | 100000000 | 4.6805493999272585  |
+---------------+------------+-------------+-----------+---------------------+
+-----------+----+-------+--------+---------------+------------+---------+------------+------------------------+---------+-----------------+
| name      | id | stage |      % | routing_state | state      | primary | table_name | relocating_node        | size_kb | partition_ident |
+-----------+----+-------+--------+---------------+------------+---------+------------+------------------------+---------+-----------------+
| NULL      |  0 | NULL  | NULL   | UNASSIGNED    | UNASSIGNED | FALSE   | big_table  | NULL                   |       0 |                 |
| cratedb03 |  0 | DONE  |  100.0 | STARTED       | STARTED    | TRUE    | my_table   | NULL                   |    9909 |                 |
| NULL      |  0 | NULL  | NULL   | UNASSIGNED    | UNASSIGNED | FALSE   | big_table  | NULL                   |       0 |                 |
| NULL      |  0 | NULL  | NULL   | UNASSIGNED    | UNASSIGNED | FALSE   | big_table  | NULL                   |       0 |                 |
| NULL      |  0 | NULL  | NULL   | UNASSIGNED    | UNASSIGNED | FALSE   | big_table  | NULL                   |       0 |                 |
| cratedb02 |  0 | DONE  |  100.0 | STARTED       | STARTED    | FALSE   | my_table   | NULL                   |    9909 |                 |
| cratedb02 |  0 | DONE  |  100.0 | RELOCATING    | STARTED    | TRUE    | big_table  | 9OZST99-RDCT9aJlAlUg3A | 4907911 |                 |
| cratedb03 |  1 | DONE  |  100.0 | STARTED       | STARTED    | TRUE    | my_table   | NULL                   |    9923 |                 |
| cratedb02 |  1 | DONE  |  100.0 | STARTED       | STARTED    | FALSE   | my_table   | NULL                   |    9923 |                 |
| cratedb03 |  2 | DONE  |  100.0 | STARTED       | STARTED    | FALSE   | my_table   | NULL                   |   11179 |                 |
| cratedb01 |  2 | DONE  |  100.0 | STARTED       | STARTED    | TRUE    | my_table   | NULL                   |   11179 |                 |
| cratedb02 |  3 | DONE  |  100.0 | STARTED       | STARTED    | TRUE    | my_table   | NULL                   |   11334 |                 |
| cratedb01 |  3 | DONE  |  100.0 | STARTED       | STARTED    | FALSE   | my_table   | NULL                   |   11334 |                 |
| cratedb03 |  4 | DONE  |  100.0 | STARTED       | STARTED    | TRUE    | my_table   | NULL                   |   10940 |                 |
| cratedb01 |  4 | DONE  |  100.0 | STARTED       | STARTED    | FALSE   | my_table   | NULL                   |   10940 |                 |
+-----------+----+-------+--------+---------------+------------+---------+------------+------------------------+---------+-----------------+

[seut]

@mannreis
Thank you for looking into this.

Could this be a reasonable view?

Somehow, the query I commented was mostly only a hint on what someone uses. The main information from such a view should be the progress/percentage of shard relocations, such unassigned or started but not relocating shards aren't interesting.
At best, a user could see an ongoing progress.

The values on the recovery['size'] column seems quite static so I've just picked percentage

Using recovery['size']['percentage'] is also fine. Or what percentage do you mean?

Once I increase the number_of_replicas for a given table the new shards are UNASSIGNED and initialized with NULL in most columns. Is this the scenario you stated here?

No, this is expected as long as only 1 node exists as replicas must be allocated on a different node, until then they are unassigned. My scenario was more related to recovery on node startup, please ignore for now ;)

[mannreis]

Thanks for the clarifications! I'm still a little bit unsure about what is the intended metric to access shard relocation progress, so I resorted to the simple formula in the referenced PR:

$$\huge{100 * \left(1 - \frac{\text{Size of all shards relocating}}{\text{Total shard size}}\right)}$$

I grouped the query using table_name, as I suspect might be more useful than total progress. So for tables with no moving shards, progress shall be 100%