SetStorageTargetInfo deserialization fails for "Invalid value 224 for conversion into enum TargetConsistencyState"

[iamjoemccormick]

On a system with meta and storage buddy mirroring (probably relevant) and quotas (probably less relevant) if I start my management after it was stopped for sometime (for example after rebuilding), I typically see these errors logged:

[2024-12-03T18:23:16Z ERROR] Closed stream from 127.0.0.1:39064: target::SetStorageTargetInfo (2099) from 127.0.0.1:39064: BeeMsg body deserialization failed: Invalid value 224 for conversion into enum TargetConsistencyState
[2024-12-03T18:23:16Z ERROR] Closed stream from 127.0.0.1:39080: target::SetStorageTargetInfo (2099) from 127.0.0.1:39080: BeeMsg body deserialization failed: Invalid value 224 for conversion into enum TargetConsistencyState
[2024-12-03T18:23:27Z ERROR] Closed stream from 127.0.0.1:38210: target::SetStorageTargetInfo (2099) from 127.0.0.1:38210: BeeMsg body deserialization failed: Invalid value 224 for conversion into enum TargetConsistencyState
[2024-12-03T18:23:27Z ERROR] Closed stream from 127.0.0.1:38216: target::SetStorageTargetInfo (2099) from 127.0.0.1:38216: BeeMsg body deserialization failed: Invalid value 224 for conversion into enum TargetConsistencyState

At the same time at debug level one of my metas logs:

(2) 18:23:10.316062 XNodeSync [MessagingTk.cpp:447] >> Unable to connect, is the node offline? node: beegfs-mgmtd management [ID: 1]; Message type: GetStatesAndBuddyGroups (1053)                                                                                                                                            
(4) 18:23:10.316433 XNodeSync [TargetStateStore.h:60] >> Setting all states. New state: Probably-offline; Called from: meta/build/beegfs-meta(_ZN15InternodeSyncer41downloadAndSyncTargetStatesAndBuddyGroupsEv+0x27c) [0xaaaaac2b5d90]; meta/build/beegfs-meta(_ZN15InternodeSyncer8syncLoopEv+0x420) [0xaaaaac2bb748]; meta/
build/beegfs-meta(_ZN15InternodeSyncer3runEv+0x1c) [0xaaaaac2bbbb4].                                                                                                                                                                                                                                                          
(4) 18:23:16.311707 XNodeSync [NodeConn (acquire stream)] >> Establishing new TCP connection to: beegfs-mgmtd@127.0.0.1:8008                                                                                                                                                                                                  
(4) 18:23:16.311869 XNodeSync [StandardSocket.cpp:142] >> Connect StandardSocket socket: 0xaaaadfa12a90; addr: 127.0.0.1:8008; bindIP: 0.0.0.0                                                                                                                                                                                
(3) 18:23:16.312104 XNodeSync [NodeConn (acquire stream)] >> Connected: beegfs-mgmtd@127.0.0.1:8008 (protocol: TCP)                                                                                                                                                                                                           
(0) 18:23:16.313482 XNodeSync [Messaging (RPC)] >> Communication error: Soft disconnect from 127.0.0.1:8008; Peer: beegfs-mgmtd management [ID: 1]. (Message type: SetStorageTargetInfo (2099))                                                                                                                               
(4) 18:23:16.313584 XNodeSync [NodeConn (invalidate stream)] >> Disconnected: beegfs-mgmtd@127.0.0.1:8008                                                                                                                                                                                                                     
(2) 18:23:16.313612 XNodeSync [MessagingTk.cpp:25] >> Retrying communication. peer: beegfs-mgmtd management [ID: 1]; message type: SetStorageTargetInfo (2099)                                                                                                                                                                
(4) 18:23:16.313620 XNodeSync [NodeConn (acquire stream)] >> Establishing new TCP connection to: beegfs-mgmtd@127.0.0.1:8008                                                                                                                                                                                                  
(4) 18:23:16.313633 XNodeSync [StandardSocket.cpp:142] >> Connect StandardSocket socket: 0xaaaadfa12a90; addr: 127.0.0.1:8008; bindIP: 0.0.0.0                                                                                                                                                                                
(3) 18:23:16.313690 XNodeSync [NodeConn (acquire stream)] >> Connected: beegfs-mgmtd@127.0.0.1:8008 (protocol: TCP)                                                                                                                                                                                                           
(0) 18:23:16.314081 XNodeSync [Messaging (RPC)] >> Communication error: Soft disconnect from 127.0.0.1:8008; Peer: beegfs-mgmtd management [ID: 1]. (Message type: SetStorageTargetInfo (2099))                                                                                                                               
(4) 18:23:16.314142 XNodeSync [NodeConn (invalidate stream)] >> Disconnected: beegfs-mgmtd@127.0.0.1:8008    

Because I have buddy mirroring enabled (meta and storage mirrors) my initial theory was that something with that TargetConsistencyState enum was not setup correctly to deserialize states other than Good (I'm guessing my targets are in needs resync). But as best I can tell all the serialization for SetStorageTargetInfo, TargetInfo and TargetConsistencyState is correct so I'm not sure where that 224 value is coming from.

Also entirely possible this is a longstanding bug in the meta/storage services.

Perhaps notable we have only observed this on aarch64 (Linux beemac 5.15.0-125-generic ThinkParQ/beegfs-rs#135-Ubuntu SMP Fri Sep 27 13:56:10 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux). If we get additional reports we should check if the issue consistently only happens on arm systems.

[rustybee42]

I can't reproduce this (using one meta and one storage group + quota). I also checked the message, serialization looks fine and matches up.

[iamjoemccormick]

@rustybee42,

Attached are trace logs that catch the issue after applying the patch you sent me earlier. The logs go until about 50s after startup so they include some successful handling of SetStorageTargetInfo as well.

20241205-mgmtd-trace-logs.txt

[rustybee42]

From slack:

From the log it's pretty clear that serialization is fine - the value in this uint8 field is just set to 224. Which points to memory corruption / uninitialized value on the meta side, but I can't find it.

Since it's only happening to you, it might be an architecture related thing that only occurs on ARM. Maybe it's converting the enum to a uint8_t for serialization - although that shouldn't be a problem since the enum only has 3 variants (and I cannot make up how 0,1 or 2 end up as 224).

[iamjoemccormick]

Note with the latest mgmtd I now see this error:

[2025-03-12T17:25:33Z ERROR] Closed stream from 127.0.0.1:36184: target::SetStorageTargetInfo (2099) from 127.0.0.1:36184: BeeMsg body deserialization failed: Invalid value 8 for conversion into enum TargetConsistencyState
[2025-03-12T17:25:33Z ERROR] Closed stream from 127.0.0.1:36192: target::SetStorageTargetInfo (2099) from 127.0.0.1:36192: BeeMsg body deserialization failed: Invalid value 8 for conversion into enum TargetConsistencyState

[rustybee42]

I tried to reproduce again (mirrored system, stop management, wait a couple of minutes, restart), nothing.

I still think it's a bug on the meta that might only happen on aarch64. To confirm that it is not the management, can you log on the meta what is actually serialized and sent out for this field? Or alternatively, dump the network traffic between this meta and management so we can see the serialized value as it comes in.

[iamjoemccormick]

As suspected this was a benign bug on the meta where the local target consistency state was temporarily set to an invalid value when the mgmtd was unreachable. The issue only occurred if the meta happened to try to resend a SetStorageTargetInfo message before updating meta states and buddy groups after reconnecting to mgmtd.