HCD-241: DSE 6.8/9 upgradeability by szymon-miezal · Pull Request #2240 · datastax/cassandra

szymon-miezal · 2026-02-19T13:42:23Z

This PR addresses critical compatibility issues discovered during DSE to HCD (Hyper-Converged Database) upgrade scenarios. The changes focus on ensuring smooth interoperability between DSE 6.8+ nodes and HCD nodes during mixed-version cluster operations.

What Problems Do These Changes Fix?

1. Enhanced Debugging Capabilities for Upgrade Scenarios

Problem: When upgrading from DSE to HCD, troubleshooting connection and gossip issues was difficult due to insufficient logging.

Solution: Added comprehensive logging throughout the handshake and gossip processes, including:

Port selection logic for DSE → HCD upgrades
Messaging version negotiation during handshake
Protocol proposals from DSE peers
Gossip state values (both loaded and saved)
Connection initialization events

Why it matters: This gives operators visibility into what's happening during the upgrade process, making it much easier to diagnose and resolve issues in production environments.

2. Gossip Deserialization Crash Prevention

Problem: Nodes would crash with ArrayIndexOutOfBoundsException when processing gossip messages from pre-4.0 nodes. This happened because the code assumed certain delimiters would always be present in address and status values, but older nodes sent data in different formats. Both INTERNAL_ADDRESS_AND_PORT and NATIVE_ADDRESS_AND_PORT could arrive without port delimiters (e.g., "10.0.0.1" instead of "10.0.0.1:7000"), and STATUS_WITH_PORT could arrive without additional port information.

Solution: Added defensive length checks after splitting gossip values to gracefully handle cases where expected delimiters are missing:

Internal and native IP addresses without ports (e.g., "10.0.0.1" vs "10.0.0.1:7000")
Status values without additional port information

Why it matters: Prevents cluster instability and crashes during mixed-version operations.

3. DSE 6.8+ PING Request Compatibility

Problem: During startup connectivity checks, HCD nodes were attempting to send PING requests to DSE 6.8+ peers, but these versions don't support PING_REQ messages (similar to DSE 6.x and Cassandra 3.x). This resulted in noisy error logs during cluster startup.

Solution: Extended the existing version detection logic to recognize and skip PING requests for DSE 6.8+ peers during the startup connectivity check phase.

Why it matters: Eliminates unnecessary error logs when HCD nodes join or restart in a mixed DSE/HCD environment.

4. User-Defined Type Preservation During Schema Migration

Problem: When keyspace schemas were updated during migration, User-Defined Types (UDTs) from the previous schema could be lost if they weren't explicitly present in the new schema. This caused failures when inherited tables depended on these types.

Solution: Modified the schema transformation logic to preserve UDTs from the previous schema when they don't exist in the new schema, ensuring dependent tables continue to function correctly.

Why it matters: Prevents data model corruption and application failures during schema migrations, particularly important when dealing with complex schemas that use inheritance and UDTs.

Overall Impact

These changes collectively improve the robustness and reliability of DSE to HCD upgrades by:

Making issues easier to diagnose through better logging
Preventing crashes from malformed gossip data
Ensuring protocol compatibility across versions
Preserving schema integrity during migrations

Latest test run: http://10.169.74.112:8081/job/ds-cassandra-build/2117/
Baseline: http://10.169.74.112:8081/job/ds-cassandra-build/10/

TODO: attache CNDB PR

github-actions · 2026-02-19T13:42:41Z

szymon-miezal · 2026-02-19T13:52:17Z

src/java/org/apache/cassandra/net/StartupClusterConnectivityChecker.java

        Message<PingRequest> large = Message.out(PING_REQ, PingRequest.forLarge);
        for (InetAddressAndPort peer : peers)
        {
+            logger.info("Peer {} has version {}", peer, MessagingService.instance().versions.get(peer));


That seems like something worth keeping.

szymon-miezal · 2026-02-20T12:05:57Z

TODO before merging: alter the commit message of cdc2d1a

szymon-miezal · 2026-02-20T15:19:58Z

src/java/org/apache/cassandra/net/OutboundConnectionInitiator.java

                int peerMessagingVersion = msg.maxMessagingVersion;
                logger.trace("received second handshake message from peer {}, msg = {}", settings.connectTo, msg);

+                logger.debug("Summary of messaing versions while connecting to {} " +


- Add logging to debug the port selecton for DSE -> HCD upgrade - Add logging to debug the messaging version selection during handshake - Logging in outbound connection at handshake success - Add the node address to logs - Log the protocol proposed by DSE peer - Log the value gossip fails on - Log the loaded/saved gossip values - Logs for Initiate, Accept and ConfirmOutboundPre40 - Logs for InboundConnectionInitiator

Adds tests validating the fix for ArrayIndexOutOfBoundsException that occurred when deserializing gossip state from pre-4.0 nodes containing address/port values without expected delimiters. The bug manifested when filterOutgoingState() attempted to split values like "10.0.0.1" or "NORMAL" and blindly access array indices [1] or [0] that didn't exist, causing crashes during gossip message processing in mixed-version clusters. The fix adds length checks after splitting to gracefully handle: - IP addresses without ports (e.g., "10.0.0.1" vs "10.0.0.1:7000") - Status values without tokens (e.g., "NORMAL" vs "NORMAL,10.0.0.1:7000")

…ivity check DSE 6.8 and later versions don't support PING_REQ messages, similar to DSE 6.x and Cassandra 3.x. This change extends the existing logic to detect and skip PING requests for DSE 6.8+ peers during the startup cluster connectivity check phase.

When updating keyspace schemas, UDTs from the previous schema are now preserved if they don't exist in the new schema. This prevents issues where inherited tables depend on types that would otherwise be lost during schema transformations.

sonarqubecloud · 2026-02-20T19:06:49Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
92.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

cassci-bot · 2026-02-20T19:13:08Z

❌ Build ds-cassandra-pr-gate/PR-2240 rejected by Butler

2 regressions found
See build details here

Found 2 new test failures

Test	Explanation	Runs	Upstream
o.a.c.index.sai.cql.VectorCompaction100dTest.testCompactionWithEnoughRowsForPQAndDeleteARow[dc true]	REGRESSION	🔴⚪	0 / 11
o.a.c.index.sai.cql.VectorSiftSmallTest.testSiftSmall[eb false]	REGRESSION	🔴⚪	0 / 11

Found 4 known test failures

szymon-miezal commented Feb 19, 2026

View reviewed changes

szymon-miezal force-pushed the hcd-241 branch 4 times, most recently from a6a0773 to 17c4380 Compare February 20, 2026 08:15

szymon-miezal mentioned this pull request Feb 20, 2026

HCD-241: Add a dedicated DSE 6.9 upgrade test datastax/cassandra-dtest#90

Open

szymon-miezal changed the title ~~HCD-241: DSE 6.8/9 upgradeability (WIP)~~ HCD-241: DSE 6.8/9 upgradeability Feb 20, 2026

szymon-miezal self-assigned this Feb 20, 2026

szymon-miezal force-pushed the hcd-241 branch from 17c4380 to 7c70156 Compare February 20, 2026 13:50

szymon-miezal commented Feb 20, 2026

View reviewed changes

szymon-miezal force-pushed the hcd-241 branch from 7c70156 to be40074 Compare February 20, 2026 17:36

szymon-miezal added 4 commits February 20, 2026 18:55

szymon-miezal force-pushed the hcd-241 branch from be40074 to 4cab315 Compare February 20, 2026 18:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HCD-241: DSE 6.8/9 upgradeability#2240

HCD-241: DSE 6.8/9 upgradeability#2240
szymon-miezal wants to merge 4 commits intomainfrom
hcd-241

szymon-miezal commented Feb 19, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Feb 19, 2026 •

edited by szymon-miezal

Loading

Uh oh!

szymon-miezal Feb 19, 2026

Uh oh!

szymon-miezal commented Feb 20, 2026

Uh oh!

szymon-miezal Feb 20, 2026

Uh oh!

sonarqubecloud bot commented Feb 20, 2026

Uh oh!

cassci-bot commented Feb 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

szymon-miezal commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problems Do These Changes Fix?

1. Enhanced Debugging Capabilities for Upgrade Scenarios

2. Gossip Deserialization Crash Prevention

3. DSE 6.8+ PING Request Compatibility

4. User-Defined Type Preservation During Schema Migration

Overall Impact

Uh oh!

github-actions bot commented Feb 19, 2026 • edited by szymon-miezal Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist before you submit for review

Uh oh!

szymon-miezal Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

szymon-miezal commented Feb 20, 2026

Uh oh!

szymon-miezal Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Feb 20, 2026

Quality Gate passed

Uh oh!

cassci-bot commented Feb 20, 2026

❌ Build ds-cassandra-pr-gate/PR-2240 rejected by Butler

Found 2 new test failures

Found 4 known test failures

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

szymon-miezal commented Feb 19, 2026 •

edited

Loading

github-actions bot commented Feb 19, 2026 •

edited by szymon-miezal

Loading