DTLS handshake fails for most clients in multi-client str0m-to-str0m setup

[Teyk0o]

Environment

• str0m version: latest (both sides)
• Setup: str0m-based SFU server + str0m-based load test client (str0m-stress), both using Rtc::builder().build()
• Network: client runs on a remote machine, SFU in Docker on a VPS (2 cores), UDP ports 16384-16385 mapped through Docker
• Scale: 2-30+ concurrent clients connecting to the same SFU

Symptom

ICE completes successfully for all clients (Checking → Connected → Completed), but DTLS only completes for ~20-30% of connections. The remaining clients never get past the DTLS handshake, no data channel opens, no media flows, and the connection eventually times out.

Observed behavior

Working client (minority):

ICE state -> Connected
ICE state -> Completed
DTLS setup is: true
ClientHello: DTLS version=DTLS1_2, cookie_len=0, offering 3 cipher suites
[ServerHello received, handshake completes]
ChannelOpen("data")
MediaData received ✓

Failing client (majority):

ICE state -> Connected
ICE state -> Completed
DTLS setup is: true
ClientHello: DTLS version=DTLS1_2, cookie_len=0, offering 3 cipher suites
[No ServerHello — nothing comes back]
Flight timeout in: 1.189s
[Retransmit ClientHello]
Flight timeout in: 0.802s
[Retransmit again, eventually gives up after connect timeout (40s)]

Key data points

1. ICE works fine for all clients — STUN binding succeeds, candidate pairs are validated, consent checks flow at ~1 pps. UDP connectivity is not the issue.
2. The server receives the DTLS ClientHello — our packet dispatch logs confirm the UDP packet is received on the correct socket, dispatched to the correct Rtc instance via handle_input(Input::Receive(...)). No packets are dropped.
3. The server does not produce a DTLS response — after handle_input processes the ClientHello, poll_output() does not return a Transmit containing a ServerHello/HelloVerifyRequest. The
   SFU's event loop calls poll_output in a tight loop until Timeout, so it's not a missed-poll issue.
4. The SFU runs 2 threads with one UdpSocket per thread. Each client is assigned to a specific thread/socket via hash-based sharding. The working and failing clients are distributed across
   both threads — it's not a per-thread issue.
5. All clients share the same pre-generated DtlsCert on the server side (via Arc). The client side generates a fresh cert per Rtc::builder().build().
6. Timing pattern: the first 1-2 clients almost always succeed. As more clients connect concurrently (even just 4-8 total), the DTLS success rate drops dramatically. With 8 concurrent
   clients, typically only 1-2 complete DTLS.
7. When a client is torn down and reconnected (new Rtc, new WHIP POST, new UDP socket), it sometimes succeeds on the retry — suggesting the issue is timing/state-dependent, not a permanent
   crypto mismatch.

SDP setup

• Client (offerer): creates offer with a=setup:actpass
• Server (answerer): accept_offer() generates answer — presumably with a=setup:active (server initiates DTLS)

The client-side log shows DTLS setup is: true followed by ClientHello, which suggests the client thinks it's the DTLS initiator (active role). If the server also considers itself active (from the SDP answer), both sides would send ClientHello to each other, and neither would respond with ServerHello — a role conflict deadlock.

However, we haven't confirmed this theory because we can't easily inspect the SDP a=setup attribute at runtime. If both sides end up as DTLS active, the handshake would deadlock exactly as observed.

Reproduction

// Server side (SFU)
  let mut rtc = Rtc::builder()                                                                                                                                                                
      .set_dtls_cert(shared_cert.clone())                                                                                                                                                     
      .set_fingerprint_verification(true)                                                                                                                                                     
      .enable_bwe(Some(Bitrate::kbps(1000)))                                                                                                                                                  
      .build(Instant::now());                                                                                                                                                                 
                                                                                                                                                                                              
  let candidate = Candidate::host(local_addr, "udp")?;                                                                                                                                        
  rtc.add_local_candidate(candidate);                                                                                                                                                         
                                                                                                                                                                                              
  let answer = rtc.sdp_api().accept_offer(client_offer)?;                                                                                                                                     
  // → send answer back to client via HTTP                                                                                                                                                    
  // → send Rtc to event loop thread                                                                                                                                                          
                                                                                                                                                                                              
  // Client side (bench tool)                                                                                                                                                                 
  let mut rtc = Rtc::builder()                                                                                                                                                                
      .set_stats_interval(Some(Duration::from_secs(2)))                                                                                                                                       
      .build(Instant::now());                                                                                                                                                                 
                                                                                                                                                                                              
  let candidate = Candidate::host(local_addr, "udp")?;                                                                                                                                        
  rtc.add_local_candidate(candidate);                                                                                                                                                         
                                                                                                                                                                                              
  let mut change = rtc.sdp_api();                                                                                                                                                             
  change.add_media(MediaKind::Audio, Direction::SendOnly, None, None, None);                                                                                                                  
  change.add_channel("data".to_string());                                                                                                                                                     
  let (offer, pending) = change.apply()?;                                                                                                                                                     
                                                                                                                                                                                              
  let answer = SdpAnswer::from_sdp_string(&server_answer)?;                                                                                                                                   
  rtc.sdp_api().accept_answer(pending, answer)?;                                                                                                                                              
                                                                                                                                                                                              
  // Event loop: poll_output → Transmit (send), recv_from → Input::Receive

Connect 8+ clients simultaneously to reproduce. First 1-2 usually succeed, rest hang at DTLS.

Questions

1. Is the DTLS role (active/passive) correctly derived from the SDP a=setup attribute in accept_offer / accept_answer? Could both sides end up as active?
2. Is there a known concurrency issue when multiple Rtc instances on the same thread/process perform DTLS handshakes simultaneously?
3. Could the shared DtlsCert cause issues when multiple handshakes use the same certificate concurrently?

Note : I used AI to wrote this issue because I'm French and I want to be clear

[Teyk0o]

We can discuss directly about this issue on the discord server : https://discordapp.com/channels/1436725650302959829/1491067471187280015

[lolgesten]

@Teyk0o the answers are,

1. Logic is:

  1. Read the remote SDP's a=setup value                                                                                                                                                                             
  2. If remote says actpass → local takes passive (i.e. DTLS server)                                                                                                                                                 
  3. Otherwise → invert the remote's value (active ↔ passive)                                                                                                                                                        
  4. If no a=setup line exists → default to passive                                                                                                                                                                  

2. I don't think sharing cert would make any difference. The SDP is using individual ufrag for each client.
3. Don't believe the cert can have any such effect.

[Teyk0o]

Root cause identified (two separate issues)

@lolgesten After extensive debugging with SFU-side packet tracing, I found two bugs that together cause near-100% DTLS failure in multi-client setups.

Bug 1: rtc.accepts() rejects DTLS ClientHello during ICE transition

When a client sends its first DTLS ClientHello (right after ICE nomination), rtc.accepts(&input) returns false for that packet. ICE has nominated a pair but hasn't fully finalized — str0m doesn't recognize the packet as belonging to this Rtc instance.

SFU-side packet trace (IPs redacted):

[DIAG-DISPATCH] NEW cache <client_addr> -> client 0    ← STUN binding (accepted)
[DIAG-DISPATCH] addr_cache stale for <client_addr>     ← next packet: accepts() = false
[DIAG-DISPATCH] DROPPED from <client_addr>             ← DTLS ClientHello lost
[DIAG-DISPATCH] NEW cache <client_addr> -> client 0    ← next STUN (accepted again)
[DIAG-ICE] Client (0) ICE state -> Connected           ← ICE works, but DTLS never completes

The SFU uses accepts() for packet demuxing (required when multiple Rtc instances share sockets). The DTLS ClientHello arrives from the same address as STUN, but accepts() rejects it during the ICE transition window.

Workaround: We bypass accepts() when the source address is already in our dispatch cache (the cache proves address→client ownership). This fully resolves the DTLS failure.

Bug 2: DTLS flight retransmissions are never delivered

Even without the accepts() workaround, the client should recover by retransmitting the ClientHello after ~1s (DTLS flight timeout). It doesn't. After the initial ClientHello, zero retransmissions are produced for ~40 seconds (until the connect timeout kills the handshake).

Client-side log showing no retransmit:

T+0.000s  Sending ClientHello (cookie_len=0, 3 cipher suites)
T+0.000s  Flight timeout in: 1.134s
          ... 40 seconds of silence, no retransmit ...
T+40.2s   Flight timeout in: 1.131s   ← first sign of life, but too late

I traced this to str0m/src/lib.rs line ~1661 in poll_output():

if let Some(timeout) = self.next_dtls_timeout {
    if timeout <= self.last_now {
        let _ = self.dtls.handle_timeout(self.last_now);
        self.next_dtls_timeout = None;  // reset to None
    }
}

After handle_timeout(), dimpl queues retransmit packets internally and arms a new flight timer. But str0m sets next_dtls_timeout = None and does not re-poll dtls.poll_output() in the same iteration to collect:

1. The retransmit Packet (the actual ClientHello resend)
2. The new Timeout value (next flight timer)

Since next_dtls_timeout stays None, str0m never calls dtls.handle_timeout() again, and the retransmit packets sit in dimpl's internal queue forever.

Impact

Bug 1 alone would be survivable if retransmissions worked. Bug 2 alone would be invisible if accepts() didn't drop the first ClientHello. Together, they make DTLS completion essentially luck-based: it only works when the first ClientHello happens to arrive after ICE has fully settled (a tight timing window).

Environment

• str0m 0.18.0, dimpl 0.5.0
• SFU (answerer/passive) + bench client (offerer/active)
• Reproduced with 1 SFU thread, 2 clients, 3s spawn interval

Note : I used AI to wrote this issue because I'm French and I want to be clear

[algesten]

@Teyk0o thanks!

I believe we have solved one problem in #943. The other needs more analysis.