enable xdp by default

CHANGELOG.md

@@ -23,10 +23,16 @@ Release channels have their own copy of this changelog:
   `getLatestBlockhash` response together with its context (notably `context.slot`).
 ### Validator
 #### Breaking
+* XDP transmit is now enabled by default on Linux in copy mode on CPU core 1. Use
+  `--xdp-cpu-cores` to override the XDP CPU assignment. Use `--xdp-zero-copy` with
+  `--xdp-interface` to opt in to zero copy. Default validator startup now requires the XDP
+  copy-mode capabilities.
+* The default PoH pinned CPU core is now CPU core 10. Use `--poh-pinned-cpu-core` to override it.
 #### Deprecations
 * `--accounts-db-access-storages-method` is now deprecated and a no-op (the `mmap` value was
   deprecated in v4.0.0; mmap mode has now been removed entirely). The flag is still accepted for
   backward compatibility, but account storages are always accessed via file I/O.
+* `--experimental-poh-pinned-cpu-core` is now deprecated. Use `--poh-pinned-cpu-core` instead.
 #### Changes
 * Turbine shred ingestion now rejects shreds more than half an epoch in the future (previously up to 2 full epochs ahead was accepted).
 ### CLI

docs/src/operations/running-with-af-xdp.md

@@ -15,27 +15,30 @@ Before rolling out XDP on a production validator, you should test it on your set
 * **Performance Gain:** Confirm that performance is improved with the new configuration (e.g. lower CPU usage or higher throughput in Turbine’s retransmit stage).
 * **Metric Visibility:** Verify that you can observe the retransmit-stage metrics, which show time spent sending shreds, to gauge the impact of XDP on network transmission.
 
-To enable XDP in Agave, add the following command-line flags to your validator startup command (using Agave v3.0.9+):
+XDP is enabled by default on Linux in Agave. The default XDP configuration uses CPU core 1 and copy mode. To use different CPU cores for XDP, pass:
 
 ```bash
---experimental-retransmit-xdp-cpu-cores 1
---experimental-retransmit-xdp-zero-copy # Do NOT pass this flag when using the bnxt_en driver.
---experimental-poh-pinned-cpu-core 10
+--xdp-cpu-cores 2
 ```
 
-Note that --experimental-retransmit-xdp-zero-copy will avoid using socket buffers for data, but this is only possible when talking directly to the Network Interface Card (NIC). As a result, zero copy cannot be used with the bonded interface itself. When using a bonded network interface, specify the underlying member interface to which the XDP program should be attached:
+Zero copy avoids using socket buffers for data, but this is only possible when talking directly to the Network Interface Card (NIC). To opt in to zero copy, pass an explicit physical interface:
 
 ```bash
---experimental-retransmit-xdp-interface <bond-member-interface>
+--xdp-zero-copy --xdp-interface <interface>
 ```
 
- Also note that XDP and PoH *must* be assigned to separate (physical) cores. The
---experimental-poh-pinned-cpu-core N flag can be used to move the PoH thread.
+Zero copy cannot be used with a bonded interface itself. When using a bonded network interface, specify the underlying member interface to which the XDP program should be attached:
 
-Next, your validator binary will need to have access to a few higher level permissions. The validator process requires the CAP_NET_RAW, CAP_NET_ADMIN, CAP_BPF, and CAP_PERFMON capabilities. These can be configured in the systemd service file by setting CapabilityBoundingSet=CAP_NET_RAW CAP_NET_ADMIN CAP_BPF CAP_PERFMON under the [Service] section or directly on the binary with the command:
+```bash
+--xdp-zero-copy --xdp-interface <bond-member-interface>
+```
+
+Also note that XDP and PoH *must* be assigned to separate (physical) cores. PoH defaults to CPU core 10, and XDP defaults to CPU core 1. The --poh-pinned-cpu-core N flag can be used to move the PoH thread.
+
+Next, your validator binary will need to have access to a few higher level permissions. With default copy-mode XDP, the validator process requires the CAP_NET_RAW and CAP_NET_ADMIN capabilities. Zero copy additionally requires CAP_BPF and CAP_PERFMON. These capabilities can be configured in the systemd service file by setting CapabilityBoundingSet=CAP_NET_RAW CAP_NET_ADMIN under the [Service] section or directly on the binary with the command:
 
 ```bash
-sudo setcap cap_net_raw,cap_net_admin,cap_bpf,cap_perfmon=p <path/to/agave-validator>
+sudo setcap cap_net_raw,cap_net_admin=p <path/to/agave-validator>
 #this command must be run each time the binary is replaced
 ```
 
@@ -78,7 +81,7 @@ modinfo bnxt_en
 | `igb` / Intel I210 | ✅ Works | ✅ Works w/ caveat | caveat: `igb` requires kernel `>= 6.14` for ZC. Field report: I210 on 6.17 enabled ZC but had severe network degradation/high skips, so fall back to non-ZC if unstable. |
 | `ixgbe` / Intel X540, X550 | ✅ Works | ⚠️ Mixed / unstable | Alessandro guidance for freeze/link-flap cases: start without ZC while `ixgbe` is debugged. Stay tuned! |
 | `ice` / Intel E800 | ✅ Works | ✅ Works | `ice` supports native XDP and AF_XDP zero-copy. Caveats: XDP is blocked for frame sizes larger than 3KB |
-| `bnxt_en` / Broadcom | ✅ Works | ❌ Does not work | `bnxt_en` works with XDP, but do not pass the zero-copy flag. Broadcom non-ZC can still be reasonably fast. But please get a non-broadcom NIC |
+| `bnxt_en` / Broadcom | ✅ Works | ❌ Does not work | `bnxt_en` works with default copy-mode XDP. Broadcom non-ZC can still be reasonably fast. But please get a non-broadcom NIC |
 | `tg3` / Broadcom | ❌ No native/driver XDP; generic XDP only at best | ❌ Does not work | Broadcom BCM5720 uses the `tg3` driver. Treat as unsupported for Agave/AF_XDP performance work: no native XDP and no AF_XDP zero-copy. |
 | `r8169` / Realtek | ❌ No native/driver XDP; generic XDP only at best | ❌ Does not work | Realtek NICs using `r8169` should be treated as unsupported for Agave/AF_XDP performance work: no native XDP and no AF_XDP zero-copy.|
 | `mlx4_en` / Mellanox ConnectX-3 | ❌ Do not use | ❌ Does not work | Driver is no longer supported. Zero-copy does not work. Do not use. |

multinode-demo/bootstrap-validator.sh

@@ -144,6 +144,7 @@ args+=(
   --no-wait-for-vote-to-start-leader
   --full-rpc-api
   --allow-private-addr
+  --no-xdp
 )
 default_arg --gossip-port 8001
 default_arg --log -

multinode-demo/validator.sh

@@ -10,6 +10,7 @@ args=(
   --max-genesis-archive-unpacked-size 1073741824
   --no-poh-speed-test
   --no-os-network-limits-test
+  --no-xdp
 )
 airdrops_enabled=1
 node_sol=500 # 500 SOL: number of SOL to airdrop the node for transaction fees and vote account rent exemption (ignored if airdrops_enabled=0)

poh/src/poh_service.rs

@@ -40,7 +40,7 @@ pub const DEFAULT_HASHES_PER_BATCH: u64 =
     TARGET_HASH_BATCH_TIME_US * DEFAULT_HASHES_PER_SECOND / 1_000_000;
 
 #[cfg(target_os = "linux")]
-pub const DEFAULT_PINNED_CPU_CORE: Option<usize> = Some(0);
+pub const DEFAULT_PINNED_CPU_CORE: Option<usize> = Some(10);
 #[cfg(not(target_os = "linux"))]
 pub const DEFAULT_PINNED_CPU_CORE: Option<usize> = None;
 
@@ -154,12 +154,12 @@ impl PohService {
                     #[cfg(target_os = "linux")]
                     if let Some(pinned_cpu_core) = pinned_cpu_core {
                         // PoH service runs in a tight loop, generating hashes as fast as possible.
-                        // Let's dedicate one of the CPU cores to this thread so that it can gain
-                        // from cache performance.
+                        // Dedicate one CPU core to this thread for cache performance.
                         let pinned_cpu = CpuId::new(pinned_cpu_core).unwrap();
+                        info!("Pinning PoH service to CPU core {pinned_cpu_core}");
                         set_cpu_affinity(None, [pinned_cpu]).unwrap_or_else(|e| {
                             panic!(
-                                "Failed to set CPU affinity for POH service to CPU \
+                                "Failed to set CPU affinity for PoH service to CPU \
                                  {pinned_cpu_core}: {e:?}. This is critical for performance."
                             )
                         });

scripts/run.sh

@@ -121,6 +121,7 @@ args=(
   --require-tower
   --no-wait-for-vote-to-start-leader
   --no-os-network-limits-test
+  --no-xdp
 )
 # shellcheck disable=SC2086
 agave-validator "${args[@]}" $SOLANA_RUN_SH_VALIDATOR_ARGS &

validator/src/cli.rs

@@ -158,19 +158,29 @@ fn deprecated_arguments() -> Vec<DeprecatedArg> {
             .conflicts_with("accounts_index_limit"),
         replaced_by: "accounts-index-limit",
     );
+    add_arg!(
+        // deprecated in v4.2.0
+        Arg::with_name("experimental_poh_pinned_cpu_core")
+            .long("experimental-poh-pinned-cpu-core")
+            .takes_value(true)
+            .value_name("CPU_ID")
+            .conflicts_with("poh_pinned_cpu_core")
+            .validator(is_parsable::<usize>)
+            .help("Specify which CPU core PoH is pinned to. Use --poh-pinned-cpu-core instead"),
+        replaced_by: "poh-pinned-cpu-core",
+    );
     add_arg!(
         // deprecated in v4.1.0
         Arg::with_name("experimental_retransmit_xdp_cpu_cores")
             .long("experimental-retransmit-xdp-cpu-cores")
             .takes_value(true)
             .value_name("CPU_LIST")
+            .conflicts_with("no_xdp")
             .conflicts_with("xdp_cpu_cores")
             .validator(|value| {
                 validate_cpu_ranges(value, "--experimental-retransmit-xdp-cpu-cores")
             })
-            .help(
-                "Enable XDP retransmit on the specified CPU cores. Use --xdp-cpu-cores instead",
-            ),
+            .help("Use the specified CPU cores for XDP. Use --xdp-cpu-cores instead"),
         replaced_by: "xdp-cpu-cores",
     );
     add_arg!(
@@ -179,18 +189,18 @@ fn deprecated_arguments() -> Vec<DeprecatedArg> {
             .long("experimental-retransmit-xdp-interface")
             .takes_value(true)
             .value_name("INTERFACE")
+            .conflicts_with("no_xdp")
             .conflicts_with("xdp_interface")
-            .requires("experimental_retransmit_xdp_cpu_cores")
-            .help("Network interface to use for XDP retransmit. Use --xdp-interface instead"),
+            .help("Network interface to use for XDP. Use --xdp-interface instead"),
         replaced_by: "xdp-interface",
     );
     add_arg!(
         // deprecated in v4.1.0
         Arg::with_name("experimental_retransmit_xdp_zero_copy")
             .long("experimental-retransmit-xdp-zero-copy")
             .takes_value(false)
+            .conflicts_with("no_xdp")
             .conflicts_with("xdp_zero_copy")
-            .requires("experimental_retransmit_xdp_cpu_cores")
             .help("Enable XDP zero copy. Use --xdp-zero-copy instead"),
         replaced_by: "xdp-zero-copy",
     );

validator/src/commands/run/args.rs

@@ -30,7 +30,7 @@ use {
     solana_send_transaction_service::send_transaction_service::Config as SendTransactionServiceConfig,
     solana_signer::Signer,
     solana_unified_scheduler_pool::DefaultSchedulerPool,
-    std::{collections::HashSet, net::SocketAddr, path::PathBuf, str::FromStr},
+    std::{collections::HashSet, net::SocketAddr, path::PathBuf},
 };
 
 const EXCLUDE_KEY: &str = "account-index-exclude-key";
@@ -864,12 +864,11 @@ pub fn add_args<'a>(app: App<'a, 'a>, default_args: &'a DefaultArgs) -> App<'a,
     )
     .arg(
         Arg::with_name("poh_pinned_cpu_core")
-            .hidden(hidden_unless_forced())
-            .long("experimental-poh-pinned-cpu-core")
+            .long("poh-pinned-cpu-core")
             .takes_value(true)
             .value_name("CPU_ID")
-            .validator(|s| usize::from_str(&s).map(|_| ()).map_err(|e| e.to_string()))
-            .help("Specify which CPU core PoH is pinned to"),
+            .validator(is_parsable::<usize>)
+            .help("Specify which CPU core PoH is pinned to. Defaults to CPU 10 on Linux"),
     )
     .arg(
         Arg::with_name("poh_hashes_per_batch")
@@ -1208,28 +1207,44 @@ pub fn add_args<'a>(app: App<'a, 'a>, default_args: &'a DefaultArgs) -> App<'a,
             .validator(|s| is_within_range(s, 1..))
             .help(DefaultSchedulerPool::cli_message()),
     )
+    .arg(
+        Arg::with_name("no_xdp")
+            .long("no-xdp")
+            .takes_value(false)
+            .conflicts_with("experimental_retransmit_xdp_cpu_cores")
+            .conflicts_with("experimental_retransmit_xdp_interface")
+            .conflicts_with("experimental_retransmit_xdp_zero_copy")
+            .conflicts_with("xdp_cpu_cores")
+            .conflicts_with("xdp_interface")
+            .conflicts_with("xdp_zero_copy")
+            .help("Do not use XDP transmit"),
+    )
+    .arg(
+        Arg::with_name("xdp_zero_copy")
+            .long("xdp-zero-copy")
+            .takes_value(false)
+            .conflicts_with("no_xdp")
+            .help("Enable XDP zero copy"),
+    )
     .arg(
         Arg::with_name("xdp_interface")
             .long("xdp-interface")
             .takes_value(true)
             .value_name("INTERFACE")
-            .requires("xdp_cpu_cores")
-            .help("Network interface to use for XDP"),
+            .conflicts_with("no_xdp")
+            .help("Network interface to use for XDP. Required when XDP zero copy is enabled"),
     )
     .arg(
         Arg::with_name("xdp_cpu_cores")
             .long("xdp-cpu-cores")
             .takes_value(true)
             .value_name("CPU_LIST")
+            .conflicts_with("no_xdp")
             .validator(|value| validate_cpu_ranges(value, "--xdp-cpu-cores"))
-            .help("Use the specified CPU cores for XDP"),
-    )
-    .arg(
-        Arg::with_name("xdp_zero_copy")
-            .long("xdp-zero-copy")
-            .takes_value(false)
-            .requires("xdp_cpu_cores")
-            .help("Enable XDP zero copy. Requires hardware support"),
+            .help(
+                "Use the specified CPU cores for XDP. Defaults to an auto-selected CPU on a \
+                 physical core separate from PoH",
+            ),
     )
     .args(&pub_sub_config::args(/*test_validator:*/ false))
     .args(&json_rpc_config::args())