fix e2e test to use an external sonar simulator and fix the integration of GPSD Client in the src directory before the test

docs/test/hardware_e2e.md

@@ -0,0 +1,111 @@
+# Hardware E2E Test Flow (run_poseidon_e2e.sh)
+
+This document describes the chronological logic of the hardware E2E test implemented in
+`test/e2e/run_poseidon_e2e.sh`.
+
+## Preconditions
+- ROS Noetic is available (`/opt/ros/noetic/setup.bash`).
+- Poseidon workspace is built (so `src/workspace/devel/setup.bash` exists).
+- Passwordless sudo is available when running as a non-root user.
+- The test host can bind to `127.0.0.1` ports 8080/9099/9002.
+- A real sonar device (or an externally simulated sonar device) is available at
+  `POSEIDON_SONAR_DEVICE` (default: `/dev/sonar`).
+
+## GPSD nodes integration
+By default, `test/e2e/run_poseidon_e2e.sh` integrates `gps_umd` before launching tests
+(`POSEIDON_E2E_INSTALL_GPSD_CLIENT=1`).
+
+Integration steps executed by the script:
+
+```bash
+echo -e "\e[35mDownloading GPSD-Client\e[0m"
+cd /opt
+sudo git clone https://github.com/CIDCO-dev/gps_umd.git
+cd /opt/gps_umd
+sudo mv gps* /opt/Poseidon/src/workspace/src/
+cd /opt
+sudo rm -rd gps_umd
+```
+
+Then the script rebuilds:
+
+```bash
+cd "$POSEIDON_ROOT/src/workspace"
+catkin_make
+```
+
+To disable this automatic integration:
+
+```bash
+export POSEIDON_E2E_INSTALL_GPSD_CLIENT=0
+```
+
+## Key environment variables (defaults)
+- `POSEIDON_ROOT`: repo root (auto-detected).
+- `POSEIDON_E2E_ARTIFACT_DIR`: `test/e2e/artifacts`.
+- `POSEIDON_HTTP_PORT`: `8080`.
+- `POSEIDON_E2E_BASE_URL`: `http://127.0.0.1:8080`.
+- `DIAGNOSTICS_WS_PORT`: `9099`.
+- `POSEIDON_TELEMETRY_WS_PORT`: `9002`.
+- `POSEIDON_E2E_REUSE_RUNNING`: `0` (start a new stack unless set to 1).
+- `POSEIDON_E2E_EXCLUSIVE`: `0` (set to 1 to stop existing services/ports first).
+- `POSEIDON_E2E_LAUNCH_AS_ROOT`: `1` (required for I2C/GPIO).
+- `POSEIDON_E2E_REQUIRED_NODES`: `Diagnostics`.
+- `POSEIDON_E2E_INSTALL_GPSD_CLIENT`: `1` (auto-install `gps_umd` if `gpsd_client` is missing).
+- `POSEIDON_CONFIG_PATH`: `config.txt` in the repo.
+- `POSEIDON_LOGGER_PATH`: temp dir created by the script.
+- `POSEIDON_SONAR_DEVICE`: `/dev/sonar` by default.
+
+## Chronological steps
+1) Initialize paths and defaults.
+   - Set repo root, artifacts directory, ports, and E2E flags.
+   - Create the artifacts directory.
+
+2) Source ROS environment.
+   - Source `/opt/ros/noetic/setup.bash` (if present).
+   - Source `src/workspace/devel/setup.bash` (if present).
+
+3) Resolve configuration and logging paths.
+   - Set `POSEIDON_CONFIG_PATH` to `config.txt` if not provided.
+   - Create a temp logger directory and export `POSEIDON_LOGGER_PATH`.
+
+4) Resolve sonar device.
+   - Use `POSEIDON_SONAR_DEVICE` if provided.
+   - Otherwise default to `/dev/sonar`.
+
+5) Enforce root access.
+   - If running as non-root and `POSEIDON_E2E_LAUNCH_AS_ROOT=1`, require passwordless sudo.
+
+6) Register cleanup handler.
+   - On exit, stop the HTTP server and terminate the ROS stack
+     (including escalating with sudo if it was launched as root).
+
+7) Phase 0: integrate GPSD client (default enabled).
+   - If `POSEIDON_E2E_INSTALL_GPSD_CLIENT=1` and `gpsd_client` is missing, clone
+     `https://github.com/CIDCO-dev/gps_umd.git`.
+   - Move `gps*` ROS packages into `src/workspace/src`.
+   - Rebuild the workspace with `catkin_make`.
+
+8) Phase 1: start ROS and wait for readiness.
+   - Start a local HTTP server from `www/webroot` on `POSEIDON_HTTP_PORT`.
+   - If `POSEIDON_E2E_EXCLUSIVE=1`, stop any existing `ros` systemd service and kill
+     websocket ports (9002/9099/9003/9004).
+   - Start `launchROSService.sh` (via sudo if required).
+   - Wait for TCP ports to open: diagnostics websocket, telemetry websocket, and HTTP.
+   - Wait for ROS nodes (by default, `Diagnostics`) to appear.
+
+9) Phase 2: run backend websocket test.
+   - Execute `src/workspace/src/diagnostics/tests/test_launchrosservice_websocket.py`.
+
+10) Phase 3: run UI test.
+   - In `test/e2e/`, install `node_modules` if missing.
+   - Run `node ui_test.js`.
+
+11) If the UI test fails, collect diagnostics.
+   - Print a tail of `test/e2e/artifacts/launchROSService.log`.
+   - Dump `rostopic list` (first 200), `rostopic hz /imu/data`, and `rostopic hz /depth`.
+   - Query the diagnostics websocket for `GNSS Fix`, `IMU Communication`, and
+     `Sonar Communication` and print status/messages.
+
+12) Exit.
+   - If any phase fails, the script exits non-zero and the cleanup handler runs.

test/e2e/run_poseidon_e2e.sh

@@ -16,6 +16,7 @@ export POSEIDON_E2E_REUSE_RUNNING="${POSEIDON_E2E_REUSE_RUNNING:-0}"
 export POSEIDON_E2E_EXCLUSIVE="${POSEIDON_E2E_EXCLUSIVE:-0}"
 export POSEIDON_E2E_LAUNCH_AS_ROOT="${POSEIDON_E2E_LAUNCH_AS_ROOT:-1}"
 export POSEIDON_E2E_REQUIRED_NODES="${POSEIDON_E2E_REQUIRED_NODES:-Diagnostics}"
+export POSEIDON_E2E_INSTALL_GPSD_CLIENT="${POSEIDON_E2E_INSTALL_GPSD_CLIENT:-1}"
 
 mkdir -p "${POSEIDON_E2E_ARTIFACT_DIR}"
 
@@ -38,15 +39,8 @@ if [[ -z "${POSEIDON_LOGGER_PATH:-}" ]]; then
 fi
 mkdir -p "${POSEIDON_LOGGER_PATH}"
 
-# If a dedicated "fake sonar" serial adapter exists, use it as the sonar source for this run.
-# This avoids touching /dev/sonar and makes /depth deterministic for E2E.
-if [[ -z "${POSEIDON_SONAR_DEVICE:-}" ]]; then
-  if [[ -n "${DIAGNOSTICS_FAKE_SERIAL_PORT:-}" && -e "${DIAGNOSTICS_FAKE_SERIAL_PORT}" && "${DIAGNOSTICS_FAKE_SERIAL_PORT}" != "/dev/sonar" ]]; then
-    export POSEIDON_SONAR_DEVICE="${DIAGNOSTICS_FAKE_SERIAL_PORT}"
-  else
-    export POSEIDON_SONAR_DEVICE="/dev/sonar"
-  fi
-fi
+# The E2E now expects a real or externally simulated sonar device.
+export POSEIDON_SONAR_DEVICE="${POSEIDON_SONAR_DEVICE:-/dev/sonar}"
 
 HTTP_LOG="${POSEIDON_E2E_ARTIFACT_DIR}/http.log"
 SERVICE_LOG="${POSEIDON_E2E_ARTIFACT_DIR}/launchROSService.log"
@@ -71,11 +65,6 @@ cleanup() {
     wait "${HTTP_PID}" 2>/dev/null || true
   fi
 
-  if [[ -n "${NMEA_WRITER_PID:-}" ]] && kill -0 "${NMEA_WRITER_PID}" 2>/dev/null; then
-    kill "${NMEA_WRITER_PID}" 2>/dev/null || true
-    wait "${NMEA_WRITER_PID}" 2>/dev/null || true
-  fi
-
   if [[ -n "${SERVICE_PID:-}" ]] && kill -0 "${SERVICE_PID}" 2>/dev/null; then
     if [[ "${SERVICE_LAUNCHED_AS_ROOT}" == "1" ]]; then
       sudo kill -INT -- "-${SERVICE_PID}" 2>/dev/null || true
@@ -98,6 +87,72 @@ cleanup() {
 }
 trap cleanup EXIT INT TERM
 
+run_with_root() {
+  if [[ "$(id -u)" -eq 0 ]]; then
+    "$@"
+  else
+    sudo "$@"
+  fi
+}
+
+integrate_gpsd_client() {
+  if [[ "${POSEIDON_E2E_INSTALL_GPSD_CLIENT}" != "1" ]]; then
+    echo "[phase 0] skip gpsd_client integration (POSEIDON_E2E_INSTALL_GPSD_CLIENT=${POSEIDON_E2E_INSTALL_GPSD_CLIENT})"
+    return 0
+  fi
+
+  local workspace_src="${POSEIDON_ROOT}/src/workspace/src"
+  local workspace_root="${POSEIDON_ROOT}/src/workspace"
+  local gpsd_repo_url="https://github.com/CIDCO-dev/gps_umd.git"
+  local gpsd_tmp_dir="/opt/gps_umd"
+
+  if [[ ! -d "${workspace_src}" ]]; then
+    echo "[!] Workspace source directory not found: ${workspace_src}"
+    exit 1
+  fi
+
+  if [[ -d "${workspace_src}/gpsd_client" ]]; then
+    echo "[phase 0] gpsd_client already integrated"
+    return 0
+  fi
+
+  echo "[phase 0] integrate gpsd_client into Poseidon workspace"
+  if ! command -v git >/dev/null 2>&1; then
+    echo "[!] git is required to install gps_umd."
+    exit 1
+  fi
+
+  # Equivalent to the field workflow: clone -> move gps* packages -> cleanup.
+  run_with_root rm -rf "${gpsd_tmp_dir}" 2>/dev/null || true
+  run_with_root git clone "${gpsd_repo_url}" "${gpsd_tmp_dir}"
+  run_with_root env GPSD_TMP_DIR="${gpsd_tmp_dir}" WORKSPACE_SRC="${workspace_src}" bash -lc '
+    set -e
+    shopt -s nullglob
+    pkgs=("$GPSD_TMP_DIR"/gps*)
+    if [[ ${#pkgs[@]} -eq 0 ]]; then
+      echo "[!] No gps* packages found in $GPSD_TMP_DIR." >&2
+      exit 1
+    fi
+    mv "${pkgs[@]}" "$WORKSPACE_SRC/"
+  '
+  run_with_root rm -rf "${gpsd_tmp_dir}"
+
+  if ! command -v catkin_make >/dev/null 2>&1; then
+    echo "[!] catkin_make not found; cannot build workspace after gps_umd install."
+    exit 1
+  fi
+
+  echo "[phase 0] rebuild workspace after gpsd_client integration"
+  pushd "${workspace_root}" >/dev/null
+  catkin_make
+  popd >/dev/null
+
+  if [[ -f "${POSEIDON_ROOT}/src/workspace/devel/setup.bash" ]]; then
+    # shellcheck disable=SC1091
+    source "${POSEIDON_ROOT}/src/workspace/devel/setup.bash"
+  fi
+}
+
 start_http_server() {
   pushd "${POSEIDON_ROOT}/www/webroot" >/dev/null
   python3 -m http.server "${POSEIDON_HTTP_PORT}" --bind 127.0.0.1 >"${HTTP_LOG}" 2>&1 &
@@ -130,37 +185,6 @@ start_poseidon_service() {
   fi
 }
 
-start_fake_nmea_writer() {
-  # Keep a fake NMEA DBT feed alive during both backend and UI phases (if the port exists).
-  if [[ -n "${DIAGNOSTICS_FAKE_SERIAL_PORT:-}" && -e "${DIAGNOSTICS_FAKE_SERIAL_PORT}" && "${DIAGNOSTICS_FAKE_SERIAL_PORT}" != "/dev/sonar" ]]; then
-    python3 - <<'PY' &
-import os
-import time
-from pathlib import Path
-
-port = Path(os.environ["DIAGNOSTICS_FAKE_SERIAL_PORT"])
-
-def checksum(payload: str) -> str:
-    cs = 0
-    for ch in payload:
-        cs ^= ord(ch)
-    return f"{cs:02X}"
-
-payload = "SDDBT,30.9,f,9.4,M,5.1,F"
-sentence = f"${payload}*{checksum(payload)}\r\n".encode("ascii")
-
-while True:
-    try:
-        with open(port, "wb", buffering=0) as fd:
-            fd.write(sentence)
-    except Exception:
-        pass
-    time.sleep(1.0)
-PY
-    NMEA_WRITER_PID=$!
-  fi
-}
-
 wait_for_ports() {
   if ! python3 - <<'PY'
 import os
@@ -262,7 +286,6 @@ phase_start_ros() {
   start_http_server
   stop_existing_poseidon
   start_poseidon_service
-  start_fake_nmea_writer
   wait_for_ports
   wait_for_ros_nodes
 }
@@ -333,6 +356,7 @@ PY
   popd >/dev/null
 }
 
+integrate_gpsd_client
 phase_start_ros
 phase_backend_test
 phase_ui_test

version.md

@@ -1,5 +1,10 @@
 # Version History (English)
 
+## 2026-02-19
+- e2e: removed the internal fake sonar DBT writer from `test/e2e/run_poseidon_e2e.sh`; E2E now expects a real sonar or an external sonar simulator via `POSEIDON_SONAR_DEVICE`.
+- e2e: added an automatic pre-test GPSD client integration phase (`POSEIDON_E2E_INSTALL_GPSD_CLIENT=1` by default) that clones `gps_umd`, moves `gps*` packages into `src/workspace/src`, and rebuilds with `catkin_make` when `gpsd_client` is missing.
+- docs: updated `docs/test/hardware_e2e.md` to reflect external sonar usage and the new automatic GPSD integration behavior.
+
 ## 2026-01-12
 - CI: clean Catkin build/devel/logs on self-hosted runners before builds to avoid `.built_by` permission errors.
 - CI: pre-clean the GitHub Actions workspace with sudo before checkout to avoid permission errors during repository cleanup.