From 46fb6174d15745737fd7363ae3c1971442c47178 Mon Sep 17 00:00:00 2001 From: dany Date: Thu, 19 Feb 2026 09:50:30 -0500 Subject: [PATCH 1/2] - e2e: removed the internal fake sonar DBT writer from `test/e2e/run_poseidon_e2e.sh`; E2E now expects a real sonar or an external sonar simulator via `POSEIDON_SONAR_DEVICE`. - e2e: added an automatic pre-test GPSD client integration phase (`POSEIDON_E2E_INSTALL_GPSD_CLIENT=1` by default) that clones `gps_umd`, moves `gps*` packages into `src/workspace/src`, and rebuilds with `catkin_make` when `gpsd_client` is missing. - docs: updated `docs/test/hardware_e2e.md` to reflect external sonar usage and the new automatic GPSD integration behavior. --- docs/test/hardware_e2e.md | 95 ++++++++++++++++++++++++++++++++++++ test/e2e/run_poseidon_e2e.sh | 48 +----------------- 2 files changed, 97 insertions(+), 46 deletions(-) create mode 100644 docs/test/hardware_e2e.md diff --git a/docs/test/hardware_e2e.md b/docs/test/hardware_e2e.md new file mode 100644 index 00000000..6bf20684 --- /dev/null +++ b/docs/test/hardware_e2e.md @@ -0,0 +1,95 @@ +# Hardware E2E Test Flow (run_poseidon_e2e.sh) + +This document describes the chronological logic of the hardware E2E test implemented in +`test/e2e/run_poseidon_e2e.sh`. + +## Preconditions +- ROS Noetic is available (`/opt/ros/noetic/setup.bash`). +- Poseidon workspace is built (so `src/workspace/devel/setup.bash` exists). +- Passwordless sudo is available when running as a non-root user. +- The test host can bind to `127.0.0.1` ports 8080/9099/9002. +- A real sonar device (or an externally simulated sonar device) is available at + `POSEIDON_SONAR_DEVICE` (default: `/dev/sonar`). + +## GPSD nodes integration (one-time setup) +If your setup requires GPSD-related nodes, install them in the ROS workspace before running E2E: + +```bash +echo -e "\e[35mDownloading GPSD-Client\e[0m" +cd /opt +sudo git clone https://github.com/CIDCO-dev/gps_umd.git +cd /opt/gps_umd +sudo mv gps* /opt/Poseidon/src/workspace/src/ +cd /opt +sudo rm -rd gps_umd +``` + +Then rebuild the workspace so the nodes are available: + +```bash +cd /opt/Poseidon/src/workspace +catkin_make +``` + +## Key environment variables (defaults) +- `POSEIDON_ROOT`: repo root (auto-detected). +- `POSEIDON_E2E_ARTIFACT_DIR`: `test/e2e/artifacts`. +- `POSEIDON_HTTP_PORT`: `8080`. +- `POSEIDON_E2E_BASE_URL`: `http://127.0.0.1:8080`. +- `DIAGNOSTICS_WS_PORT`: `9099`. +- `POSEIDON_TELEMETRY_WS_PORT`: `9002`. +- `POSEIDON_E2E_REUSE_RUNNING`: `0` (start a new stack unless set to 1). +- `POSEIDON_E2E_EXCLUSIVE`: `0` (set to 1 to stop existing services/ports first). +- `POSEIDON_E2E_LAUNCH_AS_ROOT`: `1` (required for I2C/GPIO). +- `POSEIDON_E2E_REQUIRED_NODES`: `Diagnostics`. +- `POSEIDON_CONFIG_PATH`: `config.txt` in the repo. +- `POSEIDON_LOGGER_PATH`: temp dir created by the script. +- `POSEIDON_SONAR_DEVICE`: `/dev/sonar` by default. + +## Chronological steps +1) Initialize paths and defaults. + - Set repo root, artifacts directory, ports, and E2E flags. + - Create the artifacts directory. + +2) Source ROS environment. + - Source `/opt/ros/noetic/setup.bash` (if present). + - Source `src/workspace/devel/setup.bash` (if present). + +3) Resolve configuration and logging paths. + - Set `POSEIDON_CONFIG_PATH` to `config.txt` if not provided. + - Create a temp logger directory and export `POSEIDON_LOGGER_PATH`. + +4) Resolve sonar device. + - Use `POSEIDON_SONAR_DEVICE` if provided. + - Otherwise default to `/dev/sonar`. + +5) Enforce root access. + - If running as non-root and `POSEIDON_E2E_LAUNCH_AS_ROOT=1`, require passwordless sudo. + +6) Register cleanup handler. + - On exit, stop the HTTP server and terminate the ROS stack + (including escalating with sudo if it was launched as root). + +7) Phase 1: start ROS and wait for readiness. + - Start a local HTTP server from `www/webroot` on `POSEIDON_HTTP_PORT`. + - If `POSEIDON_E2E_EXCLUSIVE=1`, stop any existing `ros` systemd service and kill + websocket ports (9002/9099/9003/9004). + - Start `launchROSService.sh` (via sudo if required). + - Wait for TCP ports to open: diagnostics websocket, telemetry websocket, and HTTP. + - Wait for ROS nodes (by default, `Diagnostics`) to appear. + +8) Phase 2: run backend websocket test. + - Execute `src/workspace/src/diagnostics/tests/test_launchrosservice_websocket.py`. + +9) Phase 3: run UI test. + - In `test/e2e/`, install `node_modules` if missing. + - Run `node ui_test.js`. + +10) If the UI test fails, collect diagnostics. + - Print a tail of `test/e2e/artifacts/launchROSService.log`. + - Dump `rostopic list` (first 200), `rostopic hz /imu/data`, and `rostopic hz /depth`. + - Query the diagnostics websocket for `GNSS Fix`, `IMU Communication`, and + `Sonar Communication` and print status/messages. + +11) Exit. + - If any phase fails, the script exits non-zero and the cleanup handler runs. diff --git a/test/e2e/run_poseidon_e2e.sh b/test/e2e/run_poseidon_e2e.sh index 392e6d26..ed01337d 100755 --- a/test/e2e/run_poseidon_e2e.sh +++ b/test/e2e/run_poseidon_e2e.sh @@ -38,15 +38,8 @@ if [[ -z "${POSEIDON_LOGGER_PATH:-}" ]]; then fi mkdir -p "${POSEIDON_LOGGER_PATH}" -# If a dedicated "fake sonar" serial adapter exists, use it as the sonar source for this run. -# This avoids touching /dev/sonar and makes /depth deterministic for E2E. -if [[ -z "${POSEIDON_SONAR_DEVICE:-}" ]]; then - if [[ -n "${DIAGNOSTICS_FAKE_SERIAL_PORT:-}" && -e "${DIAGNOSTICS_FAKE_SERIAL_PORT}" && "${DIAGNOSTICS_FAKE_SERIAL_PORT}" != "/dev/sonar" ]]; then - export POSEIDON_SONAR_DEVICE="${DIAGNOSTICS_FAKE_SERIAL_PORT}" - else - export POSEIDON_SONAR_DEVICE="/dev/sonar" - fi -fi +# The E2E now expects a real or externally simulated sonar device. +export POSEIDON_SONAR_DEVICE="${POSEIDON_SONAR_DEVICE:-/dev/sonar}" HTTP_LOG="${POSEIDON_E2E_ARTIFACT_DIR}/http.log" SERVICE_LOG="${POSEIDON_E2E_ARTIFACT_DIR}/launchROSService.log" @@ -71,11 +64,6 @@ cleanup() { wait "${HTTP_PID}" 2>/dev/null || true fi - if [[ -n "${NMEA_WRITER_PID:-}" ]] && kill -0 "${NMEA_WRITER_PID}" 2>/dev/null; then - kill "${NMEA_WRITER_PID}" 2>/dev/null || true - wait "${NMEA_WRITER_PID}" 2>/dev/null || true - fi - if [[ -n "${SERVICE_PID:-}" ]] && kill -0 "${SERVICE_PID}" 2>/dev/null; then if [[ "${SERVICE_LAUNCHED_AS_ROOT}" == "1" ]]; then sudo kill -INT -- "-${SERVICE_PID}" 2>/dev/null || true @@ -130,37 +118,6 @@ start_poseidon_service() { fi } -start_fake_nmea_writer() { - # Keep a fake NMEA DBT feed alive during both backend and UI phases (if the port exists). - if [[ -n "${DIAGNOSTICS_FAKE_SERIAL_PORT:-}" && -e "${DIAGNOSTICS_FAKE_SERIAL_PORT}" && "${DIAGNOSTICS_FAKE_SERIAL_PORT}" != "/dev/sonar" ]]; then - python3 - <<'PY' & -import os -import time -from pathlib import Path - -port = Path(os.environ["DIAGNOSTICS_FAKE_SERIAL_PORT"]) - -def checksum(payload: str) -> str: - cs = 0 - for ch in payload: - cs ^= ord(ch) - return f"{cs:02X}" - -payload = "SDDBT,30.9,f,9.4,M,5.1,F" -sentence = f"${payload}*{checksum(payload)}\r\n".encode("ascii") - -while True: - try: - with open(port, "wb", buffering=0) as fd: - fd.write(sentence) - except Exception: - pass - time.sleep(1.0) -PY - NMEA_WRITER_PID=$! - fi -} - wait_for_ports() { if ! python3 - <<'PY' import os @@ -262,7 +219,6 @@ phase_start_ros() { start_http_server stop_existing_poseidon start_poseidon_service - start_fake_nmea_writer wait_for_ports wait_for_ros_nodes } From 4b49db00f586446365b4e111ea39266dc75af4f4 Mon Sep 17 00:00:00 2001 From: dany Date: Thu, 19 Feb 2026 09:50:40 -0500 Subject: [PATCH 2/2] - e2e: removed the internal fake sonar DBT writer from `test/e2e/run_poseidon_e2e.sh`; E2E now expects a real sonar or an external sonar simulator via `POSEIDON_SONAR_DEVICE`. - e2e: added an automatic pre-test GPSD client integration phase (`POSEIDON_E2E_INSTALL_GPSD_CLIENT=1` by default) that clones `gps_umd`, moves `gps*` packages into `src/workspace/src`, and rebuilds with `catkin_make` when `gpsd_client` is missing. - docs: updated `docs/test/hardware_e2e.md` to reflect external sonar usage and the new automatic GPSD integration behavior. --- docs/test/hardware_e2e.md | 34 +++++++++++++----- test/e2e/run_poseidon_e2e.sh | 68 ++++++++++++++++++++++++++++++++++++ version.md | 5 +++ 3 files changed, 98 insertions(+), 9 deletions(-) diff --git a/docs/test/hardware_e2e.md b/docs/test/hardware_e2e.md index 6bf20684..d87f12d7 100644 --- a/docs/test/hardware_e2e.md +++ b/docs/test/hardware_e2e.md @@ -11,8 +11,11 @@ This document describes the chronological logic of the hardware E2E test impleme - A real sonar device (or an externally simulated sonar device) is available at `POSEIDON_SONAR_DEVICE` (default: `/dev/sonar`). -## GPSD nodes integration (one-time setup) -If your setup requires GPSD-related nodes, install them in the ROS workspace before running E2E: +## GPSD nodes integration +By default, `test/e2e/run_poseidon_e2e.sh` integrates `gps_umd` before launching tests +(`POSEIDON_E2E_INSTALL_GPSD_CLIENT=1`). + +Integration steps executed by the script: ```bash echo -e "\e[35mDownloading GPSD-Client\e[0m" @@ -24,13 +27,19 @@ cd /opt sudo rm -rd gps_umd ``` -Then rebuild the workspace so the nodes are available: +Then the script rebuilds: ```bash -cd /opt/Poseidon/src/workspace +cd "$POSEIDON_ROOT/src/workspace" catkin_make ``` +To disable this automatic integration: + +```bash +export POSEIDON_E2E_INSTALL_GPSD_CLIENT=0 +``` + ## Key environment variables (defaults) - `POSEIDON_ROOT`: repo root (auto-detected). - `POSEIDON_E2E_ARTIFACT_DIR`: `test/e2e/artifacts`. @@ -42,6 +51,7 @@ catkin_make - `POSEIDON_E2E_EXCLUSIVE`: `0` (set to 1 to stop existing services/ports first). - `POSEIDON_E2E_LAUNCH_AS_ROOT`: `1` (required for I2C/GPIO). - `POSEIDON_E2E_REQUIRED_NODES`: `Diagnostics`. +- `POSEIDON_E2E_INSTALL_GPSD_CLIENT`: `1` (auto-install `gps_umd` if `gpsd_client` is missing). - `POSEIDON_CONFIG_PATH`: `config.txt` in the repo. - `POSEIDON_LOGGER_PATH`: temp dir created by the script. - `POSEIDON_SONAR_DEVICE`: `/dev/sonar` by default. @@ -70,7 +80,13 @@ catkin_make - On exit, stop the HTTP server and terminate the ROS stack (including escalating with sudo if it was launched as root). -7) Phase 1: start ROS and wait for readiness. +7) Phase 0: integrate GPSD client (default enabled). + - If `POSEIDON_E2E_INSTALL_GPSD_CLIENT=1` and `gpsd_client` is missing, clone + `https://github.com/CIDCO-dev/gps_umd.git`. + - Move `gps*` ROS packages into `src/workspace/src`. + - Rebuild the workspace with `catkin_make`. + +8) Phase 1: start ROS and wait for readiness. - Start a local HTTP server from `www/webroot` on `POSEIDON_HTTP_PORT`. - If `POSEIDON_E2E_EXCLUSIVE=1`, stop any existing `ros` systemd service and kill websocket ports (9002/9099/9003/9004). @@ -78,18 +94,18 @@ catkin_make - Wait for TCP ports to open: diagnostics websocket, telemetry websocket, and HTTP. - Wait for ROS nodes (by default, `Diagnostics`) to appear. -8) Phase 2: run backend websocket test. +9) Phase 2: run backend websocket test. - Execute `src/workspace/src/diagnostics/tests/test_launchrosservice_websocket.py`. -9) Phase 3: run UI test. +10) Phase 3: run UI test. - In `test/e2e/`, install `node_modules` if missing. - Run `node ui_test.js`. -10) If the UI test fails, collect diagnostics. +11) If the UI test fails, collect diagnostics. - Print a tail of `test/e2e/artifacts/launchROSService.log`. - Dump `rostopic list` (first 200), `rostopic hz /imu/data`, and `rostopic hz /depth`. - Query the diagnostics websocket for `GNSS Fix`, `IMU Communication`, and `Sonar Communication` and print status/messages. -11) Exit. +12) Exit. - If any phase fails, the script exits non-zero and the cleanup handler runs. diff --git a/test/e2e/run_poseidon_e2e.sh b/test/e2e/run_poseidon_e2e.sh index ed01337d..f3568e76 100755 --- a/test/e2e/run_poseidon_e2e.sh +++ b/test/e2e/run_poseidon_e2e.sh @@ -16,6 +16,7 @@ export POSEIDON_E2E_REUSE_RUNNING="${POSEIDON_E2E_REUSE_RUNNING:-0}" export POSEIDON_E2E_EXCLUSIVE="${POSEIDON_E2E_EXCLUSIVE:-0}" export POSEIDON_E2E_LAUNCH_AS_ROOT="${POSEIDON_E2E_LAUNCH_AS_ROOT:-1}" export POSEIDON_E2E_REQUIRED_NODES="${POSEIDON_E2E_REQUIRED_NODES:-Diagnostics}" +export POSEIDON_E2E_INSTALL_GPSD_CLIENT="${POSEIDON_E2E_INSTALL_GPSD_CLIENT:-1}" mkdir -p "${POSEIDON_E2E_ARTIFACT_DIR}" @@ -86,6 +87,72 @@ cleanup() { } trap cleanup EXIT INT TERM +run_with_root() { + if [[ "$(id -u)" -eq 0 ]]; then + "$@" + else + sudo "$@" + fi +} + +integrate_gpsd_client() { + if [[ "${POSEIDON_E2E_INSTALL_GPSD_CLIENT}" != "1" ]]; then + echo "[phase 0] skip gpsd_client integration (POSEIDON_E2E_INSTALL_GPSD_CLIENT=${POSEIDON_E2E_INSTALL_GPSD_CLIENT})" + return 0 + fi + + local workspace_src="${POSEIDON_ROOT}/src/workspace/src" + local workspace_root="${POSEIDON_ROOT}/src/workspace" + local gpsd_repo_url="https://github.com/CIDCO-dev/gps_umd.git" + local gpsd_tmp_dir="/opt/gps_umd" + + if [[ ! -d "${workspace_src}" ]]; then + echo "[!] Workspace source directory not found: ${workspace_src}" + exit 1 + fi + + if [[ -d "${workspace_src}/gpsd_client" ]]; then + echo "[phase 0] gpsd_client already integrated" + return 0 + fi + + echo "[phase 0] integrate gpsd_client into Poseidon workspace" + if ! command -v git >/dev/null 2>&1; then + echo "[!] git is required to install gps_umd." + exit 1 + fi + + # Equivalent to the field workflow: clone -> move gps* packages -> cleanup. + run_with_root rm -rf "${gpsd_tmp_dir}" 2>/dev/null || true + run_with_root git clone "${gpsd_repo_url}" "${gpsd_tmp_dir}" + run_with_root env GPSD_TMP_DIR="${gpsd_tmp_dir}" WORKSPACE_SRC="${workspace_src}" bash -lc ' + set -e + shopt -s nullglob + pkgs=("$GPSD_TMP_DIR"/gps*) + if [[ ${#pkgs[@]} -eq 0 ]]; then + echo "[!] No gps* packages found in $GPSD_TMP_DIR." >&2 + exit 1 + fi + mv "${pkgs[@]}" "$WORKSPACE_SRC/" + ' + run_with_root rm -rf "${gpsd_tmp_dir}" + + if ! command -v catkin_make >/dev/null 2>&1; then + echo "[!] catkin_make not found; cannot build workspace after gps_umd install." + exit 1 + fi + + echo "[phase 0] rebuild workspace after gpsd_client integration" + pushd "${workspace_root}" >/dev/null + catkin_make + popd >/dev/null + + if [[ -f "${POSEIDON_ROOT}/src/workspace/devel/setup.bash" ]]; then + # shellcheck disable=SC1091 + source "${POSEIDON_ROOT}/src/workspace/devel/setup.bash" + fi +} + start_http_server() { pushd "${POSEIDON_ROOT}/www/webroot" >/dev/null python3 -m http.server "${POSEIDON_HTTP_PORT}" --bind 127.0.0.1 >"${HTTP_LOG}" 2>&1 & @@ -289,6 +356,7 @@ PY popd >/dev/null } +integrate_gpsd_client phase_start_ros phase_backend_test phase_ui_test diff --git a/version.md b/version.md index 965dd45c..a9c5bc00 100644 --- a/version.md +++ b/version.md @@ -1,5 +1,10 @@ # Version History (English) +## 2026-02-19 +- e2e: removed the internal fake sonar DBT writer from `test/e2e/run_poseidon_e2e.sh`; E2E now expects a real sonar or an external sonar simulator via `POSEIDON_SONAR_DEVICE`. +- e2e: added an automatic pre-test GPSD client integration phase (`POSEIDON_E2E_INSTALL_GPSD_CLIENT=1` by default) that clones `gps_umd`, moves `gps*` packages into `src/workspace/src`, and rebuilds with `catkin_make` when `gpsd_client` is missing. +- docs: updated `docs/test/hardware_e2e.md` to reflect external sonar usage and the new automatic GPSD integration behavior. + ## 2026-01-12 - CI: clean Catkin build/devel/logs on self-hosted runners before builds to avoid `.built_by` permission errors. - CI: pre-clean the GitHub Actions workspace with sudo before checkout to avoid permission errors during repository cleanup.