diff --git a/AnylogNetworkSetup/AnyLog-Setup.md b/AnylogNetworkSetup/AnyLog-Setup.md new file mode 100644 index 0000000..77f9a4e --- /dev/null +++ b/AnylogNetworkSetup/AnyLog-Setup.md @@ -0,0 +1,599 @@ +# AnyLog Setup Guide (Apple Silicon) + +This is the **primary method** for running EdgeFL with AnyLog on Apple Silicon Macs. For the EdgeLake alternative, see [EdgeLake-Setup.md](./EdgeLake-Setup.md). + +--- + +## Table of Contents + +1. [Prerequisites](#1-prerequisites) +2. [Docker Image Setup](#2-docker-image-setup) +3. [Docker-Compose Repository](#3-docker-compose-repository) +4. [License Key & Credentials](#4-license-key--credentials) +5. [AnyLog Configuration](#5-anylog-configuration) +6. [PostgreSQL Setup](#6-postgresql-setup) +7. [Start AnyLog Nodes](#7-start-anylog-nodes) +8. [EdgeFL Environment Setup](#8-edgefl-environment-setup) +9. [Insert MNIST Data](#9-insert-mnist-data) +10. [Run Training](#10-run-training) +11. [Inference](#11-inference) +12. [Web GUI (Optional)](#12-web-gui-optional) +13. [Quick Start (Returning Users)](#13-quick-start-returning-users) +14. [Cleanup](#14-cleanup) +15. [Troubleshooting](#15-troubleshooting) + +--- + +## 1. Prerequisites + +- **Docker Desktop** — installed and running +- **AnyLog Docker image** — Currently not publicly available. For those interested in trying AnyLog, fill out the form at https://www.anylog.network/download to receive a license key and email roy[at]anylog[dot]co for more information. +- **docker-compose (latest)** — install via Homebrew (`brew install docker-compose`); older versions may cause issues +- **Python 3.10+** with `venv` support + +--- + +## 2. Docker Image Setup + +Load the AnyLog Docker image (ARM or AMD) and verify it: + +```bash +docker load < /path/to/AnyLog-[ARCHITECTURE].tar + +docker image ls | grep anylog +# Expected: anylogco/anylog-network [TAG] ~789MB +``` + +Note the image tag from the output — you'll need it in the configuration step. + +--- + +## 3. Docker-Compose Repository + +> **Note:** These setup instructions are for **local environment only**. For instructions on deploying AnyLog globally, reach out to the AnyLog team. + +Clone and check out the correct branch: + +```bash +git clone https://github.com/AnyLog-co/docker-compose.git +cd docker-compose +git checkout roy-local +git pull origin roy-local +``` + +--- + +## 4. License Key & Credentials + +### Create the credentials file + +Export your AnyLog license key to a file in a directory of your choice. For example: + +```bash +mkdir -p [YOUR_CREDENTIALS_DIRECTORY] +``` + +Create `[YOUR_CREDENTIALS_DIRECTORY]/.anylog_key.env` with your license key: + +``` +ANYLOG_LICENSE="56b3d57....269ff{'company':'Guest','expiration':'somedate','type':'beta'}" +``` + +### Source the key + +Add the following to your shell profile (e.g. `~/.zshrc`) so it loads automatically: + +```bash +source [YOUR_CREDENTIALS_DIRECTORY]/.anylog_key.env +export ANYLOG_LICENSE +``` + +Otherwise, source it manually in each terminal session. Verify with: + +```bash +echo $ANYLOG_LICENSE +``` + +--- + +## 5. AnyLog Configuration + +All commands below assume you are in the `docker-compose` repository root. + +### 5a. Set the Docker image tag + +In the repo's `Makefile`, change line 7 to match your AnyLog image tag: + +```makefile +export TAG ?= [YOUR_IMAGE_TAG] +``` + +To find your image tag, run `docker image ls | grep anylog` and use the tag from the output. + +### 5b. Set the license key in the master config + +In `docker-makefiles/master-configs/base_configs.env`, change: + +``` +LICENSE_KEY="" +``` + +to: + +``` +LICENSE_KEY=$ANYLOG_LICENSE +``` + +### 5c. Set required base config settings + +In every base config file (master and operators: `docker-makefiles/master-configs/base_configs.env`, `docker-makefiles/operator1-configs/base_configs.env`, etc.), ensure the following are set: + +```env +TCP_BIND=true +REST_BIND=true +BROKER_BIND=true +``` +Otherwise, you might experience issues with network connectivity with multiple IP addresses per node. + +### 5d. Disable NoSQL/Blobs in each operator config + +In every operator base config file (e.g. `docker-makefiles/operator1-configs/base_configs.env`, etc.), set: + +```env +ENABLE_NOSQL=false +BLOBS_DBMS=false +BLOBS_STORAGE=false +``` + +--- + +## 6. PostgreSQL Setup + +From the `docker-compose` repo, start three PostgreSQL containers (one per operator): + +```bash +cd support-tools/postgres + +make up NAME=postgres1 HOST_PORT=5432 VOLUME=pgdata1 +make up NAME=postgres2 HOST_PORT=5433 VOLUME=pgdata2 +make up NAME=postgres3 HOST_PORT=5434 VOLUME=pgdata3 +``` + +--- + +## 7. Start AnyLog Nodes + +Return to the `docker-compose` repo root. Make sure the license is exported: + +```bash +export ANYLOG_LICENSE +``` + +### 7a. Start the master node + +```bash +make up ANYLOG_TYPE=master +``` + +**Validate:** + +```bash +docker attach master +# Press Enter a few times, then type: +test network +``` + +Expected output: + +``` +Address Node Type Node Name Status +---------------|---------|---------|------| +127.0.0.1:32048|master |master | + | +``` + +Detach with `Ctrl+P`, `Ctrl+Q`. + +### 7b. Start the operator nodes + +```bash +make up ANYLOG_TYPE=operator1 +make up ANYLOG_TYPE=operator2 +make up ANYLOG_TYPE=operator3 +``` + +**Validate** (attach to any operator): + +```bash +docker attach operator1 +# wait a few seconds, then press Enter a few times, then type: +test network +``` + +Expected output — all four nodes with `+` status: + +``` +Address Node Type Node Name Status +---------------|---------|---------|------| +127.0.0.1:32048|master |master | + | +127.0.0.1:32148|operator |operator1| + | +127.0.0.1:32248|operator |operator2| + | +127.0.0.1:32348|operator |operator3| + | +``` + +### 7c. Verify databases + +While attached to an operator: + +``` +get databases +``` + +You should see `almgm` and `system_query`, and either `customers` or `mnist_fl`: + +``` +Active DBMS Connections +Logical DBMS Database Type Owner IP:Port Configuration Storage +------------|-------------|------|--------------|-------------------------------------------------|----------| +almgm |psql |system|127.0.0.1:5432|Autocommit On, Fsync on |Persistent| +mnist_fl |psql |user |127.0.0.1:5432|Autocommit On, Fsync on |Persistent| +system_query|sqlite |system|Local |Autocommit On, RAM, Fsync full (after each write)|MEMORY | +``` + +### 7d. Verify external access + +From the host machine in a separate terminal: + +```bash +curl --location --request GET http://127.0.0.1:32149 \ + --header "User-Agent: AnyLog/1.23" \ + --header "command: get status" +``` + +Expected output for example operator1: + +``` +operator1@127.0.0.1:32148 running +``` + +> **If this fails:** In Docker Desktop → Settings → Resources → Network, enable **host networking**. Restart Docker. + +--- + +## 8. EdgeFL Environment Setup + +### 8a. Python environment + +```bash +cd /path/to/EdgeFL +python3 -m venv .venv +source .venv/bin/activate +pip3 install -r requirements.txt +pip3 install torch torchvision requests tensorflow scikit-learn dotenv "python-dotenv[cli]" uvicorn fastapi docker requests_toolbelt +``` + +### 8b. Configure env files + +Edit the files in `edgefl/env_files/mnist/`. Each file needs: + +- **`GITHUB_DIR`** — absolute path to the EdgeFL repo on your local machine + +The port mappings are: + +| File | `EXTERNAL_IP` (REST) | `EXTERNAL_TCP_IP_PORT` (TCP) | +|----------------|----------------------|------------------------------| +| `mnist-agg.env`| `127.0.0.1:32049` | `127.0.0.1:32048` | +| `mnist1.env` | `127.0.0.1:32149` | `127.0.0.1:32148` | +| `mnist2.env` | `127.0.0.1:32249` | `127.0.0.1:32248` | +| `mnist3.env` | `127.0.0.1:32349` | `127.00.0.1:32348` | + +> **Note:** If you get SQL query connection errors, try setting `EXTERNAL_TCP_IP_PORT="network"` instead of the IP:port combo. + +Make sure all env files use the same data handler (`MODULE_NAME=MnistDataHandler`, `MODULE_FILE=custom_data_handler.py` or `MODULE_NAME=MnistDataHandler`, `MODULE_FILE=mnist_data_handler.py`) + +--- + +## 9. Insert MNIST Data + +### 9a. Fix `store_data.py` (if needed) + +In `edgefl/data/mnist/store_data.py`, change the two instances of: + +```python +"image": img.numpy().flatten().tolist(), +``` + +to: + +```python +"image": json.dumps(img.numpy().flatten().tolist()), +``` + +### 9b. Insert data into each operator + +Insert data into each operator using the logical database name (in this case, `mnist_fl`): + +```bash +cd edgefl/data/mnist + +python3 store_data.py 127.0.0.1:32149 --db-name mnist_fl +python3 store_data.py 127.0.0.1:32249 --db-name mnist_fl +python3 store_data.py 127.0.0.1:32349 --db-name mnist_fl +``` + +### 9c. Validate + +Attach to any operator and run: + +``` +blockchain get table +``` + +You should see two tables: `mnist_train` and `mnist_test`. + +Further verification: + +``` +get tables where dbms = mnist_fl +get data nodes +get streaming +``` +should all output non-empty results + +--- + +## 10. Run Training + +You need **4 terminal windows**, all with the venv activated. Run each command in a separate terminal from `edgefl/`: + +**Terminal 1 — Aggregator:** + +```bash +python3 -m dotenv -f env_files/mnist/mnist-agg.env run -- \ + python3 -m uvicorn platform_components.aggregator.aggregator_server:app --host 0.0.0.0 --port 8080 +``` + +**Terminal 2 — Node 1:** + +```bash +dotenv -f env_files/mnist/mnist1.env run -- \ + uvicorn platform_components.node.node_server:app --host 0.0.0.0 --port 8081 +``` + +**Terminal 3 — Node 2:** + +```bash +dotenv -f env_files/mnist/mnist2.env run -- \ + uvicorn platform_components.node.node_server:app --host 0.0.0.0 --port 8082 +``` + +**Terminal 4 — Node 3:** + +```bash +dotenv -f env_files/mnist/mnist3.env run -- \ + uvicorn platform_components.node.node_server:app --host 0.0.0.0 --port 8083 +``` + +All should show `Uvicorn running on http://0.0.0.0:80XX` with no errors. + +### 10a. Initialize + +These steps can also be done through the GUI (see [Section 12](#12-web-gui-optional-but-recommended)). + +```bash +curl -X POST http://localhost:8080/init \ + -H "Content-Type: application/json" \ + -d '{ + "nodeUrls": [ + "http://localhost:8081", + "http://localhost:8082", + "http://localhost:8083" + ], + "index": "test-index" + }' +``` + +Each node terminal should print: + +``` +[INFO] nodeX successfully initialized for (test-index) +[INFO] [test-index][Round 1] Listening for start round 1 +``` + +### 10b. Start training + +```bash +curl -X POST http://localhost:8080/start-training \ + -H "Content-Type: application/json" \ + -d '{ + "totalRounds": 10, + "minParams": 3, + "index": "test-index" + }' +``` + +- **`totalRounds`** — number of federated rounds +- **`minParams`** — how many node weights the aggregator waits for before aggregating + +--- + +## 11. Inference + +After training completes, test each node: + +```bash +curl -X POST http://localhost:8081/inference/test-index +curl -X POST http://localhost:8082/inference/test-index +curl -X POST http://localhost:8083/inference/test-index +``` + +Example response: + +```json +{"index":"test-index","status":"success","message":"Inference completed successfully","model_accuracy":"92.0"} +``` + +--- + +## 12. Web GUI (Optional but Recommended) + +Instead of using `curl`, you can use the built-in GUI. + +From `EdgeFL/`: +```bash +cd gui/edgefl-gui +npm install +npm start +``` + +In the GUI: +1. Enter a test index name and the node URLs (`http://localhost:8081`, etc.) +2. Click **Init** — all nodes should succeed +3. Click **Next**, configure training parameters, and click **Start Training** +4. Once training is complete, click **Inference**. +5. Update the top right corner to the node URL/Port you used for an operator, NOT the aggregator. +6. Test any of the inference options provided (drawing recommended). + +--- + +## 13. Quick Start (Returning Users) + +> This assumes all configuration and first-time setup is already done. + +### Start infrastructure + +```bash +# Export license +export ANYLOG_LICENSE + +# PostgreSQL (from docker-compose/support-tools/postgres/) +cd docker-compose/support-tools/postgres/ +make up NAME=postgres1 HOST_PORT=5432 VOLUME=pgdata1 +make up NAME=postgres2 HOST_PORT=5433 VOLUME=pgdata2 +make up NAME=postgres3 HOST_PORT=5434 VOLUME=pgdata3 + +# AnyLog nodes (from docker-compose/) +cd ../ +make up ANYLOG_TYPE=master +make up ANYLOG_TYPE=operator1 +make up ANYLOG_TYPE=operator2 +make up ANYLOG_TYPE=operator3 +``` + +### Insert data + +```bash +cd EdgeFL +source .venv/bin/activate +cd edgefl/data/mnist + +python3 store_data.py 127.0.0.1:32149 --db-name mnist_fl +python3 store_data.py 127.0.0.1:32249 --db-name mnist_fl +python3 store_data.py 127.0.0.1:32349 --db-name mnist_fl +``` + +### Start servers (4 terminals, each with venv activated, from `edgefl/`) + +```bash +# Terminal 1: Aggregator +cd edgefl +python3 -m dotenv -f env_files/mnist/mnist-agg.env run -- \ + python3 -m uvicorn platform_components.aggregator.aggregator_server:app --host 0.0.0.0 --port 8080 + +# Terminal 2: Node 1 +cd edgefl +source ../.venv/bin/activate +dotenv -f env_files/mnist/mnist1.env run -- uvicorn platform_components.node.node_server:app --host 0.0.0.0 --port 8081 + +# Terminal 3: Node 2 +cd edgefl +source ../.venv/bin/activate +dotenv -f env_files/mnist/mnist2.env run -- uvicorn platform_components.node.node_server:app --host 0.0.0.0 --port 8082 + +# Terminal 4: Node 3 +cd edgefl +source ../.venv/bin/activate +dotenv -f env_files/mnist/mnist3.env run -- uvicorn platform_components.node.node_server:app --host 0.0.0.0 --port 8083 +``` + +### Train & infer + +```bash +# Init +curl -X POST http://localhost:8080/init \ + -H "Content-Type: application/json" \ + -d '{"nodeUrls":["http://localhost:8081","http://localhost:8082","http://localhost:8083"],"index":"test-index"}' + +# Train +curl -X POST http://localhost:8080/start-training \ + -H "Content-Type: application/json" \ + -d '{"totalRounds":10,"minParams":3,"index":"test-index"}' + +# Inference (after training completes) +curl -X POST http://localhost:8081/inference/test-index +``` + + + + +--- + +## 14. Cleanup + +### Stop AnyLog nodes (from `docker-compose/`) + +```bash +make clean ANYLOG_TYPE=master +make clean ANYLOG_TYPE=operator1 +make clean ANYLOG_TYPE=operator2 +make clean ANYLOG_TYPE=operator3 +``` + +### Stop PostgreSQL (from `docker-compose/support-tools/postgres/`) + +``` +cd support-tools/postgres/ +``` + +Without deleting data: + +```bash +make down NAME=postgres1 +make down NAME=postgres2 +make down NAME=postgres3 +``` + +With deleting data: + +```bash +make clean NAME=postgres1 VOLUME=pgdata1 +make clean NAME=postgres2 VOLUME=pgdata2 +make clean NAME=postgres3 VOLUME=pgdata3 +``` + +### Remove leftover Docker volumes (shouldn't be necessary with make clean) + +```bash +docker volume ls +docker volume rm $(docker volume ls -q | grep '^docker-compose-files_') +``` + +--- + +## 15. Troubleshooting + +| Problem | Solution | +|---------|----------| +| `curl` to node returns connection error | Enable **host networking** in Docker Desktop settings and restart Docker | +| `TypeError: expected string or bytes-like object, got 'list'` in `store_data.py` | Wrap the image list with `json.dumps()` (see [Section 9a](#9a-fix-store_datapyif-needed)) | +| SQL query connection error from nodes | Set `EXTERNAL_TCP_IP_PORT="network"` in the mnist env files | +| Missing `tensorflow` or `scikit-learn` errors | `pip3 install tensorflow scikit-learn` in your venv | +| `docker-compose` command fails | Install the latest version via `brew install docker-compose` | +| `mnist_fl` database not visible in `get databases` | Manually connect: `connect dbms mnist_fl where type = psql and user = demo and password = passwd and ip = 127.0.0.1 and port = 5432 and memory = true` | + +Some individuals might experience issues with the ports/IP addresses not setting up correctly. If you're not on Linux, you might need to change these lines in the `update_docker_compose.sh` file to be uncommented: +``` +if [[ "$(uname -s)" != "Linux" ]] ; then + TEMPLATE_COMPOSE_FILE="docker-makefiles/docker-compose-template-ports-base.yaml" +fi +``` +I left mine commented and had no issue, but for others, uncommenting these lines might help. \ No newline at end of file diff --git a/AnylogNetworkSetup/EdgeLake-Setup.md b/AnylogNetworkSetup/EdgeLake-Setup.md new file mode 100644 index 0000000..75cd252 --- /dev/null +++ b/AnylogNetworkSetup/EdgeLake-Setup.md @@ -0,0 +1,590 @@ +# EdgeLake Setup Guide (Apple Silicon) + +This is the **alternative method** using EdgeLake (open-source) instead of AnyLog. For the primary AnyLog method, see [AnyLog-Setup.md](./AnyLog-Setup.md). + +EdgeLake does not require a license key but uses Docker bridge networking, so you must discover container IPs and update config files accordingly. + +--- + +## Table of Contents + +1. [Prerequisites](#1-prerequisites) +2. [PostgreSQL Setup](#2-postgresql-setup) +3. [Pull the EdgeLake Image](#3-pull-the-edgelake-image) +4. [Makefile Configuration](#4-makefile-configuration) +5. [Start the Master Node](#5-start-the-master-node) +6. [Discover Container IPs](#6-discover-container-ips) +7. [Configure Operator Env Files](#7-configure-operator-env-files) +8. [Start Operator Nodes](#8-start-operator-nodes) +9. [Validate the Network](#9-validate-the-network) +10. [EdgeFL Environment Setup](#10-edgefl-environment-setup) +11. [Insert MNIST Data](#11-insert-mnist-data) +12. [Run Training](#12-run-training) +13. [Inference](#13-inference) +14. [Quick Start (Returning Users)](#14-quick-start-returning-users) +15. [Cleanup](#15-cleanup) +16. [Troubleshooting](#16-troubleshooting) + +--- + +## 1. Prerequisites + +- **Docker Desktop** — installed and running +- **Python 3.10+** with `venv` support +- **`envsubst`** — typically included with `gettext` (`brew install gettext`) + +No license key is required for EdgeLake. + +--- + +## 2. PostgreSQL Setup + +From the EdgeFL repo: + +```bash +cd EdgeLake/postgres + +make up NAME=postgres1 HOST_PORT=5432 VOLUME=pgdata1 +make up NAME=postgres2 HOST_PORT=5433 VOLUME=pgdata2 +make up NAME=postgres3 HOST_PORT=5434 VOLUME=pgdata3 +``` + +Default credentials (from the Makefile): user=`demo`, password=`passwd`, database=`mnist_fl`. + +--- + +## 3. Pull the EdgeLake Image + +Pull the image ahead of time so deploys are fast: + +```bash +docker pull anylogco/edgelake:latest +``` + +### Prevent image removal on cleanup + +In `EdgeLake/Makefile`, the `clean` target uses `--rmi all` by default, which removes all images. If you want to keep the pulled image between runs, remove the `--rmi all` flag from the `clean` target. + +--- + +## 4. Makefile Configuration + +In `EdgeLake/Makefile`, the ARM64 tag override should be **commented out** so that `TAG=latest` is used (the `latest` tag already includes ARM64 support): + +```makefile +export TAG := latest +# ifeq ($(ARCH),aarch64) +# export TAG := latest-arm64 +# else ifeq ($(ARCH),arm64) +# export TAG := latest-arm64 +# endif +``` + +Also ensure each EdgeLake env file (`EdgeLake/docker_makefile/edgelake_*.env`) contains: + +```env +INIT_TYPE=prod +``` + +The `TAG` is already set by the Makefile, so it does not need to be in the env files. + +--- + +## 5. Start the Master Node + +From `EdgeLake/`: + +```bash +make up EDGELAKE_TYPE=master TAG=latest \ + EDGELAKE_SERVER_PORT=32048 EDGELAKE_REST_PORT=32049 NODE_NAME=master +``` + +Wait ~30 seconds for the node to initialize. + +--- + +## 6. Discover Container IPs + +EdgeLake uses Docker bridge networking, so you must find the internal IPs of each container. + +### Master IP + +```bash +docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' master +``` + +Example output: `172.17.0.2` + +### PostgreSQL IPs + +```bash +docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' postgres1 +docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' postgres2 +docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' postgres3 +``` + +Example outputs: `172.18.0.2`, `172.18.0.3`, `172.18.0.4` + +> **Write these IPs down** — you will use them in the next step. + +--- + +## 7. Configure Operator Env Files + +All env files are in `EdgeLake/docker_makefile/`. + +### 7a. Set `LEDGER_CONN` (all files) + +Set `LEDGER_CONN` to the **master's Docker IP** and TCP port in **all four** env files: + +| File | Value (example) | +|----------------------------|-----------------------------| +| `edgelake_master.env` | `LEDGER_CONN=172.17.0.2:32048` | +| `edgelake_operator1.env` | `LEDGER_CONN=172.17.0.2:32048` | +| `edgelake_operator2.env` | `LEDGER_CONN=172.17.0.2:32048` | +| `edgelake_operator3.env` | `LEDGER_CONN=172.17.0.2:32048` | + +> **Important:** The master env file must also have `LEDGER_CONN` set, otherwise you will see `Metadata without network peers` errors. + +### 7b. Set `DB_IP` (operator files only) + +Map each operator to its corresponding PostgreSQL container IP: + +| File | `DB_IP` (example) | +|----------------------------|---------------------| +| `edgelake_operator1.env` | `172.18.0.2` (postgres1) | +| `edgelake_operator2.env` | `172.18.0.3` (postgres2) | +| `edgelake_operator3.env` | `172.18.0.4` (postgres3) | + +### 7c. Set `DB_PORT` (operator files) + +All operators should use port `5432` (the **internal** container port), not the host-mapped ports: + +```env +DB_PORT=5432 +``` + +> **Why:** EdgeLake containers communicate with PostgreSQL over Docker's internal network, so they use the container's internal port (`5432`), not the host-mapped ports (`5432`/`5433`/`5434`). + +### 7d. Operator port assignments + +Each operator env file should have unique ports: + +| File | `ANYLOG_SERVER_PORT` | `ANYLOG_REST_PORT` | +|----------------------------|---------------------|--------------------| +| `edgelake_operator1.env` | `32148` | `32149` | +| `edgelake_operator2.env` | `32248` | `32249` | +| `edgelake_operator3.env` | `32348` | `32349` | + +### 7e. Disable NoSQL/Blobs (if present) + +Comment out or set to `false`: + +```env +#ENABLE_NOSQL=false +#BLOBS_DBMS=false +#BLOBS_STORAGE=false +``` + +--- + +## 8. Start Operator Nodes + +From `EdgeLake/`: + +```bash +make up EDGELAKE_TYPE=operator TAG=latest \ + EDGELAKE_SERVER_PORT=32148 EDGELAKE_REST_PORT=32149 NODE_NAME=operator1 + +make up EDGELAKE_TYPE=operator TAG=latest \ + EDGELAKE_SERVER_PORT=32248 EDGELAKE_REST_PORT=32249 NODE_NAME=operator2 + +make up EDGELAKE_TYPE=operator TAG=latest \ + EDGELAKE_SERVER_PORT=32348 EDGELAKE_REST_PORT=32349 NODE_NAME=operator3 +``` + +Wait ~30 seconds for each node to initialize. + +--- + +## 9. Validate the Network + +### 9a. Master node + +```bash +docker attach master +# Press Enter a few times, then type: +test network +``` + +Expected: + +``` +Address Node Type Node Name Status +----------------|---------|---------|------| +172.17.0.2:32048|master |master | + | +``` + +Detach with `Ctrl+P`, `Ctrl+Q`. + +### 9b. Full network (from any operator) + +```bash +docker attach operator1 +# Press Enter, then: +test network +``` + +Expected — all four nodes with `+`: + +``` +Address Node Type Node Name Status +----------------|---------|---------|------| +172.17.0.2:32048|master |master | + | +172.17.0.3:32148|operator |operator1| + | +172.17.0.4:32248|operator |operator2| + | +172.17.0.5:32348|operator |operator3| + | +``` + +### 9c. Check databases + +``` +get databases +``` + +Expected (on an operator): + +``` +Active DBMS Connections +Logical DBMS Database Type Owner IP:Port Configuration Storage +------------|-------------|------|---------------|--------------------------------------------------|----------| +almgm |psql |system|172.18.0.2:5432|Autocommit On, Fsync on |Persistent| +mnist_fl |psql |user |172.18.0.2:5432|Autocommit Off, Fsync on |Persistent| +system_query|sqlite |system|Local |Autocommit On, RAM, Fsync full (after each write) |MEMORY | +``` + +If `mnist_fl` is missing, manually connect it: + +``` +connect dbms mnist_fl where type = psql and user = demo and password = passwd and ip = and port = 5432 and memory = true +``` + +--- + +## 10. EdgeFL Environment Setup + +### 10a. Python environment + +```bash +cd /path/to/EdgeFL +python3 -m venv .venv +source .venv/bin/activate +pip3 install -r requirements.txt +pip3 install torch torchvision requests tensorflow scikit-learn dotenv "python-dotenv[cli]" uvicorn fastapi docker requests_toolbelt +``` + +### 10b. Configure EdgeFL env files + +Edit the files in `edgefl/env_files/mnist/`. Set `GITHUB_DIR` to your EdgeFL repo path in each file. + +Port mappings (using `127.0.0.1` since ports are forwarded to the host): + +| File | `EXTERNAL_IP` (REST) | `EXTERNAL_TCP_IP_PORT` (TCP) | +|----------------|----------------------|------------------------------| +| `mnist-agg.env`| `127.0.0.1:32049` | `127.0.0.1:32048` | +| `mnist1.env` | `127.0.0.1:32149` | `127.0.0.1:32148` | +| `mnist2.env` | `127.0.0.1:32249` | `127.0.0.1:32248` | +| `mnist3.env` | `127.0.0.1:32349` | `127.0.0.1:32348` | + +> **Note:** If you encounter SQL query connection errors, try setting `EXTERNAL_TCP_IP_PORT="network"` instead. + +Make sure all env files use the same data handler (`MODULE_NAME=MnistDataHandler`, `MODULE_FILE=custom_data_handler.py` or `MODULE_NAME=MnistDataHandler`, `MODULE_FILE=mnist_data_handler.py`). + +--- + +## 11. Insert MNIST Data + +### 11a. Fix `store_data.py` (if needed) + +In `edgefl/data/mnist/store_data.py`, change the two instances of: + +```python +"image": img.numpy().flatten().tolist(), +``` + +to: + +```python +"image": json.dumps(img.numpy().flatten().tolist()), +``` + +### 11b. Insert data + +```bash +cd edgefl/data/mnist + +python3 store_data.py 127.0.0.1:32149 --db-name mnist_fl +python3 store_data.py 127.0.0.1:32249 --db-name mnist_fl +python3 store_data.py 127.0.0.1:32349 --db-name mnist_fl +``` + +### 11c. Validate + +Attach to any operator and verify: + +``` +blockchain get table +# Should show mnist_train and mnist_test + +get tables where dbms = mnist_fl +# Should list mnist_train and mnist_test with V under Local DBMS + +get data nodes +# Should show all three operators hosting both tables + +get streaming +# Should show rows ingested for mnist_train and mnist_test +``` + +--- + +## 12. Run Training + +Open **4 terminal windows**, each with the venv activated. All commands run from `edgefl/`. + +**Terminal 1 — Aggregator:** + +```bash +python3 -m dotenv -f env_files/mnist/mnist-agg.env run -- \ + python3 -m uvicorn platform_components.aggregator.aggregator_server:app --host 0.0.0.0 --port 8080 +``` + +**Terminal 2 — Node 1:** + +```bash +dotenv -f env_files/mnist/mnist1.env run -- \ + uvicorn platform_components.node.node_server:app --host 0.0.0.0 --port 8081 +``` + +**Terminal 3 — Node 2:** + +```bash +dotenv -f env_files/mnist/mnist2.env run -- \ + uvicorn platform_components.node.node_server:app --host 0.0.0.0 --port 8082 +``` + +**Terminal 4 — Node 3:** + +```bash +dotenv -f env_files/mnist/mnist3.env run -- \ + uvicorn platform_components.node.node_server:app --host 0.0.0.0 --port 8083 +``` + +### Initialize + +These steps can also be done through the GUI (see [Section 14](#14-web-gui-optional-but-recommended)). + +```bash +curl -X POST http://localhost:8080/init \ + -H "Content-Type: application/json" \ + -d '{ + "nodeUrls": [ + "http://localhost:8081", + "http://localhost:8082", + "http://localhost:8083" + ], + "index": "test-index" + }' +``` + +### Start training + +```bash +curl -X POST http://localhost:8080/start-training \ + -H "Content-Type: application/json" \ + -d '{ + "totalRounds": 10, + "minParams": 3, + "index": "test-index" + }' +``` + +--- + +## 13. Inference + +```bash +curl -X POST http://localhost:8081/inference/test-index +curl -X POST http://localhost:8082/inference/test-index +curl -X POST http://localhost:8083/inference/test-index +``` + +--- + +## 14. Web GUI (Optional but Recommended) + +Instead of using `curl`, you can use the built-in GUI. + +From `EdgeFL/`: +```bash +cd gui/edgefl-gui +npm install +npm start +``` + +In the GUI: +1. Enter a test index name and the node URLs (`http://localhost:8081`, etc.) +2. Click **Init** — all nodes should succeed +3. Click **Next**, configure training parameters, and click **Start Training** +4. Once training is complete, click **Inference**. +5. Update the top right corner to the node URL/Port you used for an operator, NOT the aggregator. +6. Test any of the inference options provided (drawing recommended). + +--- + +## 15. Quick Start (Returning Users) + +> Assumes all configuration is already done. + +### Start infrastructure + +```bash +# PostgreSQL +cd EdgeLake/postgres +make up NAME=postgres1 HOST_PORT=5432 VOLUME=pgdata1 +make up NAME=postgres2 HOST_PORT=5433 VOLUME=pgdata2 +make up NAME=postgres3 HOST_PORT=5434 VOLUME=pgdata3 + +# EdgeLake nodes +cd ../ +make up EDGELAKE_TYPE=master TAG=latest EDGELAKE_SERVER_PORT=32048 EDGELAKE_REST_PORT=32049 NODE_NAME=master +make up EDGELAKE_TYPE=operator TAG=latest EDGELAKE_SERVER_PORT=32148 EDGELAKE_REST_PORT=32149 NODE_NAME=operator1 +make up EDGELAKE_TYPE=operator TAG=latest EDGELAKE_SERVER_PORT=32248 EDGELAKE_REST_PORT=32249 NODE_NAME=operator2 +make up EDGELAKE_TYPE=operator TAG=latest EDGELAKE_SERVER_PORT=32348 EDGELAKE_REST_PORT=32349 NODE_NAME=operator3 +``` + +### Verify IPs (update env files if changed) + +```bash +docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' master +docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' postgres1 +``` + +> **Important:** Docker may assign different IPs on restart. Always re-check container IPs and update `LEDGER_CONN` and `DB_IP` if they changed. + +### Insert data + +```bash +cd ../../EdgeFL +source .venv/bin/activate +cd edgefl/data/mnist + +python3 store_data.py 127.0.0.1:32149 --db-name mnist_fl +python3 store_data.py 127.0.0.1:32249 --db-name mnist_fl +python3 store_data.py 127.0.0.1:32349 --db-name mnist_fl +``` + +### Start servers (4 terminals, each with venv activated, from `edgefl/`) + +```bash +# Terminal 1: Aggregator +cd edgefl +python3 -m dotenv -f env_files/mnist/mnist-agg.env run -- \ + python3 -m uvicorn platform_components.aggregator.aggregator_server:app --host 0.0.0.0 --port 8080 + +# Terminal 2: Node 1 +cd edgefl +source ../.venv/bin/activate +dotenv -f env_files/mnist/mnist1.env run -- uvicorn platform_components.node.node_server:app --host 0.0.0.0 --port 8081 + +# Terminal 3: Node 2 +cd edgefl +source ../.venv/bin/activate +dotenv -f env_files/mnist/mnist2.env run -- uvicorn platform_components.node.node_server:app --host 0.0.0.0 --port 8082 + +# Terminal 4: Node 3 +cd edgefl +source ../.venv/bin/activate +dotenv -f env_files/mnist/mnist3.env run -- uvicorn platform_components.node.node_server:app --host 0.0.0.0 --port 8083 +``` + +### Train & infer + +```bash +# Init +curl -X POST http://localhost:8080/init \ + -H "Content-Type: application/json" \ + -d '{"nodeUrls":["http://localhost:8081","http://localhost:8082","http://localhost:8083"],"index":"test-index"}' + +# Train +curl -X POST http://localhost:8080/start-training \ + -H "Content-Type: application/json" \ + -d '{"totalRounds":10,"minParams":3,"index":"test-index"}' + +# Inference (after training completes) +curl -X POST http://localhost:8081/inference/test-index +``` + +--- + +## 16. Cleanup + +### Stop EdgeLake nodes (from `EdgeLake/`) + +```bash +make clean EDGELAKE_TYPE=master TAG=latest \ + EDGELAKE_SERVER_PORT=32048 EDGELAKE_REST_PORT=32049 NODE_NAME=master + +make clean EDGELAKE_TYPE=operator TAG=latest \ + EDGELAKE_SERVER_PORT=32148 EDGELAKE_REST_PORT=32149 NODE_NAME=operator1 + +make clean EDGELAKE_TYPE=operator TAG=latest \ + EDGELAKE_SERVER_PORT=32248 EDGELAKE_REST_PORT=32249 NODE_NAME=operator2 + +make clean EDGELAKE_TYPE=operator TAG=latest \ + EDGELAKE_SERVER_PORT=32348 EDGELAKE_REST_PORT=32349 NODE_NAME=operator3 +``` + +### Stop PostgreSQL (from `EdgeLake/postgres/`) + +```bash +cd postgres/ +``` + +Without deleting data: + +```bash +make down NAME=postgres1 +make down NAME=postgres2 +make down NAME=postgres3 +``` + +With deleting data: + +```bash +make clean NAME=postgres1 VOLUME=pgdata1 +make clean NAME=postgres2 VOLUME=pgdata2 +make clean NAME=postgres3 VOLUME=pgdata3 +``` + +### Remove leftover Docker volumes (shouldn't be necessary with make clean) + +```bash +docker volume ls +docker volume rm $(docker volume ls -q | grep '^docker-compose-files_') +``` + +--- + +## 17. Troubleshooting + +| Problem | Solution | +|---------|----------| +| `Metadata without network peers` on master | Add `LEDGER_CONN=:32048` to `edgelake_master.env` | +| `Database connect error: mnist_fl using psql failed to connect` on operator2/3 | Set `DB_PORT=5432` in all operator env files (use internal port, not host port) | +| Container IPs changed after restart | Re-run `docker inspect` and update `LEDGER_CONN` and `DB_IP` in all env files | +| `TypeError: expected string or bytes-like object, got 'list'` in `store_data.py` | Wrap image list with `json.dumps()` (see [Section 11a](#11a-fix-store_datapyif-needed)) | +| SQL query connection error from EdgeFL nodes | Set `EXTERNAL_TCP_IP_PORT="network"` in the mnist env files | +| Missing `tensorflow` or `scikit-learn` | `pip3 install tensorflow scikit-learn` in your venv | +| `mnist_fl` database not visible | Manually connect inside the operator shell (see [Section 9c](#9c-check-databases)) | +| ARM64 image issues | Ensure `TAG=latest` in the Makefile and that the ARM override lines are commented out | diff --git a/README.md b/README.md index 2887854..df077ed 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,8 @@ ## Overview +EdgeFL enables federated learning across an AnyLog or EdgeLake network that is composed of data distributed across multiple nodes. + The following is instructions to simulate the continuous Federated Learning (FL) lifecycle consisting of three training nodes and one aggregator node. Each node will utilize its own EdgeLake node, such that we will deploy four EdgeLake nodes, three of which have @@ -9,71 +11,113 @@ operator roles and one with the master role. The master role is a normal EdgeLak node but also emulates the same blockchain-like functionality of the blockchain-back shared metadata layer. For more information about EdgeLake and how it operates, check the [EdgeLake website](https://edgelake.github.io/). -There are three demos supported, where each operator will locally train a ML model -utilizing its local data and EdgeFL will dynamically facilitate model sharing and aggregation -via the aggregator node. The value here is that there is no data movement. -Since the demo instructions -are for a single machine, although they can be easily adapted to execute on multiple -physical machine, we will deploy multiple Postgres databases, one for each EdgeLake operator, -to emulate physically distributed data. Nevertheless, each node will utilize its own EdgeLake -operator node (running in a Docker container) to truly simulate a distributed environment. +In this README we provide an example of a 3 training node and 1 aggregator setup, but note that you can include as many nodes in the training process as needed. This README describes three demos where each operator locally trains a model using its own data. EdgeFL dynamically facilitates model sharing and aggregation via the aggregator—**no data movement required**. We currently have three demos available for users to try EdgeFL and we also provide instructions on creating your own demo or production deployment ([see writing your own data handler](Demo-READMEs/Writing-A-Custom-DataHandler.md)). + +Since these instructions target a single machine (adaptable to multiple), we deploy multiple Postgres databases (one per operator) to emulate distributed data. Each node runs in a Docker container to simulate a true distributed environment. + +**Supported backends:** +- **AnyLog** (primary): Requires license key and Docker image. See [AnyLog Setup Guide](AnylogNetworkSetup/AnyLog-Setup.md) for detailed steps. +- **EdgeLake** (alternative, open-source): See [EdgeLake Setup Guide](AnylogNetworkSetup/EdgeLake-Setup.md) for detailed steps. + +For more about EdgeLake, visit the [EdgeLake website](https://edgelake.github.io/). Before you get started, please follow the configuration steps precisely. # Configuration -Assumptions: - - Downloaded / cloned the repository. - - Have Docker installed. -Install all necessary Python packages. Tested on Python3.12. Make sure to add all required Python packages to -your `requirement.txt` file. In addition, you can also utilize a Python virtual environment, see the [venv docs](https://docs.python.org/3/library/venv.html). +## Prerequisites +- Docker Desktop installed and running +- Python 3.12 +- Git +- **AnyLog users only:** AnyLog license key and Docker image (see [AnyLog Setup Guide](AnylogNetworkSetup/AnyLog-Setup.md) for details on obtaining access) + +## Python Environment Setup ```bash cd EdgeFL -pip install -r requirements.txt +python3 -m venv .venv +source .venv/bin/activate +pip3 install -r requirements.txt +pip3 install torch torchvision requests tensorflow scikit-learn dotenv "python-dotenv[cli]" uvicorn fastapi docker requests_toolbelt ``` -## Deploy Postgres container -You will need one Postgres container for each EdgeLake operator. -Postgres will become available on your inet IP address. You can determine this through the `ifconfig` command. +## Deploy Postgres Databases +You need three Postgres containers (one per operator). Postgres will be available on your inet IP (check with `ifconfig`). + +**EdgeLake users:** ```bash -* Start Docker * cd EdgeLake/postgres make up NAME=postgres1 HOST_PORT=5432 VOLUME=pgdata1 make up NAME=postgres2 HOST_PORT=5433 VOLUME=pgdata2 make up NAME=postgres3 HOST_PORT=5434 VOLUME=pgdata3 ``` -Note that you can change the default POSTGRES_USER, POSTGRES_PASSWORD, and POSTGRES_DB in the Makefile -or add them as arguments to the `make up` command. -To kill these services: + +**AnyLog users:** ```bash -make clean NAME=postgres1 -make clean NAME=postgres2 -make clean NAME=postgres3 +cd docker-compose/support-tools/postgres +make up NAME=postgres1 HOST_PORT=5432 VOLUME=pgdata1 +make up NAME=postgres2 HOST_PORT=5433 VOLUME=pgdata2 +make up NAME=postgres3 HOST_PORT=5434 VOLUME=pgdata3 +``` + +You can customize `POSTGRES_USER`, `POSTGRES_PASSWORD`, and `POSTGRES_DB` in the Makefile or as arguments. + +To stop: +```bash +make clean NAME=postgres1 VOLUME=pgdata1 +make clean NAME=postgres2 VOLUME=pgdata2 +make clean NAME=postgres3 VOLUME=pgdata3 ``` -## Deploy EdgeLake Master node +## Deploy Backend Nodes (AnyLog or EdgeLake) + +### Deploy Master Node + +**EdgeLake:** ```bash cd EdgeLake/ make up EDGELAKE_TYPE=master TAG=1.3.2501 EDGELAKE_SERVER_PORT=32048 EDGELAKE_REST_PORT=32049 NODE_NAME=master ``` -Now we need to determine the Master node's Docker IP address. Issue the following command + +**AnyLog:** +```bash +# Export license first +export ANYLOG_LICENSE="" + +cd docker-compose/ +make up ANYLOG_TYPE=master +``` + +Get the master node's Docker IP: ```bash -cd EdgeLake/ docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' master ``` +Save this IP (e.g., `192.1.1.1`) for operator configuration. -With this IP, we can now deploy our three EdgeLake operator nodes. For example, let's assume it's `192.1.1.1`. +### Deploy Operator Nodes -## Deploy EdgeLake Operator node -Update line 61 (LEDGER_CONN) value in the file `EdgeLake/docker_makefile/edgelake_operator1.env` -to be `LEDGER_CONN=192.1.1.1:32048` (note that you do not need to change the port). -In addition, update the `DB_IP` in line 31 with the Docker network IP of the Postgres container. +**Configure operator env files** with the master IP and Postgres IPs: -Do the same for the following files: +**EdgeLake:** Update `LEDGER_CONN` and `DB_IP` in: +- `EdgeLake/docker_makefile/edgelake_operator1.env` - `EdgeLake/docker_makefile/edgelake_operator2.env` - `EdgeLake/docker_makefile/edgelake_operator3.env` -Now we can start the operator nodes. +**AnyLog:** Update similar fields in: +- `docker-compose/docker_makefile/anylog_operator1.env` +- `docker-compose/docker_makefile/anylog_operator2.env` +- `docker-compose/docker_makefile/anylog_operator3.env` + +> **Important for AnyLog users:** Ensure the following base config settings are set correctly in your configuration files: +> ``` +> TCP_BIND=true +> REST_BIND=true +> BROKER_BIND=false +> ``` +> Otherwise, you might experience issues with network connectivity with multiple IP addresses per node. + +**Start operators:** + +**EdgeLake:** ```bash cd EdgeLake/ make up EDGELAKE_TYPE=operator TAG=1.3.2501 EDGELAKE_SERVER_PORT=32148 EDGELAKE_REST_PORT=32149 NODE_NAME=operator1 @@ -81,22 +125,32 @@ make up EDGELAKE_TYPE=operator TAG=1.3.2501 EDGELAKE_SERVER_PORT=32248 EDGELAKE_ make up EDGELAKE_TYPE=operator TAG=1.3.2501 EDGELAKE_SERVER_PORT=32348 EDGELAKE_REST_PORT=32349 NODE_NAME=operator3 ``` -## Validating your EdgeLake network is properly setup -To validate your EdgeLake network is properly setup, execute the following commands: +**AnyLog:** +```bash +cd docker-compose/ +make up ANYLOG_TYPE=operator1 +make up ANYLOG_TYPE=operator2 +make up ANYLOG_TYPE=operator3 +``` + +## Validate Backend Network + +Attach to the master node and test network connectivity: ```bash docker attach master ``` -Hit the [enter/return] key on your keyboard. -You should now see `EL master +>`. -Now type into the CLI `test network` and press [enter/return]. -You should see the following print out: +Press [enter/return]. You should see `EL master +>` (EdgeLake) or `AL master +>` (AnyLog). + +Test network: ```bash -EL master +> test network +test network +``` +Expected output (all nodes show `+` status): +``` Test Network [****************************************************************] -EL master +> Address Node Type Node Name Status ----------------|---------|---------|------| 172.19.0.2:32048|master |master | + | @@ -104,60 +158,24 @@ Address Node Type Node Name Status 172.19.0.4:32248|operator |operator2| + | 172.19.0.5:32348|operator |operator3| + | ``` -The `+` signifies that the nodes are all members of EdgeLake's p2p network. If you do not see that, then -please contact the EdgeLake maintainers through [EdgeLake's Slack Channel](https://lfedge.org/projects/edgelake/) -(the join link is at the bottom of the page). -More over, make sure each operator is connected to the DBMS (the demos utilize the DBMS `mnist_fl`). -```bash -docker attach operator1 -EL operator1 +> get databases - -Active DBMS Connections -Logical DBMS Database Type Owner IP:Port Configuration Storage -------------|-------------|------|------------------|-----------------------------------------------|----------| -almgm |psql |system|192.168.1.125:5433|Autocommit On, Failed to pull Fsync |Persistent| -mnist_fl |psql |user |192.168.1.125:5433|Autocommit Off, Failed to pull Fsync |Persistent| -system_query|psql |system|192.168.1.125:5433|Autocommit Off, Unflagged, Failed to pull Fsync|Persistent| -``` -If you don't see the `mnist_fl` line, then you will need to execute the following EdgeLake CLI command -and re-execute the above command: -```bash -connect dbms mnist_fl where type = psql and user = [user] and password = [password] and ip = [ip] and port = [port] and memory = true -``` - -# Demos -The demo instructions can be found the the `EdgeFL/Demo-READMEs/` directory. - -We currently have two demos: -- [MNIST handwriting dataset demo](Demo-READMEs/MNIST.md) -- [Winniio demo on temperature prediction](Demo-READMEs/WINNIIO.md), a telemetry dataset. Thank you to your partners [Winniio homepage](https://www.winniio.io)! -- [Xray detection bounding box](Demo-READMEs/Chest-Xray-BoundingBox.md) +If nodes don't show `+`, check your `LEDGER_CONN` configuration in operator env files. For EdgeLake support, see [EdgeLake Slack](https://lfedge.org/projects/edgelake/). +### Verify Database Connections -## Resolving common issues -After executing the init `curl` request, if your training nodes do not print out model weights, -then they do not have access to the data. -The first step is to double check that you loaded data into your Postgres instance. -Make sure that you have the following: -1. `mnist_fl` database -2. Make sure there's actually data in those tables. - -Another step to double check is that the EdgeLake operator nodes are connected to the database. -To do so you can docker attach to each container and check. -For example, to check `operator1` is connected to the PSQL database, issue the following commands: +Attach to an operator and check databases: ```bash docker attach operator1 ``` -press [enter/return] so you see `EL master +>`. -Now to validate your PSQL connection, do the following: +Press [enter/return] to see the prompt. + +Check database connections: ```bash get databases ``` -You should see the following: -```bash -EL operator1 +> get databases +Expected output: +``` Active DBMS Connections Logical DBMS Database Type Owner IP:Port Configuration Storage ------------|-------------|------|------------------|-----------------------------------|----------| @@ -166,56 +184,36 @@ mnist_fl |psql |user |192.168.1.125:5432|Autocommit Off, Fsync on system_query|psql |system|192.168.1.125:5432|Autocommit Off, Unflagged, Fsync on|Persistent| ``` -If you don't see `mnist_fl`, then execute the following EdgeLake specific command: +If `mnist_fl` is missing, connect it: ```bash -connect dbms mnist_fl where type = psql and user = demo and password = passwd and ip = 192.1.1.1 and port = 5432 and memory = true +connect dbms mnist_fl where type = psql and user = demo and password = passwd and ip = [postgres-ip] and port = 5432 and memory = true ``` -where the `192.1.1.1` is your inet ip from above. It should output `database connected`. -If not, then your IP may be wrong. -You can also view all your database tables for a certain DBMS +### Verify Tables and Data + +View all tables in the `mnist_fl` database: ```bash -EL operator1 +> get tables where dbms = mnist_fl +get tables where dbms = mnist_fl +``` +Expected output: +``` Database Table name Local DBMS Blockchain --------|----------------------------------------------------|----------|----------| mnist_fl|mnist_test | V | V | |mnist_train | V | V | |par_mnist_train_2025_07_01_d14_insert_timestamp | V | - | - |par_room_12004_test_2025_07_01_d14_insert_timestamp | V | - | - |par_room_12004_train_2025_07_01_d14_insert_timestamp| V | - | - |room_12004_test | V | V | - |room_12004_train | V | V | - |room_12055_test | - | V | - |room_12055_train | - | V | - |room_12090_test | - | V | - |room_12090_train | - | V | ``` -View the tables hosted by each EdgeLake operator: +View tables hosted by each operator: ```bash -EL operator1 +> get data nodes +get data nodes +``` +Expected output: +``` Company DBMS Table Cluster ID Cluster Status Node Name Member ID External IP/Port Local IP/Port Main Node Status -----------|--------|----------------|--------------------------------|--------------|---------|---------|-------------------|----------------|----|-----------| -New Company|mnist_fl|room_12055_train|1a4a2c6f59161cf5f7b242abcacf6ba2|active |operator1| 79|104.60.100.77:32148|172.17.0.3:32148| + |active | - | | | |active |operator2| 208|104.60.100.77:32248|172.17.0.4:32248| + |active | - | | | |active |operator3| 65|104.60.100.77:32348|172.17.0.5:32348| + |active | -New Company|mnist_fl|room_12055_test |1a4a2c6f59161cf5f7b242abcacf6ba2|active |operator1| 79|104.60.100.77:32148|172.17.0.3:32148| + |active | - | | | |active |operator2| 208|104.60.100.77:32248|172.17.0.4:32248| + |active | - | | | |active |operator3| 65|104.60.100.77:32348|172.17.0.5:32348| + |active | -New Company|mnist_fl|room_12004_train|1a4a2c6f59161cf5f7b242abcacf6ba2|active |operator1| 79|104.60.100.77:32148|172.17.0.3:32148| + |active | - | | | |active |operator2| 208|104.60.100.77:32248|172.17.0.4:32248| + |active | - | | | |active |operator3| 65|104.60.100.77:32348|172.17.0.5:32348| + |active | -New Company|mnist_fl|room_12004_test |1a4a2c6f59161cf5f7b242abcacf6ba2|active |operator1| 79|104.60.100.77:32148|172.17.0.3:32148| + |active | - | | | |active |operator2| 208|104.60.100.77:32248|172.17.0.4:32248| + |active | - | | | |active |operator3| 65|104.60.100.77:32348|172.17.0.5:32348| + |active | -New Company|mnist_fl|room_12090_train|1a4a2c6f59161cf5f7b242abcacf6ba2|active |operator1| 79|104.60.100.77:32148|172.17.0.3:32148| + |active | - | | | |active |operator2| 208|104.60.100.77:32248|172.17.0.4:32248| + |active | - | | | |active |operator3| 65|104.60.100.77:32348|172.17.0.5:32348| + |active | -New Company|mnist_fl|room_12090_test |1a4a2c6f59161cf5f7b242abcacf6ba2|active |operator1| 79|104.60.100.77:32148|172.17.0.3:32148| + |active | - | | | |active |operator2| 208|104.60.100.77:32248|172.17.0.4:32248| + |active | - | | | |active |operator3| 65|104.60.100.77:32348|172.17.0.5:32348| + |active | New Company|mnist_fl|mnist_train |1a4a2c6f59161cf5f7b242abcacf6ba2|active |operator1| 79|104.60.100.77:32148|172.17.0.3:32148| + |active | | | | |active |operator2| 208|104.60.100.77:32248|172.17.0.4:32248| + |active | | | | |active |operator3| 65|104.60.100.77:32348|172.17.0.5:32348| + |active | @@ -226,33 +224,155 @@ New Company|mnist_fl|mnist_test |1a4a2c6f59161cf5f7b242abcacf6ba2|active View row count by operator: ```bash -EL operator1 +> get rows count where dbms=mnist_fl +get rows count where dbms=mnist_fl +``` +Expected output: +``` DBMS Name Table Name Rows Count ---------|----------------------------------------------------|----------| mnist_fl |mnist_test | 0| |mnist_train | 0| |par_mnist_train_2025_07_01_d14_insert_timestamp | 50| - |par_room_12004_test_2025_07_01_d14_insert_timestamp | 1576| - |par_room_12004_train_2025_07_01_d14_insert_timestamp| 6300| - |room_12004_test | 0| - |room_12004_train | 0| - -EL operator1 +> run client (192.168.1.125:32148) sql mnist_fl select count(*) from mnist_train -[39] -EL operator1 +> +``` + +Query data directly: +```bash +run client (192.168.1.125:32148) sql mnist_fl select count(*) from mnist_train +``` + +Expected output: +``` {"Query":[{"count(*)":50}], "Statistics":[{"Count": 1, "Time":"00:00:00", "Nodes": 1}]} ``` -Note, to detach from EdgeLake, press ctrl+p+q simultaneously. +**To detach from a container:** Press `Ctrl+P` then `Ctrl+Q` (do NOT use `Ctrl+C` or `exit`). + +# Running EdgeFL Demos + +EdgeFL supports three demos. The following shows the unified setup for MNIST (the most common demo). For other demos and platform-specific details, see the guides below. + +**Detailed setup guides:** +- [AnyLog Setup Guide (Apple Silicon)](AnylogNetworkSetup/AnyLog-Setup.md) — Primary method with license +- [EdgeLake Setup Guide (Apple Silicon)](AnylogNetworkSetup/EdgeLake-Setup.md) — Open-source alternative +- [MNIST demo details](Demo-READMEs/MNIST.md) +- [Winniio temperature prediction](Demo-READMEs/WINNIIO.md) +- [Chest X-ray bounding box](Demo-READMEs/Chest-Xray-BoundingBox.md) + +## MNIST Demo Setup + +### Configure EdgeFL Environment Files + +Edit files in `edgefl/env_files/mnist/` and set: +- `GITHUB_DIR` to your EdgeFL repo path (e.g., `/Users/yourname/Desktop/Code/EdgeFL`) +- `EXTERNAL_IP` and `EXTERNAL_TCP_IP_PORT` to localhost ports (recommended when using port forwarding): + +| File | EXTERNAL_IP (REST) | EXTERNAL_TCP_IP_PORT (TCP) | +|----------------|-------------------|---------------------------| +| mnist-agg.env | 127.0.0.1:32049 | 127.0.0.1:32048 | +| mnist1.env | 127.0.0.1:32149 | 127.0.0.1:32148 | +| mnist2.env | 127.0.0.1:32249 | 127.0.0.1:32248 | +| mnist3.env | 127.0.0.1:32349 | 127.0.0.1:32348 | + +> **Note:** If you get SQL connection errors, try `EXTERNAL_TCP_IP_PORT="network"` instead. + +### Insert MNIST Data + +Insert data into each operator using the logical database name (in this case, `mnist_fl`): + +```bash +cd edgefl/data/mnist +python3 store_data.py 127.0.0.1:32149 --db-name mnist_fl +python3 store_data.py 127.0.0.1:32249 --db-name mnist_fl +python3 store_data.py 127.0.0.1:32349 --db-name mnist_fl +``` + +### Start EdgeFL Servers + +Open 4 terminals and run the following (from `EdgeFL/edgefl`): + +**Terminal 1 — Aggregator:** +```bash +cd edgefl +source ../.venv/bin/activate +python3 -m dotenv -f env_files/mnist/mnist-agg.env run -- \ + python3 -m uvicorn platform_components.aggregator.aggregator_server:app --host 0.0.0.0 --port 8080 +``` + +**Terminal 2 — Node 1:** +```bash +cd edgefl +source ../.venv/bin/activate +dotenv -f env_files/mnist/mnist1.env run -- uvicorn platform_components.node.node_server:app --host 0.0.0.0 --port 8081 +``` + +**Terminal 3 — Node 2:** +```bash +cd edgefl +source ../.venv/bin/activate +dotenv -f env_files/mnist/mnist2.env run -- uvicorn platform_components.node.node_server:app --host 0.0.0.0 --port 8082 +``` + +**Terminal 4 — Node 3:** +```bash +cd edgefl +source ../.venv/bin/activate +dotenv -f env_files/mnist/mnist3.env run -- uvicorn platform_components.node.node_server:app --host 0.0.0.0 --port 8083 +``` + +All should show `Uvicorn running on http://0.0.0.0:80XX` with no errors. -## Redoing simulation / Clean up -To redo the simulation, you need to delete the `edgefl/file_write` directory. -In addition, you need to kill and restart the EdgeLake operators and master node. -To do so, follow the following instructions: +### Train and Infer + +Initialize nodes: +```bash +curl -X POST http://localhost:8080/init \ + -H "Content-Type: application/json" \ + -d '{"nodeUrls":["http://localhost:8081","http://localhost:8082","http://localhost:8083"],"index":"test-index"}' +``` + +Start training: +```bash +curl -X POST http://localhost:8080/start-training \ + -H "Content-Type: application/json" \ + -d '{"totalRounds":10,"minParams":3,"index":"test-index"}' +``` + +Run inference (against a **node**, not the aggregator): +```bash +curl -X POST http://localhost:8081/inference/test-index +``` + +### Web GUI (Optional but Recommended) + +From `EdgeFL/`: +```bash +cd gui/edgefl-gui +npm install +npm start +``` + +In the GUI: +1. Enter test index name and node URLs (`http://localhost:8081`, etc.) +2. Click **Init** — all nodes should succeed +3. Click **Next**, configure training, click **Start Training** +4. After training, click **Inference** +5. Update top-right URL to a **node** (e.g., `http://localhost:8081`), NOT the aggregator +6. Test inference (drawing recommended) + + +## Cleanup + +To stop and clean up: + +**Stop EdgeFL servers:** Press `Ctrl+C` in each terminal. + +**Stop backend nodes:** + +**EdgeLake:** ```bash cd EdgeLake/ make clean EDGELAKE_TYPE=master TAG=1.3.2501 EDGELAKE_SERVER_PORT=32048 EDGELAKE_REST_PORT=32049 NODE_NAME=master @@ -260,15 +380,43 @@ make clean EDGELAKE_TYPE=operator TAG=1.3.2501 EDGELAKE_SERVER_PORT=32148 EDGELA make clean EDGELAKE_TYPE=operator TAG=1.3.2501 EDGELAKE_SERVER_PORT=32248 EDGELAKE_REST_PORT=32249 NODE_NAME=operator2 make clean EDGELAKE_TYPE=operator TAG=1.3.2501 EDGELAKE_SERVER_PORT=32348 EDGELAKE_REST_PORT=32349 NODE_NAME=operator3 ``` -Note that you do not need to restart Postgres. -After this step, if you want to restart the simulation follow the Deploy EdgeLake Operator/Master Node from above. -To stop Postgres: +**AnyLog:** ```bash -cd EdgeLake/postgres -docker compose down +cd docker-compose/ +make clean ANYLOG_TYPE=master +make clean ANYLOG_TYPE=operator1 +make clean ANYLOG_TYPE=operator2 +make clean ANYLOG_TYPE=operator3 +``` + +**Stop Postgres:** +```bash +# EdgeLake users: cd EdgeLake/postgres +# AnyLog users: cd docker-compose/support-tools/postgres +make clean NAME=postgres1 VOLUME=pgdata1 +make clean NAME=postgres2 VOLUME=pgdata2 +make clean NAME=postgres3 VOLUME=pgdata3 +``` + +**Clean EdgeFL artifacts:** +```bash +rm -rf edgefl/file_write/* +rm -rf edgefl/tmp_dir/* ``` +## Troubleshooting + +| Problem | Solution | +|---------|----------| +| Training nodes don't print model weights | Check data was inserted to Postgres (`mnist_fl` database with `mnist_train`/`mnist_test` tables). Verify operator database connections with `get databases` (see [Validate Backend Network](#validate-backend-network)). | +| `curl` to node returns connection error | Enable **host networking** in Docker Desktop settings and restart Docker. | +| Nodes don't show `+` in `test network` | Check `LEDGER_CONN` in operator env files points to correct master IP. | +| `mnist_fl` database not connected | Attach to operator, run `connect dbms mnist_fl where type = psql and user = demo and password = passwd and ip = [postgres-ip] and port = 5432 and memory = true` | +| SQL query connection errors | Try setting `EXTERNAL_TCP_IP_PORT="network"` in env files instead of IP:port. | + +--- + ## Docker Containerization of APIs The APIs are containerized using Docker. Before starting the APIs, ensure that