initial examples

README.md

@@ -4,9 +4,67 @@
 </p>
 
 # What is Heracles?
-This repository provides a library for encoding a [Hydra](https://github.com/MIT-SPARK/Hydra) 3D scene graph into a Neo4j database. It expects a scene graph to initially be loaded from file via [spark\_dsg](https://github.com/MIT-SPARK/Spark-DSG) and provides utilities for translating back and forth between spark\_dsg and the Neo4j database.
 
-## Setup
+This repository provides a library for encoding a
+[Hydra](https://github.com/MIT-SPARK/Hydra) 3D scene graph into a Neo4j
+database. It expects a scene graph to initially be loaded from file via
+[spark\_dsg](https://github.com/MIT-SPARK/Spark-DSG) and provides utilities for
+translating back and forth between spark\_dsg and the Neo4j database.
+
+We provide three examples that should be pretty easy to run. They require that
+Docker is installed, but they should have no other dependencies.
+
+## Minimal Example
+
+To run the first example, run
+```bash
+./heracles_demo.sh
+```
+
+This script will load an example scene graph into the graph database and put
+you in a python terminal where you can run example queries. For example, try
+running the following query to print the location of all trashcans:
+```python
+db.query("MATCH (n: Object {class: 'trash'}) RETURN n.center as center""")
+```
+
+## Interactive ROS Example
+
+The second example will render the current state of the scene graph in the
+database in Rviz. You do _not_ need ROS installed to run this demo. However,
+you will need to specify an OpenAI api key `HERACLES_OPENAI_API_KEY` if you
+want to use the LLM interaction functionality.
+
+```bash
+export HERACLES_OPENAI_API_KEY=<YOUR OPENAI API KEY>
+xhost +local:docker && ./chatdsg_ros_demo.sh
+```
+
+Try asking the LLM how many objects are in the scene graph (press `ctrl-b` to
+send your message to the LLM). Then try telling it to update some of the
+objects (e.g., turn `sign`s into `light`s).
+
+## Interactive Viser Example
+
+Finally, we provide an example of visualizing the scene graph using Viser, so
+that no ROS is necessary. Again, you will need an OpenAI API key if you want to
+use the LLM functionality. Make sure to go to http://127.0.0.1:8081 in your
+browser after running the script!
+
+```bash
+export HERACLES_OPENAI_API_KEY=<YOUR OPENAI API KEY>
+./chatdsg_viser_demo.sh # Then navigate to http://127.0.0.1:8081 in your browser
+```
+
+You may have to pan the camera around if the scene graph is rendered out of the
+initial camera frame.
+
+
+# Development Setup
+
+If you want functionality beyond these basic demos, you will want to install
+`heracles` locally. This requires 1) spinning up a database, and 2) installing
+`heracles.
 
 Pull the Neo4j database image:
 ```bash
@@ -35,15 +93,13 @@ for an example, as that successfully installs and runs the library for CI.
 If you don't have a scene graph to test with, you can use a large example scene
 graph [here](https://drive.google.com/file/d/1aktyS792PUrj2ACRu1DoxMGse55GWloB/view?usp=drive_link).
 This scene graph has 2D places and objects, and 3D places (that don't make much
-sense), but no regions or rooms.
-
-The script `examples/dsg_test.py` is a small example of loading an existing
-scene graph file into a graph database and running some simple queries. You can
-run it in interaction mode (`ipython3 -i heracles/examples/dsg_test.py`) and
-then try executing some of the other example queries below.
+sense), but no regions or rooms. This repo contains an example scene graph as well.
 
 ## Demo - Loading a Scene Graph
-We provide a demo program that loads a scene graph from file, encodes it into a database, and performs a simple query for validation. The demo program is implemented with interactive mode in mind to make it easy to try other kinds of queries.
+We provide a demo program that loads a scene graph from file, encodes it into a
+database, and performs a simple query for validation. The demo program is
+implemented with interactive mode in mind to make it easy to try other kinds of
+queries.
 
 ### Environment Variables
 We expect certain environment variables to be set, listed below. `HERACLES\_NEO4J\_URI` can alternatively be provided via the command line. An example of `HERACLES\_NEO4J\_URI` is `neo4j://127.0.0.1:7688`.

chatdsg_ros_demo.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+set -e
+
+docker compose -f docker/docker-compose.yaml up -d neo4j hydra_visualization
+docker compose -f docker/docker-compose.yaml run --rm chatdsg # Blocks until terminal exits
+docker compose -f docker/docker-compose.yaml down hydra_visualization

chatdsg_viser_demo.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+set -e
+
+docker compose -f docker/docker-compose.yaml up -d neo4j viser_visualization
+docker compose -f docker/docker-compose.yaml run --rm chatdsg
+docker compose -f docker/docker-compose.yaml down viser_visualization

docker/chatdsg_demo.Dockerfile

@@ -0,0 +1,8 @@
+FROM python:3.12-slim
+
+RUN apt-get update
+RUN apt-get install -y git
+COPY heracles/examples/scene_graphs/example_dsg.json example_dsg.json
+RUN git clone https://github.com/GoldenZephyr/heracles_agents.git
+RUN pip install ./heracles_agents[all]
+WORKDIR /heracles_agents/examples/chatdsg

docker/docker-compose.yaml

@@ -0,0 +1,87 @@
+---
+services:
+
+  # Graph database storing scene graph
+  neo4j:
+    image: neo4j:5.25.1
+    environment:
+      NEO4J_server_bolt_listen__address: :7683
+      NEO4J_server_http_advertised__address: :7469
+      NEO4J_server_http_listen__address: :7469
+      NEO4J_AUTH: neo4j/neo4j_pw
+    network_mode: host
+    healthcheck:
+      test: [CMD-SHELL, wget --no-verbose --tries=1 --spider http://localhost:7469 || exit 1]
+      interval: 1s
+      timeout: 10s
+      retries: 10
+      start_period: 2s
+
+  # Standalone python interpreter example for running queries against database
+  cli:
+    build:
+      context: ..
+      dockerfile: docker/minimal_db_cli.Dockerfile
+    depends_on:
+      neo4j:
+        condition: service_healthy
+    environment:
+      HERACLES_NEO4J_USERNAME: neo4j
+      HERACLES_NEO4J_PASSWORD: neo4j_pw
+      HERACLES_NEO4J_URI: neo4j://127.0.0.1:7683
+    network_mode: host
+    stdin_open: true
+    tty: true
+    entrypoint: [ipython, -i, /heracles/examples/load_scene_graph.py, --, --scene_graph, /heracles/examples/scene_graphs/example_dsg.json]
+
+  # Interactive LLM-based agent interaction with database
+  chatdsg:
+    build:
+      context: ..
+      dockerfile: docker/chatdsg_demo.Dockerfile
+    depends_on:
+      neo4j:
+        condition: service_healthy
+    environment:
+      HERACLES_NEO4J_USERNAME: neo4j
+      HERACLES_NEO4J_PASSWORD: neo4j_pw
+      HERACLES_NEO4J_URI: neo4j://127.0.0.1:7683
+      HERACLES_OPENAI_API_KEY: $HERACLES_OPENAI_API_KEY
+      HERACLES_AGENTS_PATH: /heracles_agents
+      ADT4_HERACLES_IP: 127.0.0.1
+      ADT4_HERACLES_PORT: 7683
+    network_mode: host
+    stdin_open: true
+    tty: true
+    entrypoint: [python3, chatdsg.py, --scene-graph, /example_dsg.json]
+
+  # Visualize scene graph in RVIZ using Hydra ROS visualizer
+  hydra_visualization:
+    build:
+      dockerfile: hydra.Dockerfile
+    depends_on:
+      neo4j:
+        condition: service_healthy
+    environment:
+      HERACLES_NEO4J_USERNAME: neo4j
+      HERACLES_NEO4J_PASSWORD: neo4j_pw
+      HERACLES_IP: 127.0.0.1
+      HERACLES_PORT: 7683
+      HERACLES_VENV: ''
+      DISPLAY: $DISPLAY
+      QT_X11_NO_MITSHM: 1
+    network_mode: host
+    volumes: [/tmp/.X11-unix:/tmp/.X11-unix:rw]
+    entrypoint: [/bin/bash, /run_hydra_visualization.sh]
+
+  # Visualize scene graph in Viser using spark_dsg Viser visualization
+  viser_visualization:
+    build:
+      context: ..
+      dockerfile: docker/spark_dsg_viser.Dockerfile
+    environment:
+      HERACLES_NEO4J_USERNAME: neo4j
+      HERACLES_NEO4J_PASSWORD: neo4j_pw
+      HERACLES_NEO4J_URI: neo4j://127.0.0.1:7683
+    network_mode: host
+    entrypoint: [python, /heracles/src/heracles/heracles_viser_publisher.py, --ip, 127.0.0.1, --port, '8081']

docker/hydra.Dockerfile

@@ -0,0 +1,26 @@
+FROM ros:jazzy-ros-core
+
+RUN apt-get update
+RUN apt-get install -y git ros-dev-tools python3.12-venv python3-pip libgoogle-glog-dev
+RUN rosdep init && rosdep update
+
+RUN mkdir -p /hydra_ws/src
+WORKDIR /hydra_ws
+RUN echo "build: {cmake-args: [-DCMAKE_BUILD_TYPE=Release]}" > colcon_defaults.yaml
+
+WORKDIR /hydra_ws/src
+RUN git clone https://github.com/MIT-SPARK/Hydra-ROS.git hydra_ros
+RUN git clone https://github.com/GoldenZephyr/heracles_agents.git
+RUN git clone https://github.com/GoldenZephyr/heracles.git
+
+RUN python3 -m venv /venv --system-site-packages
+RUN git clone https://github.com/MIT-SPARK/config_utilities.git
+RUN git clone --branch ros2 https://github.com/MIT-SPARK/Kimera-PGMO.git
+RUN git clone https://github.com/MIT-SPARK/Spark-DSG.git
+RUN git clone https://github.com/MIT-SPARK/ianvs.git
+
+RUN rosdep install --from-paths . --ignore-src -r -y
+WORKDIR /hydra_ws
+RUN . /opt/ros/jazzy/setup.sh && colcon build --packages-up-to hydra_visualizer && colcon build --packages-up-to heracles heracles_ros heracles_agents
+RUN . /venv/bin/activate && pip install ./src/heracles_agents[all]
+COPY run_hydra_visualization.sh /run_hydra_visualization.sh

docker/minimal_db_cli.Dockerfile

@@ -0,0 +1,5 @@
+FROM python:3.12-slim
+
+RUN apt update
+COPY heracles heracles
+RUN pip install ./heracles ipython

docker/run_hydra_visualization.sh

@@ -0,0 +1,5 @@
+#!/bin/sh
+. /opt/ros/jazzy/setup.sh
+. /hydra_ws/install/setup.sh
+. /venv/bin/activate
+ros2 launch heracles_ros demo.launch.yaml launch_heracles_publisher:=true launch_hydra_visualizer:=true launch_rviz:=true

docker/spark_dsg_viser.Dockerfile

@@ -0,0 +1,7 @@
+FROM python:3.12-slim
+
+RUN apt update
+RUN apt install -y git libzmq3-dev nlohmann-json3-dev build-essential
+COPY heracles heracles
+RUN git clone -b feature/viser_aaron --single-branch https://github.com/MIT-SPARK/Spark-DSG.git spark_dsg
+RUN pip install ./heracles ipython ./spark_dsg

heracles_demo.sh

@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+set -e
+docker compose -f docker/docker-compose.yaml up -d neo4j
+docker compose -f docker/docker-compose.yaml run --rm cli