HST COMBO

Introduction

This repository is built for the social navigation project at fluent robotics lab at the University of Michigan Robot Institute.

In this repository, we have three packages, including hst_infer, skeleton_extractor(Multi-human Skeleton RGBD) and skeleton_interfaces. NOTE, their own github repo are updated to the latest, so please use the packages here in this repo.

• skeleton_extractor: Branch ros2_humble is developed in Ubuntu22(focal) ROS2 humble to track human and skeleton by YOLOv8 and kalman filters. The skeleton_extractor generates 3D human skeleton data from RGB-D camera and publish skeleton data in a node through message/interface skeleton_interfaces.
• skeleton_interfaces:
  • MultiHumanSkeleton
    • std_msgs/Header header: common ROS message header
    • HumanSkeleton[] multi_human_skeleton: a list of HumanSkeleton
  • HumanSkeleton
    • uint32 human_id: human id generated by YOLO
    • geometry_msgs/Point32 human_center: human geometrical center
    • geometry_msgs/Point32[] keypoint_data: human skeleton data
    • bool[] keypoint_mask: True -> valid keypoint, False -> invalid/meaningless keypoint
• hst_infer: The hst_infer is built to load skeleton data in a deque then the deque will generate a time-series horizon, including historical data and future predition(np.nan). The Human Scene Transformer loads the data array from deque and make human trajectory prediction in a horizon.
  • horizon total length: 19 step
  • history length: 6 step
  • presence: 1 step
  • futrue: 12 step
  • timestep: 1/3 second per step

Get Started

Dependency Requirement

• Ubuntu22.04, docker is highly recommend. Select the repository branch according to your requirement.
• Python3.10, poetry is highly recommend as the python virtual environment management tool. The poetry will be installed automatically in the dockerfile. Anaconda3 is not recommended because there might be some conflict between ROS packages and Anaconda3 env.
• ROS2 Humble
• Python dependent packages are shown in pyproject.toml in hst_node and Multi-human_Skeleton_RGBD. You can use poetry to install them.
  • poetry shell to start poetry integrated shell, then poetry install to install dependecies and virtual environment automatically.

Run

If you are using FLUENT-LAB-11, you do not have to configure the environment and can just go to the Run step.

• Get into the virtual environment (Only if working with Stretch):

cd /path/to/ros-devcontainer
./enter-container.sh

• Git clone or move this repo to your ROS2 workspace like */*_ws/HST_combo and get into your ROS2 workspace cd /path/to/workspace.

• Configuration

• Compile and build(Optional):

colcon build --pakcge-select hst_infer skeleton_extractor skeleton_interface

• Launch ros nodes:

ros2 launch hst_infer socialnav_core.launch

The node will be waiting for the ROS bag topics or real-time ROS topics.

• Launch rviz:

ros2 launch hst_infer rviz.launch

• If you encounter a tensorflow error "Input dims must be <= 5 and >=1" when running hst_infer:

export TF_DISABLE_MKL=1
export TF_ENABLE_ONEDNN_OPTS=0