Error with CNN Launch file. [segmapper-2] process has died

[jeffkimkorea-git]

@RozDavid Thanks alot for your work again. I tried to learn both kitti and cnn launch files and Kitti one did great job on my environment too. On the other hand, cnn based version stops here. Do you have any idea with it? Thanks a lot.

root@jeff-laptop:/# roslaunch segmapper cnn_loam_segmap.launch
... logging to /root/.ros/log/7d25cbd2-2d7b-11ec-a336-e86a64425b4b/roslaunch-jeff-laptop-23047.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://jeff-laptop:34109/

SUMMARY

PARAMETERS

• /SegMapper/Benchmarker/enable_live_output: False
• /SegMapper/Benchmarker/results_directory: /tmp/segmap_bench...
• /SegMapper/Benchmarker/save_statistics_only: False
• /SegMapper/LaserSlamWorker/assembled_cloud_sub_topic: /velodyne_cloud_s...
• /SegMapper/LaserSlamWorker/create_filtered_map: True
• /SegMapper/LaserSlamWorker/distant_map_pub_topic: distant_map
• /SegMapper/LaserSlamWorker/full_map_pub_topic: full_map
• /SegMapper/LaserSlamWorker/local_map_pub_topic: local_map
• /SegMapper/LaserSlamWorker/map_publication_rate_hz: 3
• /SegMapper/LaserSlamWorker/odom_frame: world
• /SegMapper/LaserSlamWorker/odometry_trajectory_pub_topic: odometry_trajectory
• /SegMapper/LaserSlamWorker/path_to_export_trajectory: /tmp/loam-segmap/...
• /SegMapper/LaserSlamWorker/publish_distant_map: False
• /SegMapper/LaserSlamWorker/publish_full_map: False
• /SegMapper/LaserSlamWorker/publish_local_map: True
• /SegMapper/LaserSlamWorker/sensor_frame: aft_mapped
• /SegMapper/LaserSlamWorker/trajectory_pub_topic: trajectory
• /SegMapper/LaserSlamWorker/use_odometry_information: True
• /SegMapper/OnlineEstimator/LaserTrack/add_m_estimator_on_icp: False
• /SegMapper/OnlineEstimator/LaserTrack/add_m_estimator_on_odom: False
• /SegMapper/OnlineEstimator/LaserTrack/force_priors: False
• /SegMapper/OnlineEstimator/LaserTrack/icp_noise_model: [0.5, 0.05, 0.05,...
• /SegMapper/OnlineEstimator/LaserTrack/nscan_in_sub_map: 5
• /SegMapper/OnlineEstimator/LaserTrack/nscan_to_match: 3
• /SegMapper/OnlineEstimator/LaserTrack/odometry_noise_model: [0.5, 0.5, 0.5, 0...
• /SegMapper/OnlineEstimator/LaserTrack/save_icp_results: False
• /SegMapper/OnlineEstimator/LaserTrack/use_icp_factors: False
• /SegMapper/OnlineEstimator/LaserTrack/use_odom_factors: True
• /SegMapper/OnlineEstimator/add_m_estimator_on_loop_closures: True
• /SegMapper/OnlineEstimator/do_icp_step_on_loop_closures: False
• /SegMapper/OnlineEstimator/loop_closure_noise_model: [0.0005, 0.0005, ...
• /SegMapper/OnlineEstimator/loop_closures_sub_maps_radius: 3
• /SegMapper/SegMatchWorker/SegMatch/Classifier/apply_hard_threshold_on_feature_distance: False
• /SegMapper/SegMatchWorker/SegMatch/Classifier/do_not_use_cars: True
• /SegMapper/SegMatchWorker/SegMatch/Classifier/enable_two_stage_retrieval: False
• /SegMapper/SegMatchWorker/SegMatch/Classifier/feature_distance_threshold: 100
• /SegMapper/SegMatchWorker/SegMatch/Classifier/knn_feature_dim: 64
• /SegMapper/SegMatchWorker/SegMatch/Classifier/max_distance_for_accepting_matches: 50.0
• /SegMapper/SegMatchWorker/SegMatch/Classifier/max_eigen_features_values: [2493.5, 186681.0...
• /SegMapper/SegMatchWorker/SegMatch/Classifier/n_nearest_neighbours: 120
• /SegMapper/SegMatchWorker/SegMatch/Classifier/normalize_eigen_for_hard_threshold: False
• /SegMapper/SegMatchWorker/SegMatch/Classifier/normalize_eigen_for_knn: False
• /SegMapper/SegMatchWorker/SegMatch/Classifier/threshold_to_accept_match: 0.3
• /SegMapper/SegMatchWorker/SegMatch/Descriptors/cnn_model_path: /root/.segmap/tra...
• /SegMapper/SegMatchWorker/SegMatch/Descriptors/descriptor_types: ['CNN', 'Eigenval...
• /SegMapper/SegMatchWorker/SegMatch/Descriptors/semantics_nn_path: /root/.segmap/tra...
• /SegMapper/SegMatchWorker/SegMatch/GeometricConsistency/max_consistency_distance_for_caching: 10.0
• /SegMapper/SegMatchWorker/SegMatch/GeometricConsistency/min_cluster_size: 4
• /SegMapper/SegMatchWorker/SegMatch/GeometricConsistency/recognizer_type: Incremental
• /SegMapper/SegMatchWorker/SegMatch/GeometricConsistency/resolution: 0.4
• /SegMapper/SegMatchWorker/SegMatch/LocalMap/max_vertical_distance_m: 999.0
• /SegMapper/SegMatchWorker/SegMatch/LocalMap/min_points_per_voxel: 1
• /SegMapper/SegMatchWorker/SegMatch/LocalMap/min_vertical_distance_m: -999.0
• /SegMapper/SegMatchWorker/SegMatch/LocalMap/neighbors_provider_type: KdTree
• /SegMapper/SegMatchWorker/SegMatch/LocalMap/radius_m: 50
• /SegMapper/SegMatchWorker/SegMatch/LocalMap/voxel_size_m: 0.1
• /SegMapper/SegMatchWorker/SegMatch/Segmenters/max_cluster_size: 15000
• /SegMapper/SegMatchWorker/SegMatch/Segmenters/min_cluster_size: 100
• /SegMapper/SegMatchWorker/SegMatch/Segmenters/radius_for_growing: 0.2
• /SegMapper/SegMatchWorker/SegMatch/Segmenters/segmenter_type: IncrementalEuclid...
• /SegMapper/SegMatchWorker/SegMatch/boundary_radius_m: 999.0
• /SegMapper/SegMatchWorker/SegMatch/centroid_distance_threshold_m: 2
• /SegMapper/SegMatchWorker/SegMatch/check_pose_lies_below_segments: False
• /SegMapper/SegMatchWorker/SegMatch/filter_boundary_segments: True
• /SegMapper/SegMatchWorker/SegMatch/filter_duplicate_segments: True
• /SegMapper/SegMatchWorker/SegMatch/min_time_between_segment_for_matches_s: 0
• /SegMapper/SegMatchWorker/SegMatch/segmentation_height_above_m: 999.0
• /SegMapper/SegMatchWorker/SegMatch/segmentation_height_below_m: 999.0
• /SegMapper/SegMatchWorker/SegMatch/segmentation_radius_m: 50.0
• /SegMapper/SegMatchWorker/align_target_map_on_first_loop_closure: False
• /SegMapper/SegMatchWorker/close_loops: False
• /SegMapper/SegMatchWorker/distance_between_segmentations_m: 0.0
• /SegMapper/SegMatchWorker/distance_to_lower_target_cloud_for_viz_m: 25.0
• /SegMapper/SegMatchWorker/export_segments_and_matches: True
• /SegMapper/SegMatchWorker/line_scale_loop_closures: 5.0
• /SegMapper/SegMatchWorker/line_scale_matches: 0.4
• /SegMapper/SegMatchWorker/localize: True
• /SegMapper/SegMatchWorker/min_time_between_matches_s: 1
• /SegMapper/SegMatchWorker/publish_predicted_segment_matches: False
• /SegMapper/SegMatchWorker/ransac_max_iterations: 1000
• /SegMapper/SegMatchWorker/ransac_outlier_trashold: 0.25
• /SegMapper/SegMatchWorker/ratio_of_points_to_keep_when_publishing: 1.0
• /SegMapper/SegMatchWorker/save_map_full_path: /tmp/loam-segmap/...
• /SegMapper/SegMatchWorker/target_cloud_filename: /root/.segmap/kit...
• /SegMapper/clear_local_map_after_loop_closure: False
• /SegMapper/number_of_robots: 1
• /SegMapper/publish_world_to_odom: True
• /SegMapper/robot_prefix: na
• /SegMapper/tf_publication_rate_hz: 10
• /SegMapper/world_frame: map
• /laserMapping/scanPeriod: 0.1
• /laserOdometry/scanPeriod: 0.1
• /multiScanRegistration/lidar: HDL-64E
• /multiScanRegistration/scanPeriod: 0.1
• /rosdistro: kinetic
• /rosversion: 1.12.17
• /segmapper/icp_configuration_file: /root/Catkin/LOL/...
• /segmapper/icp_input_filters_file: /root/Catkin/LOL/...
• /use_sim_time: True

NODES
/
laserMapping (loam_velodyne/laserMapping)
laserOdometry (loam_velodyne/laserOdometry)
multiScanRegistration (loam_velodyne/multiScanRegistration)
player (rosbag/play)
segmapper (segmapper/segmapper_node)
static_broadcaster (tf2_ros/static_transform_publisher)
transformMaintenance (loam_velodyne/transformMaintenance)

ROS_MASTER_URI=http://localhost:11311

process[player-1]: started with pid [23064]
[ INFO] [1634292411.984395989]: Opening /root/.segmap/kitti/2011_09_30_drive_18.bag
Waiting 0.2 seconds after advertising topics... done.

Hit space to toggle paused, or 's' to step.
process[segmapper-2]: started with pid [23065]
process[static_broadcaster-3]: started with pid [23088]
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1015 10:06:53.645305 23065 segmatch.cpp:27] Do not forget to initialize SegMatch.
I1015 10:06:53.744660 23065 laser_track.cpp:16] Loading ICP configurations from: /root/Catkin/LOL/src/segmap/segmapper/launch/kitti/icp_dynamic_outdoor.yaml
I1015 10:06:53.746557 23065 laser_track.cpp:26] Loading ICP input filters from: /root/Catkin/LOL/src/segmap/segmapper/launch/kitti/input_filters_outdoor.yaml
I1015 10:06:53.747190 23065 incremental_estimator.cpp:31] Creating loop closure noise model with cauchy.
I1015 10:06:53.747244 23065 incremental_estimator.cpp:54] Loading ICP configurations from: /root/Catkin/LOL/src/segmap/segmapper/launch/kitti/icp_dynamic_outdoor.yaml
[ INFO] [1634292413.749479821]: Number of laser_slam workers: 1
I1015 10:06:53.749758 23065 segmapper.cpp:86] Robot 0 subscribes to /velodyne_cloud_segmatch world and aft_mapped
I1015 10:06:53.749797 23065 segmapper.cpp:89] Robot 0 publishes to trajectory and local_map
I1015 10:06:53.770653 23065 cnn.hpp:20] Loading CNN model in /root/.segmap/trained_models/segmap64/
2021-10-15 10:06:53.770740: I /root/Catkin/LOL/src/segmap/tf_graph_executor/src/tf_graph_executor.cpp:17] Entering TensorflowGraphExecutor with path /root/.segmap/trained_models/segmap64/model.ckpt.meta
2021-10-15 10:06:53.902945: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-10-15 10:06:53.903578: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties:
name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:01:00.0
totalMemory: 5.94GiB freeMemory: 5.02GiB
2021-10-15 10:06:53.903614: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
process[multiScanRegistration-4]: started with pid [23089]
[ INFO] [1634292414.325236902]: Set scanPeriod: 0.1
[ INFO] [1634292414.344890556]: Set HDL-64E scan mapper.
process[laserOdometry-5]: started with pid [23286]
[ INFO] [1634292415.039679347]: Set scanPeriod: 0.1
2021-10-15 10:06:55.565654: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-10-15 10:06:55.565695: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0
2021-10-15 10:06:55.565711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N
2021-10-15 10:06:55.565977: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4786 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
process[laserMapping-6]: started with pid [23333]
2021-10-15 10:06:55.653823: I /root/Catkin/LOL/src/segmap/tf_graph_executor/src/tf_graph_executor.cpp:26] calling custom function
2021-10-15 10:06:55.660305: I /root/Catkin/LOL/src/segmap/tf_graph_executor/src/tf_graph_executor.cpp:28] Custom function called
2021-10-15 10:06:55.696094: I /root/Catkin/LOL/src/segmap/tf_graph_executor/src/tf_graph_executor.cpp:41] Loading checkpoint /root/.segmap/trained_models/segmap64/model.ckpt
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /root/Catkin/LOL/devel/lib/loam_velodyne/laserMapping...(no debugging symbols found)...done.
Starting program: /root/Catkin/LOL/devel/lib/loam_velodyne/laserMapping __name:=laserMapping __log:=/root/.ros/log/7d25cbd2-2d7b-11ec-a336-e86a64425b4b/laserMapping-6.log
warning: Error disabling address space randomization: Operation not permitted
[tcsetpgrp failed in terminal_inferior: Inappropriate ioctl for device]
[tcsetpgrp failed in terminal_inferior: Invalid argument]
I1015 10:06:55.903869 23065 cnn.hpp:38] Loading semantics model in /root/.segmap/trained_models/segmap64_semantics/
2021-10-15 10:06:55.903902: I /root/Catkin/LOL/src/segmap/tf_graph_executor/src/tf_graph_executor.cpp:17] Entering TensorflowGraphExecutor with path /root/.segmap/trained_models/segmap64_semantics/model.ckpt.meta
2021-10-15 10:06:55.903980: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2021-10-15 10:06:55.904023: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-10-15 10:06:55.904049: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0
2021-10-15 10:06:55.904070: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N
2021-10-15 10:06:55.904222: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4786 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
2021-10-15 10:06:55.904489: I /root/Catkin/LOL/src/segmap/tf_graph_executor/src/tf_graph_executor.cpp:26] calling custom function
2021-10-15 10:06:55.905730: I /root/Catkin/LOL/src/segmap/tf_graph_executor/src/tf_graph_executor.cpp:28] Custom function called
2021-10-15 10:06:55.910175: I /root/Catkin/LOL/src/segmap/tf_graph_executor/src/tf_graph_executor.cpp:41] Loading checkpoint /root/.segmap/trained_models/segmap64_semantics/model.ckpt
I1015 10:06:55.919739 23065 cnn.hpp:43] Loaded all TensorFlow models.
[ INFO] [1634292415.939343211, 1317376479.232045294]: Loading target cloud.
I1015 10:06:55.939399 23065 utilities.hpp:24] Loading cloud: /root/.segmap/kitti/drive18_target_map.pcd.
Failed to find match for field 'ed_cluster_id'.
Failed to find match for field 'sc_cluster_id'.
process[transformMaintenance-7]: started with pid [23399]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[tcsetpgrp failed in terminal_inferior: Invalid argument]
[tcsetpgrp failed in terminal_inferior: Invalid argument]
[New Thread 0x7f1b4dc5a700 (LWP 23490)]
[New Thread 0x7f1b4d459700 (LWP 23491)]
[New Thread 0x7f1b4cc58700 (LWP 23492)]
[New Thread 0x7f1b47fff700 (LWP 23504)]
[ INFO] [1634292419.279634124]: Set scanPeriod: 0.1
I1015 10:07:24.746826 23065 segmatch.cpp:202] Removing too near segments from source map.
[FATAL] [1634292448.374863591, 1317376479.232045294]: Target Cloud processed
[FATAL] [1634292448.374909806, 1317376479.232045294]: Target Set
[segmapper-2] process has died [pid 23065, exit code -11, cmd /root/Catkin/LOL/devel/lib/segmapper/segmapper_node __name:=segmapper __log:=/root/.ros/log/7d25cbd2-2d7b-11ec-a336-e86a64425b4b/segmapper-2.log].
log file: /root/.ros/log/7d25cbd2-2d7b-11ec-a336-e86a64425b4b/segmapper-2*.log

[RozDavid]

Hey @jeffkimkorea-git,

Could you check your GDB stacktrace? That migh help where the segmapper node failed. With this error only it is a bit hard to tell.

This page might help you how to do it.

[jeffkimkorea-git]

Thanks @RozDavid for the detailed reply. But I am not familiar to the GDB and I could not use it. When launching the launch file, I could find gdb is running and I entered "gdb bt" to back trace, I could got no information. I need to know how to use gdb I think.

Furthermore, I am trying to use eigen based localization using 'LOL' and I changed the topic name of velodyne_points in loam_velodyne/launch/loam_velodyne.launch ,
loam_velodyne/launch/hector_loam_velodyne.launch , laser_slam/sensor_drivers/velodyne_assembler/launch/velodyne_assembler.launch , segmap/segmapper/launch/kitti/kitti_loam_segmap.launch , laser_slam/sensor_drivers/velodyne_assembler/src/velodyne_assembler_ros.cpp

and I changed the pcd file instead of drive18_target_map.pcd in kitti_loam_segmap.launch

but nothing comes out on rviz that I am stucked here.

What I have changed in map file is getting rid of ground data on pcd map and arranged the map to start point of bag file.
Is there something else that I have to be done? Thanks for your kindly help.

[RozDavid]

Generally when you don't know if the communication between specific nodes is up and running it is a good practice to check if anything is published by running the command rostopic echo <TOPIC_NAME>.

Additionally, you might want to open up the rqt consoles and check the Node Graph there to see if your pcd files and streaming cloud topics are connected to the right nodes.

[jeffkimkorea-git]

Thanks @RozDavid for the detailed reply. But I am not familiar to the GDB and I could not use it. When launching the launch file, I could find gdb is running and I entered "gdb bt" to back trace, I could got no information. I need to know how to use gdb I think.

Furthermore, I am trying to use eigen based localization using 'LOL' and I changed the topic name of velodyne_points in loam_velodyne/launch/loam_velodyne.launch , loam_velodyne/launch/hector_loam_velodyne.launch , laser_slam/sensor_drivers/velodyne_assembler/launch/velodyne_assembler.launch , segmap/segmapper/launch/kitti/kitti_loam_segmap.launch , laser_slam/sensor_drivers/velodyne_assembler/src/velodyne_assembler_ros.cpp

and I changed the pcd file instead of drive18_target_map.pcd in kitti_loam_segmap.launch

but nothing comes out on rviz that I am stucked here.

What I have changed in map file is getting rid of ground data on pcd map and arranged the map to start point of bag file. Is there something else that I have to be done? Thanks for your kindly help.

Thanks @RozDavid for the detailed instruction. The problem of the previous case was because of the pcd map data. And still I could not get the reason why pcd data of previous one do not work. For now I could make a map with ndt_mapping, and exactly right after making map with ground removal data of point cloud in real time, it could make a map quite nice.

And another problem has occured here, I have seen most of issues here and I believe the problem here is just same as #12 I think.

Loam do not work as well. I am not sure what the problem will be. I have checked the raw data of it with rviz, any kind of difference I couldn't find with it. Do you have any idea with it?

Thanks a lot for the work again.

All the best,
Jeff Kim

[RozDavid]

It's quite hard to say from only an image. Could you provide a rqt generated node graph? Maybe you should try to debug the components independently. Try the streaming pointcloud with the basic loam compponents without any kind of relocalization. Or if you have any ground truth with your data (maybe from gps or other odometry source) try feeding those poses instead of the loam component for checking if matches can be found.

I am afrad with such a sparse and incomplete map that you have in this example it will be hard to use the algorithm. Are there any usable objects/structures in the scene, or haw many rays does the lidar have?

[jeffkimkorea-git]

Thanks @RozDavid for the quick reply. Here is the graph first.

Let me first try to learn LOAM independently. And for now I could make the poses with 'ndt matching' that I would try to feed the pose into it too.

The objects in the pictures are poles from traffic lights and traffic signes and so on. The rosbag data here has 64 layers from ouster and 2 32 channels velodyne on the side of the vehicle. The raw data for the scene is down here. I am sure it was able to build a map with these sensors with ndt_mapping and also available with the matching. The thing is, when I tried it with the kitti-data set, one you provided, works nice.

As I have ground truth map data, I wanted to remove the ground and use it as the map for the algorithm, but it doesn't work for me. It says,

Hope all the best for you.
Jeff Kim.

I1021 20:29:03.579560 2955 segmatch.cpp:202] Removing too near segments from source map.
F1021 20:29:03.580590 2955 eigenvalue_based.cpp:86] Check failed: e1 != 0.0 (0 vs. 0)
[segmapper-3] process has died [pid 2955, exit code -6, cmd /root/Catkin/LOL/devel/lib/segmapper/segmapper_node __name:=segmapper __log:=/root/.ros/log/ec5ef948-32a2-11ec-9dd8-e86a64425b4b/segmapper-3.log].
log file: /root/.ros/log/ec5ef948-32a2-11ec-9dd8-e86a64425b4b/segmapper-3*.log

[jeffkimkorea-git]

This is the starting view of mapdata modified from high-definition map.