Lens distortion not fully corrected in aligned colored point cloud

[Navifra-Brad]

Describe the bug
When generating a colored point cloud using the Femto Bolt by aligning RGB and Depth frames, the resulting point cloud appears to be based on distorted images. Although the distortion is subtle, it becomes more noticeable as objects get closer to the camera. It seems that the internal processing does not fully account for lens undistortion.

Reproduction Steps

1. Initialize the pipeline using the Femto Bolt SDK.
2. Enable Align Mode to align RGB and Depth frames.
3. Generate and visualize a colored point cloud.
4. Observe the edges of objects, especially when they are positioned close to the lens.

Expected behavior
The colored point cloud should be generated using undistorted frames (or internal calibration parameters should be applied) so that straight edges in the real world appear straight in the point cloud, regardless of the distance from the camera.

Log Output

• original image

• undistorted image

System Environment:

• OS: Windows 11, Ubuntu 22.04 both of them.
• Architecture: x86_64, armv7 both of them.
• Visual Studio/GCC：Version VS2022 / GCC 13.3

Tool/SDK Version:

• SDK: Version: pyorbbecsdk2==2.0.18
• OrbbecViewer：: V2.4.11-main

Device Information:

• Model: Femtobolt.
• Firmware Version: 1.0.9

Additional context
I have tested this by manually applying cv::remap for undistortion using OpenCV, and the results appeared much more accurate.
Does the SDK internally apply undistortion when aligning RGB and Depth for point cloud generation?
If so, why does the distortion become more prominent as objects move closer to the camera?
I have attached images comparing the distorted (original output) and undistorted (after manual OpenCV correction) results for your reference.

[blanklavender]

Hi, I am not sure if this will help but I have been working with the femto bolt camera too and I am facing a similar issue. I looked at the OrbbecSDKv2 repo and observed that the alignfilter function automatically grabs the parameters and applies them from the camera itself. I looked at the SDK code here and it seems like the distortion is false by default. You can check here.

[Navifra-Brad]

After @blanklavender comment, I took a much closer look at the source code and realized that the align function actually has nothing to do with undistorting the color image.I found that addTargetDistortion is false by default. As shown in the code below, when this is the case, the SDK simply initializes all distortion coefficients to 0:

        if(!addTargetDistortion_) {
            OBCameraDistortion alignDistortion = { 0 };
            auto               distortion      = fromProfile->getDistortion();
            auto               model           = distortion.model;
            alignDistortion.model              = model;
            alignProfile->bindDistortion(alignDistortion);
        }
        else {
            alignProfile->bindDistortion(toProfile->getDistortion());
        }

What this essentially does is map the depth data to an ideal pinhole camera model (creating a perfectly flat depth image). However, the raw color image still retains its actual lens distortion, which naturally causes stretching and misalignment as you move from the center toward the edges of the frame. Because of this mismatch, to make them perfectly align, we have to manually undistort the color image ourselves using OpenCV functions like cv2.initUndistortRectifyMap and cv2.remap, applying the color camera's intrinsic (K) and distortion (D) coefficients. Only after doing this manual step do they truly match up.If any Orbbec SDK maintainers are reading this: wouldn't it be much more convenient for developers if the SDK provided a built-in, intuitive way to handle this color undistortion process?

[wolfd]

I was also surprised by this, but ended up dropping AlignFilter altogether in favor of doing the undistortion and reprojection outside of the SDK. It definitely confused me the first time, since the cameras are so close to linear looking that I assumed that I was getting undistorted results.

Even if you turn on the distortion correction in the AlignFilter, what it does is it redistorts the depth data into the distortion of the color camera, which is neat but a little dancing around the problem.

The main reason I was using the D2C was to get the "hardware" support, but I've since realized that "hardware" is actually just SIMD operations on the host machine if you're on x86_64. I had assumed that it was done in camera.