Add an ignore_mask argument to ignore certain source pixels

[arch1t3cht]

Explanation

Descaling models resampling as a linear transformation, but in practice the resampling process is not always entirely linear: After rescaling, pixel values might be clamped to their value range. For example, when a white image is resampled with zero-padding and the kernel has negative lobes, the second and third pixel lines from the edge will be clamped.

When descaling, these clamped pixels give wrong information, so they will negatively affect the descale result. In the above example, they will cause dirty borders in the descaled image.

One approach to fixing this problem is to simply discard any pixels which are suspected to be clipped from the linear system of equations. Since the system of equations is overdetermined, this can still give an accurate descale result, as long as no longer contiguous stretches of pixels are discarded.

This commit adds an ignore_mask argument to the descale functions. Pixels selected by this mask will have their corresponding equations dropped from the system of equations. This requires partially recomputing the matrix and its LDLT decomposition whenever this mask changes, but this is still reasonably efficient since the matrix is banded.

The ignore_mask is only supported when only descaling along one axis: When descaling along both axes, a second mask at an intermediate resolution would be required for the second descale, so this is better off being left to the user

Other details

• Value clipping was the main motivation for this, but it might also be useful in some other cases like when there are native-res elements in the frame that affect the descale result.
• I only implemented a C function for the actual descale process, so descaling with an ignore_mask forces DESCALE_OPT_NONE. Getting this to work with AVX2 is likely somewhere between very hard and impossible.
• Right now the ignore_mask is required to use 8-bit integer samples, this could be improved in the future.
• I only actually exposed this option for the VapourSynth plugin since I don't really have any experience with Avisynth.
• In some simple tests I ran, using an ignore_mask to handle clipping around borders was around four times slower than a normal descale using AVX2 and around two times slower than a normal descale using the C function.

Example usage

This code masks clipped values around borders to prevent dirty edges when descaling. Other masks can be used to also handle clipping around high-contrast edges in the middle of the frame, as long as care is taken to not have too many contiguous ignored pixels.

def clippedmask(clip, vertical=False):
    return clip.akarin.Expr(f"x 234 >= x 1 <= or {'height' if vertical else 'width'} {'Y' if vertical else 'X'} - 5 < {'Y' if vertical else 'X'} 4 < or and 255 0 ?")


def clipmasked_descale(clip, width, height):
    if width != clip.width and height != clip.height:
        clip = clipmasked_descale(clip, clip.width, height)
        clip = clipmasked_descale(clip, width, height)
        return clip

    return clip.descale.Debicubic(width, height, 0, 1, border_handling=1, ignore_mask=clippedmask(clip, clip.height != height))

Note also the order of the descales in the clipmasked_descale function - first descaling vertically, and then horizontally. The order is important here and needs to match the order used when originally resampling to properly handle values clipping in between the two steps. Getting the order wrong causes artifacts around corners.

[arch1t3cht]

By the way, I recently realized that this ignore_mask concept could be extended to giving pixels higher or lower weights in the linear system of equations. Instead of just dropping the pixel's corresponding equation, this would just multiply all its coefficients by that weight. Though I'm not really sure if there's any practical uses for this.

[arch1t3cht]

(This was merged in the JET fork, in case anyone wants to use it)