Gtk4 layer shell auto exclusive zone fails to take into account margin

[AKArien]

Wayfire version

0.10.1-unknown (Dec 31 2025) branch unknown wlroots-0.19.2

GPU / Driver

- glxinfo -B
name of display: :0
display: :0  screen: 0
direct rendering: Yes
Extended renderer info (GLX_MESA_query_renderer):
    Vendor: AMD (0x1002)
    Device: AMD Radeon 780M Graphics (radeonsi, phoenix, ACO, DRM 3.64, 7.0.11_1) (0x1900)
    Version: 26.0.6
    Accelerated: yes
    Video memory: 4096MB
    Unified memory: yes
    Preferred profile: core (0x1)
    Max core profile version: 4.6
    Max compat profile version: 4.6
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.2
Memory info (GL_ATI_meminfo):
    VBO free memory - total: 3344 MB, largest block: 3344 MB
    VBO free aux. memory - total: 13650 MB, largest block: 13650 MB
    Texture free memory - total: 3344 MB, largest block: 3344 MB
    Texture free aux. memory - total: 13650 MB, largest block: 13650 MB
    Renderbuffer free memory - total: 3344 MB, largest block: 3344 MB
    Renderbuffer free aux. memory - total: 13650 MB, largest block: 13650 MB
Memory info (GL_NVX_gpu_memory_info):
    Dedicated video memory: 4096 MB
    Total available memory: 17949 MB
    Currently available dedicated video memory: 3344 MB
OpenGL vendor string: AMD
OpenGL renderer string: AMD Radeon 780M Graphics (radeonsi, phoenix, ACO, DRM 3.64, 7.0.11_1)
OpenGL core profile version string: 4.6 (Core Profile) Mesa 26.0.6
OpenGL core profile shading language version string: 4.60
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile

OpenGL version string: 4.6 (Compatibility Profile) Mesa 26.0.6
OpenGL shading language version string: 4.60
OpenGL context flags: (none)
OpenGL profile mask: compatibility profile

OpenGL ES profile version string: OpenGL ES 3.2 Mesa 26.0.6
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20

Describe the bug

Gtk4 layer shell’s documentation for gtk_layer_auto_exclusive_zone_enable specifies that the exclusive zone is of the size of « the size of the window + relevant margin ». On wayfire specifically (also tested river and sway), the auto exclusive zone size will always be the size of the shell, without taking into account the margin.

To Reproduce

When running the gtk4-layer-demo example, attach the window to an edge and set a margin. Said margin will not be added to the exclusive zone, resulting in something of the sort :

Expected behavior

The auto exclusive zone behaves like on other compositors and adds the margin to the reserved space.

[wmww]

The .set_margin docs say "The exclusive zone includes the margin.". I don't think this is worded particularly clearly, but other compositors (sway, labwc and hyprland tested) interpret that to mean that an exclusive zone of 10 for a right-anchored surface with a margin should push other windows out of the way the distance of the margin and then 10 more pixels.

• X: maximized window
• L: layer surface window
• M: margin (layer surface decided)
• E: exclusive (layer surface decided)
• W: layer surface width (layer surface decided, but irrelevant)
• D: distance between screen edge and maximized window (calculated by compositor as M + E)

          __M__
     __W__
   ___E___
   _____D______
--+  +---+     |
  |  |   |     |
X |  | L |     |
  |  |   |     |
--+  +---+     |

It appears that wayfire incorrectly assume D = E.