uninstalling a DKMS package with an active OBSOLETE_BY messes the DKMS state up

[morgwai]

system info: dkms-3.3.0-1 (from Nvidia DC repos) on Debian-13

I maintain an unofficial Debian Trixie package, that backports ntsync kernel module to Trixie's 6.12 kernel: ntsync was merged to the mainline in 6.14 and enabled on Debian in 6.14.5. Therefore, I use OBSOLETE_BY="6.14.5" directive to avoid building for kernels that already have their own ntsync:

PACKAGE_NAME="ntsync"
PACKAGE_VERSION="1"
AUTOINSTALL=yes
OBSOLETE_BY="6.14.5"

BUILT_MODULE_NAME[0]="$PACKAGE_NAME"
DEST_MODULE_LOCATION[0]="/kernel/drivers/misc/"

If I install this package on a pristine Trixie system with only its base 6.12.x kernel, then everything works fine.

However, If I install it on a system that also has some kernel >=6.14.5 (for example some 6.17.x from trixie-backports), then the package correctly skips the build on that kernel, but if I uninstall it later, it messes my DKMS state and dkms status reports ntsync/1: broken and Manual intervention is required!

An example full terminal output:

install the package:

$ sudo apt-get install ntsync-kernel-dkms 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Suggested packages:
  winehq
The following NEW packages will be installed:
  ntsync-kernel-dkms
0 upgraded, 1 newly installed, 0 to remove and 1 not upgraded.
Need to get 9,276 B of archives.
After this operation, 45.1 kB of additional disk space will be used.
Get:1 http://download.opensuse.org/repositories/home:/morgwai:/ntsync/Debian_13  ntsync-kernel-dkms 6.14.0~repack-2 [9,276 B]
Fetched 9,276 B in 0s (99.1 kB/s)              
Selecting previously unselected package ntsync-kernel-dkms.
(Reading database ... 285837 files and directories currently installed.)
Preparing to unpack .../ntsync-kernel-dkms_6.14.0~repack-2_all.deb ...
Unpacking ntsync-kernel-dkms (6.14.0~repack-2) ...
Setting up ntsync-kernel-dkms (6.14.0~repack-2) ...
Loading new ntsync/1 DKMS files...
Building for 6.12.57+deb13-amd64 and 6.17.8+deb13-amd64

Building initial module ntsync/1 for 6.12.57+deb13-amd64
Sign command: /lib/modules/6.12.57+deb13-amd64/build/scripts/sign-file
Signing key: /var/lib/dkms/mok.key
Public certificate (MOK): /var/lib/dkms/mok.pub

Building module(s)... done.
Signing module /var/lib/dkms/ntsync/1/build/ntsync.ko
Installing /lib/modules/6.12.57+deb13-amd64/updates/dkms/ntsync.ko.xz
Running depmod... done.

Building initial module ntsync/1 for 6.17.8+deb13-amd64
Warning: The /var/lib/dkms/ntsync/1/6.17.8+deb13-amd64/x86_64/dkms.conf
for module ntsync/1 includes BUILD_EXCLUSIVE directives
which do not match this kernel/arch/config for any modules.
This indicates that it should not be built.

check the DKMS status:

$ sudo dkms status
ntsync/1, 6.12.57+deb13-amd64, x86_64: installed
nvidia/590.48.01, 6.12.57+deb13-amd64, x86_64: installed
nvidia/590.48.01, 6.17.8+deb13-amd64, x86_64: installed
tuxedo-drivers/4.18.1, 6.12.57+deb13-amd64, x86_64: installed
tuxedo-drivers/4.18.1, 6.17.8+deb13-amd64, x86_64: installed
tuxedo-yt6801/1.0.30tux5, 6.12.57+deb13-amd64, x86_64: installed
tuxedo-yt6801/1.0.30tux5, 6.17.8+deb13-amd64, x86_64: installed

remove the package:

$ sudo apt-get remove ntsync-kernel-dkms 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  ntsync-kernel-dkms
0 upgraded, 0 newly installed, 1 to remove and 1 not upgraded.
After this operation, 45.1 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 285845 files and directories currently installed.)
Removing ntsync-kernel-dkms (6.14.0~repack-2) ...
Module ntsync/1 for kernel 6.12.57+deb13-amd64 (x86_64):
Before uninstall, this module version was ACTIVE on this kernel.
Deleting /lib/modules/6.12.57+deb13-amd64/updates/dkms/ntsync.ko.xz
Running depmod.... done.
Module ntsync/1 is not built for kernel  (). Skipping...

(please notice the empty kernel version in the line above)

DKMS status will report the error:

$ sudo dkms status
ntsync/1: broken

Error! ntsync/1: Missing the module source directory or the symbolic link pointing to it.
Manual intervention is required!
nvidia/590.48.01, 6.12.57+deb13-amd64, x86_64: installed
nvidia/590.48.01, 6.17.8+deb13-amd64, x86_64: installed
tuxedo-drivers/4.18.1, 6.12.57+deb13-amd64, x86_64: installed
tuxedo-drivers/4.18.1, 6.17.8+deb13-amd64, x86_64: installed
tuxedo-yt6801/1.0.30tux5, 6.12.57+deb13-amd64, x86_64: installed
tuxedo-yt6801/1.0.30tux5, 6.17.8+deb13-amd64, x86_64: installed

The situation may be manually fixed with the following commands:

$ sudo mkdir /usr/src/ntsync-1
$ sudo dkms remove ntsync/1 --all

After that the DKMS status is clean:

$ sudo dkms status
nvidia/590.48.01, 6.12.57+deb13-amd64, x86_64: installed
nvidia/590.48.01, 6.17.8+deb13-amd64, x86_64: installed
tuxedo-drivers/4.18.1, 6.12.57+deb13-amd64, x86_64: installed
tuxedo-drivers/4.18.1, 6.17.8+deb13-amd64, x86_64: installed
tuxedo-yt6801/1.0.30tux5, 6.12.57+deb13-amd64, x86_64: installed
tuxedo-yt6801/1.0.30tux5, 6.17.8+deb13-amd64, x86_64: installed

[scaronni]

$ sudo apt-get install ntsync-kernel-dkms
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Suggested packages:
winehq
The following NEW packages will be installed:
ntsync-kernel-dkms
0 upgraded, 1 newly installed, 0 to remove and 1 not upgraded.
Need to get 9,276 B of archives.
After this operation, 45.1 kB of additional disk space will be used.
Get:1 http://download.opensuse.org/repositories/home:/morgwai:/ntsync/Debian_13 ntsync-kernel-dkms 6.14.0repack-2 [9,276 B]
Fetched 9,276 B in 0s (99.1 kB/s)
Selecting previously unselected package ntsync-kernel-dkms.
(Reading database ... 285837 files and directories currently installed.)
Preparing to unpack .../ntsync-kernel-dkms_6.14.0repack-2_all.deb ...
Unpacking ntsync-kernel-dkms (6.14.0repack-2) ...
Setting up ntsync-kernel-dkms (6.14.0repack-2) ...
Loading new ntsync/1 DKMS files...
Building for 6.12.57+deb13-amd64 and 6.17.8+deb13-amd64

Building initial module ntsync/1 for 6.12.57+deb13-amd64
Sign command: /lib/modules/6.12.57+deb13-amd64/build/scripts/sign-file
Signing key: /var/lib/dkms/mok.key
Public certificate (MOK): /var/lib/dkms/mok.pub

Building module(s)... done.
Signing module /var/lib/dkms/ntsync/1/build/ntsync.ko
Installing /lib/modules/6.12.57+deb13-amd64/updates/dkms/ntsync.ko.xz
Running depmod... done.

Building initial module ntsync/1 for 6.17.8+deb13-amd64
Warning: The /var/lib/dkms/ntsync/1/6.17.8+deb13-amd64/x86_64/dkms.conf
for module ntsync/1 includes BUILD_EXCLUSIVE directives
which do not match this kernel/arch/config for any modules.
This indicates that it should not be built.

This is all correct.

$ sudo dkms status
ntsync/1, 6.12.57+deb13-amd64, x86_64: installed
nvidia/590.48.01, 6.12.57+deb13-amd64, x86_64: installed
nvidia/590.48.01, 6.17.8+deb13-amd64, x86_64: installed
tuxedo-drivers/4.18.1, 6.12.57+deb13-amd64, x86_64: installed
tuxedo-drivers/4.18.1, 6.17.8+deb13-amd64, x86_64: installed
tuxedo-yt6801/1.0.30tux5, 6.12.57+deb13-amd64, x86_64: installed
tuxedo-yt6801/1.0.30tux5, 6.17.8+deb13-amd64, x86_64: installed

Same, all correct.

$ sudo apt-get remove ntsync-kernel-dkms
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
ntsync-kernel-dkms
0 upgraded, 0 newly installed, 1 to remove and 1 not upgraded.
After this operation, 45.1 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 285845 files and directories currently installed.)
Removing ntsync-kernel-dkms (6.14.0~repack-2) ...
Module ntsync/1 for kernel 6.12.57+deb13-amd64 (x86_64):
Before uninstall, this module version was ACTIVE on this kernel.
Deleting /lib/modules/6.12.57+deb13-amd64/updates/dkms/ntsync.ko.xz
Running depmod.... done.
Module ntsync/1 is not built for kernel (). Skipping...

_(please notice the empty kernel version in the line above)_

DKMS status will report the error:

$ sudo dkms status
ntsync/1: broken

Error! ntsync/1: Missing the module source directory or the symbolic link pointing to it.
Manual intervention is required!
nvidia/590.48.01, 6.12.57+deb13-amd64, x86_64: installed
nvidia/590.48.01, 6.17.8+deb13-amd64, x86_64: installed
tuxedo-drivers/4.18.1, 6.12.57+deb13-amd64, x86_64: installed
tuxedo-drivers/4.18.1, 6.17.8+deb13-amd64, x86_64: installed
tuxedo-yt6801/1.0.30tux5, 6.12.57+deb13-amd64, x86_64: installed
tuxedo-yt6801/1.0.30tux5, 6.17.8+deb13-amd64, x86_64: installed

Which kernel are you running when executing the remove command? The 6.17 which skips the build?

The situation may be manually fixed with the following commands:

$ sudo mkdir /usr/src/ntsync-1
$ sudo dkms remove ntsync/1 --all

The issue is that the module is removed while some link is still pointing at it; so my guess is that you removed the module package while running the kernel version that does not need it (6.17+) and so due to some corner case the symlink/var/lib/dkms/ntsync/1/source is not removed (along with the /var/lib/dkms/ntsync folder).

When you reach this:

$ sudo dkms status
ntsync/1: broken

It would be helpful if you could also do:

# uname -r
# ls -aR /var/lib/dkms/ntsync/1/

[morgwai]

Thanks for the reply! :)

Which kernel are you running when executing the remove command? The 6.17 which skips the build?

Your guess is correct, sorry for not mentioning it before.

The issue is that the module is removed while some link is still pointing at it; so my guess is that you removed the module package while running the kernel version that does not need it (6.17+) and so due to some corner case the symlink/var/lib/dkms/ntsync/1/source is not removed (along with the /var/lib/dkms/ntsync folder).

When you reach this:

$ sudo dkms status
ntsync/1: broken

It would be helpful if you could also do:

# uname -r
# ls -aR /var/lib/dkms/ntsync/1/

Please see below:

$ sudo dkms status
ntsync/1: broken

Error! ntsync/1: Missing the module source directory or the symbolic link pointing to it.
Manual intervention is required!
nvidia/590.48.01, 6.12.57+deb13-amd64, x86_64: installed
nvidia/590.48.01, 6.17.8+deb13-amd64, x86_64: installed
tuxedo-drivers/4.18.1, 6.12.57+deb13-amd64, x86_64: installed
tuxedo-drivers/4.18.1, 6.17.8+deb13-amd64, x86_64: installed
tuxedo-yt6801/1.0.30tux5, 6.12.57+deb13-amd64, x86_64: installed
tuxedo-yt6801/1.0.30tux5, 6.17.8+deb13-amd64, x86_64: installed
$ uname -r
6.17.8+deb13-amd64
$ ls -aR /var/lib/dkms/ntsync/1/
/var/lib/dkms/ntsync/1/:
.  ..  6.12.57+deb13-amd64  source

/var/lib/dkms/ntsync/1/6.12.57+deb13-amd64:
.  ..  x86_64

/var/lib/dkms/ntsync/1/6.12.57+deb13-amd64/x86_64:
.  ..  log  module

/var/lib/dkms/ntsync/1/6.12.57+deb13-amd64/x86_64/log:
.  ..  make.log

/var/lib/dkms/ntsync/1/6.12.57+deb13-amd64/x86_64/module:
.  ..  Module.symvers  ntsync.ko.xz

Just to be 100% clear, I have 2 kernels and their headers installed: Trixie's base 6.12.57+deb13-amd64 and 6.17.8+deb13-amd64 from trixie-backports.

[anbe42]

From dkms(8)

OBSOLETE_BY=
This directive allows you to specify a kernel version that obsoletes the necessity for this particular DKMS module. This can be specified as a particular upstream kernel or an ABI bump of a kernel. For example, "2.6.24" would be an upstream
kernel and "2.6.24-16" would represent an ABI bump for a kernel. Both are valid in this area.

Please consider using BUILD_EXCLUSIVE_KERNEL_MAX instead of OBSOLETE_BY as the former is a more general approach to solve the same problem.

Please avoid the use of OBSOLETE_BY wherever possible. It's use indicates a lack of proper module versioning using MODULE_VERSION() tags in the module source itself. It is better to fix the MODULE_VERSION() tags than use OBSOLETE_BY. This also introduces a implicit distribution/version dependency on the package, as the value of OBSOLETE_BY is meaningful only in the context of a single distribution/version.

OBSOLETE_BY is not covered at all by the testsuite. Given the description in the manpage I've never considered it as a possible approach for my problems and thus I've never tried to understand how it is (supposed to) work. Instead I implemented BUILD_EXCLUSIVE_KERNEL_{MIN,MAX}

[anbe42]

Which version of the dkms package are you using?

dpkg --list dkms

I vaguely remember fixing the "empty version + arch" being reported.

[morgwai]

@anbe42

Which version of the dkms package are you using?

dkms-3.3.0-1 (from Nvidia DC repos)

From dkms(8)

(...)

Please consider using BUILD_EXCLUSIVE_KERNEL_MAX instead of OBSOLETE_BY as the former is a more general approach to solve the same problem.

Thanks for pointing this! I could swear this text was not there when I was reading this man-page back in May, but of course I would be wrong ;-] Just another example of my progressing brain dysfunction ;-)
Indeed replacing it with BUILD_EXCLUSIVE_KERNEL_MAX solves the problem. Having the knowledge that I should look up to BUILD_EXCLUSIVE_KERNEL_MAX, also has made it easy to find the root cause of this bug: will send a PR as soon as I manage to force my old Github SSH key to work (or give up and configure a new one or something).

[scaronni]

OBSOLETE_BY is not covered at all by the testsuite. Given the description in the manpage I've never considered it as a possible approach for my problems and thus I've never tried to understand how it is (supposed to) work. Instead I implemented BUILD_EXCLUSIVE_KERNEL_{MIN,MAX}

@anbe42 shall we deprecate OBSOLETE_BY by making it an alias of BUILD_EXCLUSIVE_KERNEL_MAX?

• Point it out in the man page
• Print a warning that it's superseded by BUILD_EXCLUSIVE_KERNEL_MAX
• Make it behave like BUILD_EXCLUSIVE_KERNEL_MAX

[morgwai]

@scaronni OBSOLETE_BY is slightly different than BUILD_EXCLUSIVE_KERNEL_MAX: the part of the code where the bug was is exactly responsible for that: OBSOLETE_BY is completely ignored if any of the modules have their individual BUILD_EXCLUSIVE_KERNEL_MAX[#] defined (as opposed to BUILD_EXCLUSIVE_KERNEL_MAX, which serves as a default for the remaining modules in such case). So in theory making them exactly the same may break a few packages out there, but I would not be surprised if there are none such.

OBSOLETE_BY is also passed to check_version_sanity at line 1771, while BUILD_EXCLUSIVE_KERNEL_MAX is not, but I haven't yet figured out how it affects behavior in practice: it seems to me this check may be currently redundant, but not sure yet.

I think just to be 100% safe, it's better to start with clearly marking OBSOLETE_BY as obsolete and printing a notice similarly to CLEAN, but wait another 6 months or so with changing the behavior to exactly match BUILD_EXCLUSIVE_KERNEL_MAX.