signature from <package> is invalid

[naclander]

Hey,

Sometimes when using flexo, I've noticed errors when my machine is validating packages in the form of:

error: <package>: signature from "<person>" is invalid
:: File /var/cache/pacman/pkg/<package>.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n] 

Neither Y nor n fixes the problem. Clearing out the flexo cache does.
Re-downloading the archlinux-keyring also does not fix this issue.

Could this be a stale package cached in flexo somehow? Is there any mitigation for this instead of clearing out the flexo cache?

[nroi]

Edit: This is either caused by misbehaving mirrors or a bug in flexo, but I think a bug in flexo is more likely. I'm looking into this issue, but I still haven't found a way to reproduce it. If anyone stumbles on this issue, you can help me troubleshoot this issue as follows:

1. Run the flexo-verify-package python script on the machine that's running flexo and paste the output of that script.
2. The journalctl log (or output from docker logs if you use docker) from the time frame when the download has happened, e.g. journalctl --unit=flexo --since today.

The more output I receive, the better!

---

Original answer:

Could this be a stale package cached in flexo somehow?

I guess so. Probably either the package itself, or the signature (the corresponding file ending with .sig) has changed, but flexo serves the stale version from cache.

I'd like to find out if the package file (the file ending with .zst), or the signature file (ending with .sig), or both files are stale. Next time when this issue happens, can you please check which files change after you clear out the flexo cache? For example, you can run sha256sum <package>.zst and sha256sum <package>.zst.sig before clearing the flexo cache (replace <package> by any package from the error message shown by pacman), and then run the same two commands again after clearing the flexo cache to compare the checksums and see which files have changed.

Once I understand the problem, I can think of a solution.

[naclander]

So it just happened again. This happens on the client machine, during the Checking package Integrity step:

error: linux-lts: signature from "Andreas Radke <andyrtr@archlinux.org>" is invalid
:: File /var/cache/pacman/pkg/linux-lts-5.15.45-1-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n] 
error: failed to commit transaction (invalid or corrupted package)
Errors occurred, no packages were upgraded.

The sha256sum before clearing out the package cache for all matching files is:

# sha256sum $( find ./ | grep linux-lts-5.15.45-1-x86_64.pkg.tar.zst | xargs)
d83324c5710c0b59371a53194821d31ff38a50e18af17364248add71ba011d6a  ./core/os/x86_64/linux-lts-5.15.45-1-x86_64.pkg.tar.zst.sig
bfa9b234f73416c3f7d3bb668b28d25467e1007087aadc3d8571a9c838ac316a  ./core/os/x86_64/.linux-lts-5.15.45-1-x86_64.pkg.tar.zst.sig.cfs
1fc9438708ffeabaa9a9bdb95a8cb6e8df922838f8f3f62b30c34180a0d18f8c  ./core/os/x86_64/.linux-lts-5.15.45-1-x86_64.pkg.tar.zst.cfs
1af8f502f87da3c76a3cb5e25f15a90c7ecacb42b535842889aee85077f1dbb7  ./core/os/x86_64/linux-lts-5.15.45-1-x86_64.pkg.tar.zst

I cleared out the pkg cache and restarted the flexo service, ran pacman -Sc on the client, and re-ran the update on the client.
Running the sha26sum command on the flexo server again, I see:

# sha256sum $( find ./ | grep linux-lts-5.15.45-1-x86_64.pkg.tar.zst | xargs)
d83324c5710c0b59371a53194821d31ff38a50e18af17364248add71ba011d6a  ./core/os/x86_64/linux-lts-5.15.45-1-x86_64.pkg.tar.zst.sig
bfa9b234f73416c3f7d3bb668b28d25467e1007087aadc3d8571a9c838ac316a  ./core/os/x86_64/.linux-lts-5.15.45-1-x86_64.pkg.tar.zst.sig.cfs
8bdfbbf5bdff5d2ff5863ee25ff027c3532a14b445df39de3af3e0f7265f0c35  ./core/os/x86_64/.linux-lts-5.15.45-1-x86_64.pkg.tar.zst.cfs
c825a66094df4458f410c2bf2f04753a7ed7bb4eefe896d6fbff43ed229a1737  ./core/os/x86_64/linux-lts-5.15.45-1-x86_64.pkg.tar.zst

[nroi]

Ok, so the output tells us that:

1. the signature file has not changed, neither locally nor at the mirror.
2. the package file itself has changed both locally and at the remote mirror (this can be seen from the .cfs file, which includes the content length sent by the mirror. The .cfs file has changed, therefore, the package must also have changed at the remote mirror).

If the mirror itself served an incomplete file, this would explain this issue. Can you please run the following and show me the output?

journalctl --unit=flexo | grep Primary | tail -n20

I'd like to investigate further if any of those mirrors have issues that would explain this before ruling out that this is a bug in flexo.

If you want to try a different mirror, you can configure flexo to blacklist the previously used mirrors. Edit /etc/flexo/flexo.toml and add the mirrors to the mirrors_blacklist array. Copy the exact values that you got from the previous journalctl command, i.e., with the trailing slash. Then, remove the latency results:

sudo rm /var/cache/flexo/state/latency_test_results.json

Then, restart flexo, so it runs the latency tests again and picks up another mirror. Wait a moment until the latency tests have completed before running pacman.

[naclander]

# journalctl --unit=flexo | grep Primary | tail -n20
Apr 25 20:17:05 dmz-cache flexo[10406]: [2022-04-25T20:17:05.363Z INFO  flexo] Primary mirror: "https://arch.hu.fo/archlinux/"
Apr 25 20:17:11 dmz-cache flexo[10421]: [2022-04-25T20:17:11.610Z INFO  flexo] Primary mirror: "https://arch.hu.fo/archlinux/"
Apr 25 20:17:17 dmz-cache flexo[10631]: [2022-04-25T20:17:17.999Z INFO  flexo] Primary mirror: "https://arch.hu.fo/archlinux/"
Apr 25 20:17:24 dmz-cache flexo[12197]: [2022-04-25T20:17:24.102Z INFO  flexo] Primary mirror: "https://arch.hu.fo/archlinux/"
Apr 25 20:17:30 dmz-cache flexo[12214]: [2022-04-25T20:17:30.441Z INFO  flexo] Primary mirror: "https://arch.hu.fo/archlinux/"
Apr 25 20:17:36 dmz-cache flexo[12226]: [2022-04-25T20:17:36.606Z INFO  flexo] Primary mirror: "https://arch.hu.fo/archlinux/"
Apr 25 20:17:42 dmz-cache flexo[12239]: [2022-04-25T20:17:42.781Z INFO  flexo] Primary mirror: "https://arch.hu.fo/archlinux/"
Apr 25 20:17:48 dmz-cache flexo[12301]: [2022-04-25T20:17:48.787Z INFO  flexo] Primary mirror: "https://arch.hu.fo/archlinux/"
Apr 25 20:17:54 dmz-cache flexo[12315]: [2022-04-25T20:17:54.762Z INFO  flexo] Primary mirror: "https://arch.hu.fo/archlinux/"
Apr 25 20:18:00 dmz-cache flexo[14078]: [2022-04-25T20:18:00.850Z INFO  flexo] Primary mirror: "https://arch.hu.fo/archlinux/"
Apr 25 20:18:07 dmz-cache flexo[14090]: [2022-04-25T20:18:07.153Z INFO  flexo] Primary mirror: "https://arch.hu.fo/archlinux/"
Apr 25 20:18:13 dmz-cache flexo[14107]: [2022-04-25T20:18:13.279Z INFO  flexo] Primary mirror: "https://arch.hu.fo/archlinux/"
Apr 25 20:18:25 dmz-cache flexo[65]: [2022-04-25T20:18:25.555Z INFO  flexo] Primary mirror: "https://arch.hu.fo/archlinux/"
Apr 25 20:18:31 dmz-cache flexo[87]: [2022-04-25T20:18:31.632Z INFO  flexo] Primary mirror: "https://arch.hu.fo/archlinux/"
Apr 25 20:18:37 dmz-cache flexo[658]: [2022-04-25T20:18:37.600Z INFO  flexo] Primary mirror: "https://arch.hu.fo/archlinux/"
Apr 25 20:18:43 dmz-cache flexo[1890]: [2022-04-25T20:18:43.609Z INFO  flexo] Primary mirror: "https://arch.hu.fo/archlinux/"
May 10 03:23:14 dmz-cache flexo[58]: [2022-05-10T03:23:14.246Z INFO  flexo] Primary mirror: "https://mirror.fcix.net/archlinux/"
May 15 22:40:18 dmz-cache flexo[7743]: [2022-05-15T22:40:18.056Z INFO  flexo] Primary mirror: "https://arch.hu.fo/archlinux/"
Jun 05 18:22:22 dmz-cache flexo[65]: [2022-06-05T18:22:22.993Z INFO  flexo] Primary mirror: "https://archmirror1.octyl.net/"
Jun 09 00:59:45 dmz-cache flexo[2178]: [2022-06-09T00:59:45.662Z INFO  flexo] Primary mirror: "https://mirror.fcix.net/archlinux/"

I went ahead and updated /etc/pacman.d/mirrorlist on my flexo server. I ran reflector, so if this was the case of a bad mirror, hopefully it is gone now. I'll go ahead and remove the latency results and restart flexo.

EDIT: I just realized the mirrors potentially causing a problem ( arch.hu.fo, mirror.fcix.net/, etc. ) are still present even after updating the mirrorlist and running reflector. I suppose blacklisting them from reflector is the guaranteed way of not having this issue happen ( if it is related to those mirrors ).

Another way would just be to remove these mirrors from my mirrolist file. I am curious why these specific mirrors would be causing a problem with reflector.

EDIT2: I now see that even after pruning my /etc/pacman.d/mirrorlist file and removing mirror.fcix.net, flexi still chooses this as the primary mirror? I'm not sure how that's happening.

[nroi]

Thanks for the journalctl output.

I went ahead and updated /etc/pacman.d/mirrorlist on my flexo server. I ran reflector, so if this was the case of a bad mirror, hopefully it is gone now.

Keep in mind that flexo ignores /etc/pacman.d/mirrorlist, it fetches a list of mirrors from the official JSON API and then runs latency tests to select fast mirrors. So, if you want to avoid flexo choosing bad mirrors, you need to use the mirrors_blacklist option, as explained in my previous post.

[naclander]

Right, I see this in the [mirrors_auto]section of the toml file now.

I'm curious why not have an option to have flexo check /etc/pacman.d/mirrorlist? Like I mentioned I already use reflector so I know those mirrors work.

[nroi]

I'm curious why not have an option to have flexo check /etc/pacman.d/mirrorlist? Like I mentioned I already use reflector so I know those mirrors work.

One of flexo's goal is to minimize the time users need to invest for mirrorlist maintenance, and not everyone uses reflector. But of course, it would be possible to implement this as an additional option.

[naclander]

Just wanted to update that this is still happening, even after denylisting several mirrors in flexo. Do you know if there's a more consistent way of preventing the signature error from happening?

[nroi]

I'm still trying to figure out why it happens in the first place.

Can you please check if this script runs on your machine? https://gist.github.com/nroi/492ccf2d55400746cb8084984e04002f
It's a python script that I just wrote for this particular issue. If you get an error due to requests not being available, you need to install it through pip:

pip3 install requests

On my machine, the output is:

No mismatches were detected. Matching files: 1344

If you run this script after the signature errors happen again, you should get output like the following:

Mismatches were detected for the following files:
Filename: cowsay-3.04-2-any.pkg.tar.zst
File path: /var/cache/flexo/pkg/extra/os/x86_64/cowsay-3.04-2-any.pkg.tar.zst
Expected sha256sum: c3a5b79405c206a6e0eccbaa632e8534a6457b144dfa84eaabb29a02b8b37dbc
actual sha256sum: fe4f1fd73c81532d16c75497c8cb2702e74a8cc15d8e93b3e0eb0eda46b375d5
Expected filesize: 19990
Actual filesize: 19989
CFS filesize: 19989
Modification time: 2022-07-31 21:10:21.530959

If you get this output, please post it in this issue, so I can get a better understanding of what's going on.

The script also provides a --delete flag which you can use to just delete the broken files, this may be better than clearing the entire cache.

[naclander]

I'll run this the next time I'm getting errors ( usually to move forward I change the mirror, then change it back ).

Looking at the script, it looks like I'm supposed to run it on the machine hosting the flexo service? The way I have it set up is that I have multiple machines, and the flexo service is running on one machine dedicated to be the package server. It is not all running on localhost.

I'll also add that sometimes I get this errors after a pacman -Syu, but if I explicitly do pacman -S archlinux-keyring && pacman -Syu the issue sometimes goes away. I am not sure why.

[nroi]

Looking at the script, it looks like I'm supposed to run it on the machine hosting the flexo service?

Correct. The script checks the files downloaded by flexo, including the files with the .cfs extension, which are only available at the server running flexo.

I'll run this the next time I'm getting errors

Thanks!

[avionix-g]

I've run into this as well. I'm running Flexo in Docker. Any commands run on the Flexo cache were run on the exported container filesystem. My local Pacman cache was cleared with pacman -Scc before running the system update.

Pacman output:

sudo pacman -Syu
:: Synchronizing package databases...
 kde-unstable                                                          158.2 KiB   227 KiB/s 00:01 [##########################################################] 100%
 liquorix is up to date
 core                                                                  159.1 KiB   250 KiB/s 00:01 [##########################################################] 100%
 extra                                                                1709.3 KiB  1269 KiB/s 00:01 [##########################################################] 100%
 community                                                               6.7 MiB  1647 KiB/s 00:04 [##########################################################] 100%
 multilib                                                              176.0 KiB   285 KiB/s 00:01 [##########################################################] 100%
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...
warning: dependency cycle detected:
warning: mesa will be installed before its libglvnd dependency
warning: dependency cycle detected:
warning: lib32-mesa will be installed before its lib32-libglvnd dependency

Packages (19) aspnet-runtime-6.0.8.sdk108-1  calibre-6.3.0-1  github-cli-2.14.4-1  lib32-libglvnd-1.4.0-3  lib32-libvpx1.3-1.3.0-3  lib32-mesa-22.1.6-1
              lib32-pango-1:1.50.9-1  lib32-smpeg-0.4.5-4  libbluray-1.3.2-1  libglvnd-1.4.0-3  libnfs-5.0.2-1  libnm-1.38.4-1  mesa-22.1.6-1
              mono-msbuild-16.10.1.xamarinxplat.2021.05.26.14.00-3  networkmanager-1.38.4-1  npm-8.17.0-1  pango-1:1.50.9-1  python-tomlkit-0.11.3-1
              python-urllib3-1.26.10-1

Total Installed Size:  373.55 MiB
Net Upgrade Size:        0.72 MiB

:: Proceed with installation? [Y/n] y
(19/19) checking keys in keyring                                                                   [##########################################################] 100%
(19/19) checking package integrity                                                                 [##########################################################] 100%
error: lib32-mesa: signature from "Laurent Carlier <lordheavym@gmail.com>" is invalid
:: File /var/cache/pacman/pkg/lib32-mesa-22.1.6-1-x86_64.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n] n
error: failed to commit transaction (invalid or corrupted package)
Errors occurred, no packages were upgraded.

SHA256 checksums

Flexo filesystem:

fd lib32-mesa -X sha256sum | sort
5bb969fedcf9d194722345476cacd2937033a44758bd0be50d1193788cb95722  var/cache/flexo/pkg/multilib/os/x86_64/lib32-mesa-22.1.6-1-x86_64.pkg.tar.zst.sig
8b7fa49db9de03cadb58fbb9c9055dc31c482f80a8c38884b09f72ee73d9dc23  var/cache/flexo/pkg/multilib/os/x86_64/lib32-mesa-22.1.5-1-x86_64.pkg.tar.zst.sig
c7608a37fa0758b6b2701fd41cf77d3e8d1b119721dd1851281a8b6f04e95ea0  var/cache/flexo/pkg/multilib/os/x86_64/lib32-mesa-22.1.6-1-x86_64.pkg.tar.zst

/var/cache/pacman/pkg on my local machine:

fd lib32-mesa -X sha256sum | sort
5bb969fedcf9d194722345476cacd2937033a44758bd0be50d1193788cb95722  ./lib32-mesa-22.1.6-1-x86_64.pkg.tar.zst.sig
c7608a37fa0758b6b2701fd41cf77d3e8d1b119721dd1851281a8b6f04e95ea0  ./lib32-mesa-22.1.6-1-x86_64.pkg.tar.zst

flexo-verify-packages script output

python3 flexo-verify-packages --flexo-pkg-dir ./var/cache/flexo/pkg
No mismatches were detected. Matching files: 49

The correct hashes:

After disabling Flexo in my mirrorlist, clearing the pacman cache, and running pacman -Syu:

fd lib32-mesa -X sha256sum | sort
5bb969fedcf9d194722345476cacd2937033a44758bd0be50d1193788cb95722  ./lib32-mesa-22.1.6-1-x86_64.pkg.tar.zst.sig
9c4ed10babfb14832d040a971c62305f034328b98f6c7f2738c75e677b127cef  ./lib32-mesa-22.1.6-1-x86_64.pkg.tar.zst

[avionix-g]

Here are the differing .pkg.tar.zst files:
package_tars.zip

[nroi]

Here are the differing .pkg.tar.zst files: package_tars.zip

Interesting, the corrupted file seems to include only the end of the file. The corrupted file is 6338539 bytes large, and the last 6338539 bytes of the good file are identical to the corrupted file:

$ sha256sum bad_lib32-mesa-22.1.6-1-x86_64.pkg.tar.zst
c7608a37fa0758b6b2701fd41cf77d3e8d1b119721dd1851281a8b6f04e95ea0

$ tail -c 6338539 good_lib32-mesa-22.1.6-1-x86_64.pkg.tar.zst | sha256sum
c7608a37fa0758b6b2701fd41cf77d3e8d1b119721dd1851281a8b6f04e95ea0

The same thing seems to have happened when naclander had this issue:

tail -c 134014308 linux-lts-5.15.45-1-x86_64.pkg.tar.zst | sha256sum
1af8f502f87da3c76a3cb5e25f15a90c7ecacb42b535842889aee85077f1dbb7

1af8f502f87da3c76a3cb5e25f15a90c7ecacb42b535842889aee85077f1dbb7 is the shasum posted by naclander in this thread, which shows that only the last 134014308 bytes were downloaded.

This might be caused by a bug related to how flexo handles HTTP range requests, but so far, I wasn't able to reproduce it. I'm going to look into it.

Also, I've updated the flexo-verify-package script to consider all repositories (the multilib repo previously wasn't included, which is why no mismatches have been detected in your case).

Anyone who stumbles on this issue again could help me by providing the following info:

1. Run the flexo-verify-package python script on the machine that's running flexo and paste the output of that script.
2. The journalctl log (or output from docker logs if you use docker) from the time frame when the download has happened, e.g. journalctl --unit=flexo --since today.

Thank you both for your input to help me troubleshoot this. The most recent version (1.6.7) includes a few more log statements which might help me to isolate the issue.