Current arm64 6.6.79 (6.6) image is faulty

[florianl]

Minimal reproducer:

$ echo "FROM ghcr.io/cilium/ci-kernels:6.6.79" | docker buildx build --platform linux/arm64 --quiet --pull --output="./ci-kernels/arm64/6.6.79/" -
$ qemu-system-aarch64 -nographic  -append "console=ttyAMA0"  -M virt -cpu cortex-a57 -m 4G -kernel ./ci-kernels/arm64/6.6.79/boot/vmlinuz 

If one replace 6.6.79 with 6.6.41 in the above reproducer one can verify that the previous kernel image was working fine.
With the 6.6.79 image qemu-system-aarch64 does not even try to boot the kernel, as far as I can tell.

[lmb]

I've re-run the build: https://github.com/cilium/ci-kernels/actions/runs/13507765667 Does that fix it? Seems like sometimes the packaging goes wrong? No idea what it could be.

[florianl]

Looks like triggering a rebuild did not help. vmlinuz of 6.6.41 is of size 24M while 6.6.79 is still at 9.6M, so significantly smaller.

[lmb]

Does compiling on your machine reproduce as well? You should be able to use buildx.sh as outlined in the README.

[florianl]

I did run into multiple issues trying to build the ci-kernels locally - e.g. https://gist.github.com/florianl/2ff809fe4eb11d48e757821c7ae1ec98

Would you be open, to extend the CI and add a test to check the produced arm64 kernel? Something like this:

  test-arm64:
    name: Test ARM64 Kernel
    needs: build-and-push
    runs-on: ubuntu-24.04-arm
    if: ${{ inputs.build_arm64 == true }}
    
    permissions:
      contents: read
      packages: read
    
    steps:
    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v3
    
    - name: Set up Go
      uses: actions/setup-go@v4
      with:
        go-version: 'stable'
    
    - name: Install vimto and dependencies
      run: |
        sudo apt-get update
        sudo apt-get install -y --no-install-recommends qemu-system-aarch64
        CGO_ENABLED=0 go install lmb.io/vimto@latest
    
    - name: Fetch precompiled kernel
      run: |
        install -d ci-kernels
        echo "FROM ghcr.io/${{ github.repository_owner }}/ci-kernels:${{ inputs.kernel_version }}" \
          | docker buildx build --platform linux/arm64 \
            --quiet --pull --output="ci-kernels" -
    
    - name: Test kernel with vimto
      run: |
        kernel="$(pwd)/ci-kernels/boot/vmlinuz"
        echo "Testing kernel at $kernel"
        vimto -kernel "$kernel" exec -- /bin/true && echo "ARM64 kernel test successful"

[lmb]

Weird, I just tried it on my machine and it works. Kind of looks like a network connectivity issue?
I'd be happy to add a CI check, but I think the test has to happen before we push the image to the registry because that only happens on main builds. As far as I know this is kind of cumbersome to achieve in actions, because sharing the image between jobs is a pain.

[florianl]

On Fedora 40 I get the error from the linked gist. And on Debian 12 I do get the following error:

=> WARN: Requested platform "linux/amd64" does not match result platform "linux/arm64"                                                                                                                       0.0s
  1 warning found (use --debug to expand):
  - Requested platform "linux/amd64" does not match result platform "linux/arm64"

Time is limited atm so I can't debug docker buildx ....

Fair point - a CI test should happen before the image is pushed to the registry. And I agree, that handling docker builds in CI is tedious.

[ti-mo]

@florianl Is this still an issue?

[florianl]

Just tested some of the current builds.

The images for 6.12.40, 6.15.8 and 6.16 are just fine. But the images for 6.6.100, 6.1.147 and 5.15.189 fail as described in the initial comment. So I would consider this still an issue.