Improved support for cross compilation

[Popax21]

Is your feature request related to a problem? Please describe.
Currently, cross compilation using crane requires that the user sets up a bunch of environment variables on their own (see the cross compilation example on crane.dev).
This is an additional setup step required by all projects which want to allow users to cross-compile, and in most cases leads to non-target-independent derivations (the above-linked example is hardcoded to compile for aarch64 for example - it will pull in an aarch64 compiler even if the user does not want to cross compile, and changing the cross compilation target requires manual adjustment of the environment variables since they hard-code the rustc target).
Additionally, upstream flakes / repos using crane which have not configured cross compilation can't easily be cross compiled without patching the upstream repo.

Describe the solution you'd like
Ideally, cross compilation should "just work", as in not require any adjustments compared to a minimal native-only build setup - all that should be required to cross-compile a project is plugging in an instance of nixpkgs which has been set up for cross compilation (like e.g. a member of pkgsCross) into crane.mkLib.
Crane then automatically sets the required environment variables to make cross compilation work, based on the instance of nixpkgs it has been given (this is similar to how nixpkgs's buildRustPackage currently works).

Describe alternatives you've considered
Keep the current system of delegating the responsibility of configuring cargo's toolchain to the user.
This gives the user more control by default, however a setup where cross compilation "just works", with the the user being able to override any default options they want to change, would still be preferable in my opinion (additionally, a "kill switch" can be added to disable any default cross compilation settings, for users who want to configure cross compilation completely on their own).

Additional context
I have already implemented this myself for one of my projects, however I've decided to open an issue first instead of opening an unsolicited PR for an potentially unwanted feature.
If this is something upstream crane is interested in I would be happy to PR said implementation. :)

[dpc]

I do support cross-compilation on top of crane in https://github.com/rustshop/flakebox , among many things (support for cargo build profiles, etc.)

The cross-compilation of Rust stuff is relatively easy, but cross-compilation of C/C++ dependencies requires setting up all sorts of env variables and then in practice fixup based on combination of bugs in C/C++ toolchains, C/C++ source code, exact building system wrapping C compilation (build.rs).

Flakebox had to invent some tooling to make some C/C++ building stuff universally switchable: e.g. https://github.com/rustshop/flakebox/blob/master/lib/universalLlvmConfig.nix for this to work, and it generally takes a "set up env vars for each supported target" approach

One user contributed a better support for pkgsCross, but I wasn't happy about how it works under the hood and never had time to redoit: rustshop/flakebox#161

Having some more universal cross-compilation support somewhere makes sense. In Flakebox I've coupled it with bunch of other things, which maybe would make sense to have separately and then re-use.

[Popax21]

The cross-compilation of Rust stuff is relatively easy, but cross-compilation of C/C++ dependencies requires setting up all sorts of env variables and then in practice fixup based on combination of bugs in C/C++ toolchains, C/C++ source code, exact building system wrapping C compilation (build.rs).

That's been my experience as well, however setting up the appropriate env variables for the most common C/C++ build crates (like cc) gets you most of the way there in practice (other aspects of the cross-compiling setup like pkg-config / etc. should already be covered by nixpkgs own support). And even if this does not work out of the box in all cases, like if a project depends on a more obscure build crate, they can always set the appropriate env variables themselves on top of the ones crane would set (or even better, they could PR in these additional environment variables into crane so that upstream can always set them as well). Disregarding all of the above for a second, even having crane handle cross compilation just for pure Rust projects would already be an improvement compared to the status quo in my opinion.

[ipetkov]

Hi @Popax21 thanks for the discussion! I think at a high level my decision to include cross-compilation support that "just works" will boil down to:

• how difficult is it to implement and maintain (e.g. are we just setting some env vars? how difficult is the logic to select those)
• how reliable it is (does it actually "just work" or does it cause more headaches when things go wrong?)
• how easy it might be for a user to at least understand what is going wrong without having to understand/reverse engineer the default behavior
• whether it is opt-in or opt-out by default

So at a high level, could you describe what additional behavior/implementation you'd like to see being done by default?

---

I've somewhat intentionally avoided baking in cross-compilation support (and left it with an example) because of all the perils it is normally associated with like:

• setting up cargo variables (the easiest part!)
• making sure that dependencies are correctly split up between depsBuildBuild, buildInputs, nativeInputs (and hopefully not many others)
• making sure that said inputs are actually spliced in correctly (e.g. via callPackage and not just pkgs.foo)
• making sure cargo is instructed to build the correct target (either via cli flag or environment variable)
• making sure the stdenv is capable and configured for cross compilation
• making sure C/C++ compilers and linkers are set up right (usually this means setting up things like HOST_CC and TARGET_CC but sometimes that might include things like TARGET_AR or other equivalents)
  • what makes things worse is different ecosystems treat some of these slightly differently. what nixpkgs typically refers to as "host dependencies" (i.e. the platform that will run the code) is often called the "target" in the Rust ecosystem (i.e. the "target of the compiler's output" which is the host that will run the code; nixpkgs calls "target" the platform which if the current project is a compiler what it would produce outputs for)
• all of this also assumes that downstream crates are compatible for cross-compilation (i.e. will honor cross compilation variables) and will not do silly things like assume it can download prebuilt binaries, and that it will honor configs/variables that are commonly used and not require their own bespoke variables to make things work right

I'm totally open to changing my mind around this, but the goal of "just works" can be a pretty tall order once you start trying to handle more and more quirks in a general purpose way without any of the context and understanding a project maintainer would have in deciding the correct way to solve things for their specific use case. Some of the problems above are things that we could influence, but others there is little we can do by the time the derivation is being evaluated.

[dpc]

As flakebox shows, all of this can be (attempted at least) achieved in an external crate. Flakebox is a bit opinionated, includes some extra stuff (CI, etc.), and is tailored to suit our needs at $dayjob, so I don't want to hack and slash it now. I also reserve a right to not knowing what I am doing w.r.t ideal Nix xcompilation. If someone wants to take a lead at something more proper, I would be happy to try to help, and eventually switch Flakebox to use this new crane-xcompile crate as a base, instead of crane itself. But it really comes down to someone being able to put the effort.

[Popax21]

Hey, thanks yall for taking the time to engage with this! As briefly mentioned in the original issue text I already have a (WIP) implementation of this feature I created for one of my projects (in retrospective I should have put more emphasis on this in the original issue - this issue is mostly about checking if upstream crane would be interested in adopting such a feature before I put more effort into a complete / generic implementation for a PR), and as such I think I can answer some of those questions:

how difficult is it to implement and maintain (e.g. are we just setting some env vars? how difficult is the logic to select those)

It's pretty much just that - setting some environment variables to hook up the Rust build process to standard nixpkgs cross-compilation infrastructure. Namely, this involves configuring cargo's build target / linker / emulator settings, as well as setting environment variables for the cc crate (namely HOST_CC / TARGET_CC and related variables), which as far as I can tell is the standard way to integrate C/C++ components into the Rust build process. The logic for selecting their values is pretty straight forward - all of the information is pulled straight out of nixpkgs platform descriptors / appropriate stdenv environments - so it should be maintainable with even just a rough grasp of how nixpkgs' platform mechanisms work.

how reliable it is (does it actually "just work" or does it cause more headaches when things go wrong?)

In my experience it has worked just fine for anything I have thrown at it since I first wrote it. Of course a world where it always "just works" is an unachievable utopia, so I have no doubts that some more obscure crates / build processes will break this, but even in those cases it's almost certainly an issue of some additional required configuration being missing instead of the default configuration being wrong, so setting those environment variables by default still gets you closer to a functional cross-compilation setup than not doing anything (which is guaranteed to not work no matter the crate/...).

how easy it might be for a user to at least understand what is going wrong without having to understand/reverse engineer the default behavior

I'm not quite sure what you are referring to with this. Currently, when attempting to build a crane derivation without having made any cross compilation adjustments one will most likely just receive an error from cargo complaining that cc can't be found. Setting those environment variables by default would allow the build to progress further, which might cause some more "cursed" errors to appear, but these errors aren't caused by crane configuring the Rust toolchain by default - they would have just as well occurred if a user configured cross compilation manually in a similar manner to the current example.

If the user does encounter a rare circumstance where the toolchain configured by crane does indeed cause an error (because a crate, for example, needs to be built with either gcc or clang specifically), then they're free to overwrite the relevant environment variables themselves in the derivation just as they would do with the current state of things (depending on the targeted level of complexity for this feature it might also be possible for crane to allow one to e.g. configure which stdenv to use, to allow derivations to configure the toolchain in a more ergonomic manner themselves). However, I highly doubt that these crates are very common, and that they wouldn't cause issues in a regular native build as well.

whether it is opt-in or opt-out by default

There's no inherent aspect of this feature which would require it to be either opt-in or opt-out. Personally, I would prefer if it were opt-out, since this should allow most crates to cross-compile out of the box, and of the few remaining crates for which this is not the case, there will be even fewer where the settings set by crane would be a cause of additional problems.

To summarize what such a feature would look like once again, it would pretty much just be a matter of setting the environment variables set in the cross compilation example automatically, in a similar manner as to how nixpkgs' Rust infrastructure does it right now. While this obviously won't create a "it just works" scenario for >all< crates, it should create such a scenario for >most< crates, and remove part of the setup required for crates where it does not work out of the box.

I'm currently on vacation, so I don't have access to the source code of my current WIP implementation, but I would be happy to provide concrete example Nix code to illustrate how an implementation might look like once I'm back home. If deemed acceptable, I would be willing to polish it up further and turn it into a PR.

---

To briefly address your second set of points: obviously all of this will be in vain if the derivations themselves aren't properly set up for cross compilation / the crates don't support it out right. However, for derivations which are written "correctly" (as in using callPackage, with correct dependency splitting / etc.), this should hopefully make setting up cross compilation a lot easier. Also note that this requirement of derivations having to be set up correctly is just as much of a requirement for manual setups (the status quo), and this feature would not affect native builds at all (i.e. it would not force them to suddenly adhere to correct Nix package setup standards - all of this only affects cross compilation).

[ipetkov]

I'll happily review a PR for setting the environment variables to appropriate default values (but otherwise prioritize what the caller has set)!

[Popax21]

I'll happily review a PR for setting the environment variables to appropriate default values (but otherwise prioritize what the caller has set)!

I've now opened #794, which is based on my original implementation of this feature I developed for my own projects. It should serve as a good base for further discussion / rounds of adjustments / bikeshedding, if required.

[eureka-cpu]

Hey there, I'm attempting to get basic wasm32 cross compilation working using crane+fenix and I'm running into a few errors. I do get a warning about package splicing which is how I found this issue/PR, but I wonder if the correct pattern here is to create an overlay from fenix or not. The fenix library only includes rust-analyzer-nightly in its overlay, since the toolchain itself is custom. Any help is appreciated.

Here's my expression:

let
  inherit (pkgs) lib stdenv;

  sources = import ./npins;
  system = builtins.currentSystem;

  pkgs =
    let
      pkgs = import sources.nixpkgs {
        localSystem = { inherit system; };
      };
    in
    pkgs.pkgsCross.wasi32;

  fenix = import sources.fenix {
    inherit system pkgs;
  };
  toolchain = with fenix; combine [
    (stable.withComponents [
      "rust-analyzer"
      "rust-src"
      "cargo"
      "rustc"
      "rustfmt"
      "clippy"
    ])
    targets.wasm32-unknown-unknown.latest.rust-std
  ];
  crane =
    let
      crane = import sources.crane {
        inherit pkgs;
      };
    in
    crane.overrideToolchain toolchain;

  lathe =
    { lib
    , stdenv
    , libiconv
    , autoPatchelfHook
    }:
    crane.buildPackage {
      src = crane.cleanCargoSource ./.;
      strictDeps = true;

      nativeBuildInputs = [
        # Additional compile time deps
      ] ++ lib.optionals stdenv.isDarwin [
        # macOS only compile time deps
        libiconv
      ] ++ lib.optionals stdenv.isLinux [
        # linux only compile time deps
        autoPatchelfHook
      ];

      buildInputs = with pkgs; [
        # runtime dependencies
      ];
    };
in
{
  default = pkgs.callPackage lathe { };

  devshell = crane.devShell {
    packages = with pkgs; [
      npins
    ];
  };
}

nix-build:

error: Cannot build '/nix/store/6111pp5flbhl8g7xc074byz2x32g6j8v-lathe-deps-static-wasm32-unknown-wasi-0.1.0.drv'.
       Reason: builder failed with exit code 127.
       Output paths:
         /nix/store/nn7dlx8vgjnc4ng7z88nly1vazp70lw0-lathe-deps-static-wasm32-unknown-wasi-0.1.0
       Last 25 log lines:
       > AR_WASM32_WASIP1=wasm32-unknown-wasi-ar
       > AR_X86_64_UNKNOWN_LINUX_GNU=ar
       > CARGO_BUILD_TARGET=wasm32-wasip1
       > CARGO_TARGET_WASM32_WASIP1_LINKER=wasm32-unknown-wasi-cc
       > CARGO_TARGET_WASM32_WASIP1_RUNNER=/nix/store/fry49nv1w7x3w3pd4srd9dk5kk6id7zd-wasmtime-42.0.1/bin/wasmtime
       > CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=cc
       > CC_WASM32_WASIP1=wasm32-unknown-wasi-cc
       > CC_X86_64_UNKNOWN_LINUX_GNU=cc
       > CXX_WASM32_WASIP1=wasm32-unknown-wasi-c++
       > CXX_X86_64_UNKNOWN_LINUX_GNU=c++
       > HOST_AR=ar
       > HOST_CC=cc
       > HOST_CXX=c++
       > TARGET_AR=wasm32-unknown-wasi-ar
       > TARGET_CC=wasm32-unknown-wasi-cc
       > TARGET_CXX=wasm32-unknown-wasi-c++
       > ----------------------------------------------------------------------------------
       > Running phase: updateAutotoolsGnuConfigScriptsPhase
       > Running phase: updateAutotoolsGnuConfigScriptsPhase
       > Running phase: configurePhase
       > will append /build/source/.cargo-home/config.toml with contents of /nix/store/p44r6zirc60p8kw3nqs3hfbiqym8gimj-vendor-cargo-deps/config.toml
       > default configurePhase, nothing to do
       > Running phase: buildPhase
       > +++ command cargo --version
       > /nix/store/7hbjid69agb2b8ag195psng3blw2aqw1-cargoHelperFunctionsHook/nix-support/setup-hook: line 6: /nix/store/mf0539n7iwza5minf75fwzi9bv4bs34i-rust-mixed/bin/cargo: cannot execute: required file not found
       For full logs, run:
         nix log /nix/store/6111pp5flbhl8g7xc074byz2x32g6j8v-lathe-deps-static-wasm32-unknown-wasi-0.1.0.drv
error: Cannot build '/nix/store/jrmy0iwqnaakimm8h8qybhkpvv2v70d3-lathe-static-wasm32-unknown-wasi-0.1.0.drv'.
       Reason: 1 dependency failed.
       Output paths:
         /nix/store/kb43j9gyhgm1w9hn358ss5dqprk7rnnz-lathe-static-wasm32-unknown-wasi-0.1.0
error: Build failed due to failed dependency
FAIL

nix-shell:

Could not start dynamically linked executable: cargo
NixOS cannot run dynamically linked executables intended for generic
linux environments out of the box. For more information, see:
https://nix.dev/permalink/stub-ld
FAIL

[eureka-cpu]

OK, my issue is that I was using pkgsCross, when in this case I should be just using pkgs.