Make it easier to build the standard library

[jyn514]

Proposal

This is a two part proposal. The second part depends on the first being completed, but the first part is IMO worthwhile in itself.

1. Get cd library/std && cargo +nightly build to work (or cargo +beta build --cfg bootstrap if that's easier).
2. Move library to a subtree and allow building out of tree with cargo build (using rust-toolchain as appropriate). This change is no longer planned.

Motivation

1. A common concern contributors bring up is that x.py is confusing to use, and that they don't understand why the standard library needs a special tool to build. See for example https://internals.rust-lang.org/t/contributing-changes-to-std/12829, https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/why.20is.20x.2Epy.20necessary.3F. Unlike the rest of the compiler, building the standard library once doesn't involve staging, just very specific versions of the rust compiler. In theory, it should be possible to build it with cargo directly.

2. Cloning the whole rust-lang/rust repo takes a long time and is mostly unnecessary for only working on the standard library. If you only had to clone the library/ folder, it would take a fraction of the time. It would also allow cargo build to work out of the box, since the only target left out of tree is the standard library (there's no ambiguity between stages or tools). This change is no longer planned.

Implementation

This is mostly just me making things up currently - I'll update it as I get a more solid understanding of the problem.

• Split apart compiler/ and library/ so they no longer use the same global Cargo.lock and Cargo.toml. This might require rebuilding a few dependencies for the compiler, but it will be very few and they'll all be small. This is essentially Use a separate workspace for the library crates rust#76533. @ehuss you had concerns there about rustc-dep-of-std that I didn't quite understand, do you mind explaining that a little more?
• Add a build script that detects whether the current compiler is beta or nightly; automatically enable cfg(bootstrap) for beta.

2. Make a new rust-lang/library repository. Use subtree syncs between that and rust-lang/rust. Since the library is still in-tree in the main repository, it will be easy to make changes to the compiler that require changing the standard library. This change is no longer planned.

Mentors or Reviewers

@Mark-Simulacrum for the implementation of 1; @Manishearth for the implementation of 2.
I don't think mentors make sense for forming a consensus, but pinging the relevant parties anyway: @matklad, @ehuss, @Mark-Simulacrum

Process

The main points of the Major Change Process is as follows:

• File an issue describing the proposal.
• A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
  • Finding a "second" suffices for internal changes. If however you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
  • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
• Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

Comments

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

[rustbot]

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

cc @rust-lang/compiler @rust-lang/compiler-contributors

[jyn514]

@ehuss and @nagisa brought up that a separate repository would need its own issue tracker and CI, as well as some way to adapt the existing tooling (such as rustc-perf). For all those reasons, and since subtree syncs are an ongoing maintenance burden, I've dropped the subtree portion of the MCP.

[nagisa]

@rustbot second

With the caveat that I'm not super familiar with the scope of work involved here, being able to just cargo build library/ seems very desirable.