Need another escape hatch for experimental

[odersky]

@experimental definitions and experimental language imports can currently only be used on code that is itself @experimental or that is compiled with a nightly version of the compiler.

This effectively locks experimental away in the poison cabinet.

For instance, I see no practical scheme to use experimental features in the compiler itself. This is a shame. First, there is no issue in using experimental features in the compiler. If they change, we can change the compiler as well, it's the same repo. So the stability argument does not apply. Second, we forgo the large benefits of dog-fooding, which means that experimental features are pushed out with much less testing and user experience than would have been the case otherwise.

To change that, I propose we create an setting -Yexperimental which would make the compiler pretend it is a nightly version. I am not sure what this should mean for the Tasty version, however.

[bishabosha]

Perhaps we can check that the source file of the code is in a git repo matching the dotty repository?

[odersky]

That's one option. But it means we still put up very high hurdles for everyone else.

[smarter]

-Yexperimental could act as if all classes and traits were defined with @experimental.

[smarter]

... one issue there is that part of metals depend on the compiler and would need to be experimental too, so this would make it more important to have a stable API for the presentation compiler that metals can call into.

[prolativ]

This idea seems familiar #14306 (comment)

[sjrd]

part of metals depend on the compiler

As well as every compiler plugin out there, for example Scala Native.

[sjrd]

Also tooling such as Metals will not support these things. So if you do use them in the compiler, you cannot expect Metals to work for the compiler codebase anymore. Nor any IDE, for that matter.

[odersky]

Also tooling such as Metals will not support these things. So if you do use them in the compiler, you cannot expect Metals to work for the compiler codebase anymore. Nor any IDE, for that matter.

Not necessarily. All I want is refer to some classes that are marked @experimental. I don't see why Metals would stop working on the compiler codebase that way.

If we do introduce experimental language imports then it's different. It depends what they are. If they affect syntax it's possible that IDEs would not longer recognize this correctly. But that's not a complete show-stopper either. I have been working with occasional red underlines from metals for a long time previously.

If we label the compiler transitively as experimental then I agree it won't work. We need to have a way to override experimental rules locally without the codebase becoming itself experimental.

[odersky]

So, the radical option would indeed be to compile the compiler with -Yexperimental, which would force all plugins and other code using the compiler to be compiled with -Yexperimental as well. Actually, for metals it would only be those parts that rely on compiler internals, but it would take some work to factor them out cleanly.

In a sense that's just being honest. Internal compiler APIs are not stable, and can be changed at any time, which is our definition of what experimental is. Maybe that would push the compiler maintainers to develop richer stable APIs and other tools to use these APIs instead of compiler internals. But that's a process that requires buy in from many parties.

In summary I see the following options:

• Add -Yexperimental in the sense that every compiled unit is treated as if it was annotated @experimental. That allows applications to use experimental features but it's not an attractive options for libraries or the compiler itself (since that one is also a library that others use).
• Have a special exclude that allows the compiler to access experimental features without itself being experimental. The idea is that if an experimental feature changes, the compiler will change as well. Since everybody using the compiler needs to be prepared for any changes in the compiler, this is OK.
• Make that special exclude also available to others in some way.

What do you think? Which (combinations of) options should we go for?

[soronpo]

In any case, I think this should be brought up as a SIP for a committee vote.

[makkarpov]

-Yexperimental could act as if all classes and traits were defined with @experimental.

-Yskip:crossVersionChecks just acts as if @experimental has never existed.