Graceful exit when there are no tests to run

[mx-psi]

If you codspeed run on a Go module with no benchmarks it will fail cryptically:

❯ codspeed run --mode walltime --skip-upload -- 'go test -bench=. ./...'

►►► Preparing the environment 
Tool already installed, skipping installation
Sudo privileges are required to continue. Please enter your password.
[sudo] password for pablo.baeyens: 
Environment ready

►►► Running the benchmarks 
[INFO  codspeed_go_runner] Discovered 0 packages
[INFO  codspeed_go_runner] Total benchmarks discovered: 0
[INFO  codspeed_go_runner] Parsed 0 raw results
Perf is enabled, but failed to detect benchmarks. If you wish to disable this warning, set CODSPEED_PERF_ENABLED=false
Error: Failed to copy log file
Error: No such file or directory (os error 2)

and if you codspeed run on a Go module with all benchmarks skipped it will panic:

❯ codspeed run --mode walltime --skip-upload -- 'go test -bench=. ./...'

►►► Preparing the environment 
Tool already installed, skipping installation
Environment ready

►►► Running the benchmarks 
[INFO  codspeed_go_runner] Discovered 1 packages
[INFO  codspeed_go_runner] Total benchmarks discovered: 1
[INFO  codspeed_go_runner] Found BenchmarkSkip                  in "profiles_test.go"
[INFO  codspeed_go_runner] Generating custom runner for package: xexporterhelper
[INFO  codspeed_go_runner] Building binary for package: xexporterhelper
PASS
[INFO  codspeed_go_runner] Parsed 0 raw results
  ⠂ Running the benchmarks...                                                                                                                                                                                                                                                   
thread 'main' panicked at src/run/runner/wall_time/perf/mod.rs:491:33:
called `Result::unwrap()` on an `Err` value: No such file or directory (os error 2)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

The former I can identify with go test ./... -list=Benchmark | grep -q Benchmark but the latter I cannot: since benchmarks can be skipped in an arbitrary fashion I cannot determine statically what benchmarks would be run in the general case.

It would be useful if, when the runner parses 0 results, the error is a bit more descriptive and the exit code is 0 (or it is configurable to be zero).

[GuillaumeLagrange]

Hello again and thanks for the issue!

We are completely aligned on the fact that the error message needs to be clearer when no benchmark to upload is found.

However, we believe that running codspeed run with no benchmarks should result in an error, because it is most often something that is misconfigured.

What I'm inclined to implement would be a specific error code for this case, most likely in the 3-63 custom range, next best candidate being BSD's 78 for configuration error.
It would then be the user's responsibility to handle this specific error as they wish.

Would that be OK with you ?

[mx-psi]

Yeah that would work! I think given the discussion on open-telemetry/opentelemetry-collector#14160 (comment) maybe the error message is the only thing needed since the alternative you suggested works

[GuillaumeLagrange]

Adressed in #157

I have not added a particular error code, just a dedicated error message for now, given your last message

Error: The command did not produce any CodSpeed result to process, please check your configuration.

Details: No walltime results found in profile folder: "/tmp/profile.LhpdSIA20J.out/results".

Check out https://codspeed.io/docs/instruments/walltime for more information.