binary.sh.tpl doesn't check that arguments are passed when used as a test

[JasonLunn]

It is possible to end up in a situation where an rb_test rule produces a shell script from binary.sh.tpl that never receives arguments. In this case, executing the rule results in a false negative - in other words, bazel test will pass even though the Ruby test has never been executed because the only thing that runs is bundle exec ruby which returns successfully.

I'm still trying to determine what I've done in my local setup to misconfigure the rb_test in this way, but having defense in depth could help others avoid losing the same time that I have tracking this down.

Something like:

if [[ -n "${TEST_BINARY:-}" && $# -le 0 ]]; then
  echo "Error! No arguments provided to test binary"
  exit -1
fi

This approach would rely on TEST_BINARY (set by test-setup.sh) to be present so that the check for missing arguments doesn't apply to vanilla rb_binary rules.

WDYT, @p0deje ?

[p0deje]

@JasonLunn This makes sense, would you mind opening a PR for this? We'll also need to do the same in binary.cmd.tpl.

[JasonLunn]

Will do.

On the subject of how I got here, it appears that the reason I don't get any args to bundle exec ruby is that I'm not specifying any in my rb_test rule explicitly.

For argument's sake, let's say my test rules reads like this:

rb_test(
    name = "basic",
    srcs = ["basic.rb"],
    deps = [
        ":test_lib",
        "@my_bundle",
    ],
)

My casual reading of https://github.com/bazel-contrib/rules_ruby/blob/main/docs/rules.md#rb_test had left me with the impression that args would be args to the test itself, which I omitted because my basic test doesn't take any arguments, and that the test would execute the equivalent of bundle exec ruby basic.rb. It seems like I'm mistaken, and that when there's no main specified, the script isn't the first argument to Ruby automatically. Is this working as intended?

Maybe the examples could be extended to demonstrate how minitest users (and anyone that is used to executing their tests without an explicit Ruby binary) should construct their test rules.

[p0deje]

My casual reading of https://github.com/bazel-contrib/rules_ruby/blob/main/docs/rules.md#rb_test had left me with the impression that args would be args to the test itself, which I omitted because my basic test doesn't take any arguments, and that the test would execute the equivalent of bundle exec ruby basic.rb. It seems like I'm mistaken, and that when there's no main specified, the script isn't the first argument to Ruby automatically. Is this working as intended?

I haven't thought about this scenario and now I think the code should be changed so that if both main and args are omitted, it makes sense to populate args with srcs rather than silently succeed by running ruby.

Maybe the examples could be extended to demonstrate how minitest users (and anyone that is used to executing their tests without an explicit Ruby binary) should construct their test rules.

Maybe @noahkawasakigoogle was running into a similar issue in #223. His proposal for rb_minitest would work better for your case as well.