Generating coverage over a series of separate invocations (and docs for that!)

[ELLIOTTCABLE]

So, my project includes a command-line executable; and to exercise that, I don't use JavaScript tests, but shell-script ones (specifically, using bats).

Can I generate coverage reports when not executing any JavaScript test suite at all, just an arbitrary executable (i.e. covering the results of my_exec --blah)? More importantly, can I generate a single coverage report that covers multiple invocations (i.e. mocha my_tests, and then later my_exec --blah, my_exec --widget)?

If this is possible: it'd be great to see this covered in your excellent new HOWTO writeups! Command-line apps are common in npm, and getting full coverage information for them would be ideal. (=

[jwalton]

Yeah, this is definitely possible. I was going to say "easy" but I gave it a quick go and it turns out to be more complicated than I thought it would be. :)

If you check out the section on mixed coffee and JS projects, you'll find a big hint. It shows how to drive coffee-coverage from istanbul, and you can run istanbul to cover any executable (provided that executable doesn't fork a child). So I would have thought you could do:

./node_modules/.bin/istanbul cover ./node_modules/.bin/coffee -- \
    --require coffee-script/register --require coffee-coverage/register-istanbul ./my_exec.coffee

But this doesn't actually work. :P You need to require coffee-script/resister, because coffee-coverage/register-istanbul relies on it, and weirdly coffee-script loads up --requires before registering itself. This almost works, but all your coverage ends up being zeros. -_- I'll see if I can figure out how to make this work...

But, you can make this work by creating a small js file. Create, for example, launcher.js:

require('coffee-script/register');
require('coffee-coverage/register-istanbul');
require('./my_exec');

And then you can:

./node_modules/.bin/istanbul cover ./launcher.js

and this works.

As for multiple invocations, when you run istanbul report, it will load up all the coverage files in ./coverage/coverage*.json, and combine them to generate a report. So you can do:

./node_modules/.bin/istanbul cover ./launcher.js -- --blah
mv ./coverage/coverage.json ./coverage/coverage-blah.json
./node_modules/.bin/istanbul cover ./launcher.js -- --widget
mv ./coverage/coverage.json ./coverage/coverage-widget.json

Or, if you want something slightly less verbose:

rm ./coverage/coverage*.json
./node_modules/.bin/istanbul cover --include-pid ./launcher.js -- --blah
./node_modules/.bin/istanbul cover --include-pid ./launcher.js -- --widget

which should work well (provided you're not very low on PIDs. :)

Is that helpful? If you get this working, I'd gladly accept a PR for a HOWTO for running generic commands.

[ELLIOTTCABLE]

Yeah, I discovered some of the same stuff shortly after submitting this! At least, that istanbul very-usefully already supports multiple coverage.json files.

I'm going to make a go of it later on, and I'll get back to you as to whether I was able to do it usefully without writing JavaScript glue.

[ryoqun]

Hi, first of all, thanks for creating a great npm package.

At our test environment, we succeeded to generate coverage of command invocations written in Coffee!

Here is the gist of it (sorry for being too terse, I'll tidy up this once I have some spare time):

# a require hack file
 +fs = require("fs")
 +CoffeeScript = require("coffee-script")
 +
 +CoffeeScript.run = (code, options) ->
 +  mainModule = require.main
 +  mainModule.filename = process.argv[1] = if options.filename
 +    fs.realpathSync(options.filename)
 +  else
 +    '.'
 +  mainModule.moduleCache && (mainModule.moduleCache = {});
 +  require.extensions[".coffee"](mainModule, options.filename)


# invocation (bash code)

 +  node \
 +    ./node_modules/.bin/istanbul cover --include-pid --print none -- \
 +    ./node_modules/.bin/coffee \
 +      -r coffee-script/register  \
 +      -r path/to/register_istanbul \
 +      -r path/to/hack \
 +      -- "$@"

[ryoqun]

By the way, I found a bug in CoffeeScript, so I filed an issue for that: jashkenas/coffeescript#4209