Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/php.yml
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,6 @@ jobs:
# Add a test script to composer.json, for instance: "test": "vendor/bin/phpunit"
# Docs: https://getcomposer.org/doc/articles/scripts.md
- name: Run PHPUnit
run: ./vendor/bin/phpunit --fail-on-warning
run: composer phpunit
- name: Run Behat
run: ./vendor/bin/behat --colors --strict --stop-on-failure
run: composer behat
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,5 @@
/.phpcs-cache
/.phpunit.result.cache
/test.json
/composer.lock
/tests/fixtures/*/chunk_test.json
/composer.lock
35 changes: 27 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,15 @@ If you Properly integrate your app with this extension then it can be dramatical
* Extension can cancel your tests when you hit CTRL+C.
* When you have failed tests in **Parallel scenario** mode then can you rerun this test with Behat option `--rerun`.
* For each worker you can set environment variables.
* You can group scenarios or features into a single job to improve test execution time by using the same kernel bootstrapped once.

### Main modes

Behat Parallel Extension can work in two main modes:

* **Parallel scenario** witch can be enabled by option `--parallel` or `-l`.
* **Parallel feature** to enable this you need use behat option `--parallel-feature`.

Parallel feature option does not support's `--rerun` option.
- **Parallel chunk size** to group multiple scenarios or features into a single job, use the --parallel-chunk-size option (default is 1). Only available with Behat >= 3.23

## Requirements

Expand Down Expand Up @@ -159,25 +159,44 @@ Use `--parallel` or `-l` option for start in parallel scenario mode.

```
$ vendor/bin/behat -l 8 --colors
Starting parallel scenario tests with 8 workers
Starting parallel tests with 8 workers
Feature: Parallel
Scenario: Test behat tests with failed result
3/3 [============================] 100% 12 secs/12 secs
```

```
$ vendor/bin/behat --parallel-feature 8 --colors
Starting parallel scenario tests with 8 workers
Starting parallel tests with 8 workers
Feature: Parallel
Scenario: Test behat tests with failed result

3/3 [============================] 100% 12 secs/12 secs
```

## Test with the oldest supported PHP version
```
$ vendor/bin/behat --parallel 8 --colors --parallel-chunk-size 2
Starting parallel tests with 8 workers
Feature: Parallel, Any other feature
Scenario: Test behat tests with failed result, Any other scenario
3/3 [============================] 100% 12 secs/12 secs
```

## Test with the oldest supported PHP version and oldest vendors

With docker, without Dockerfile:

```bash
composer update --prefer-lowest --prefer-stable
docker run --rm -ti -v $(pwd):/app -w /app -u $(id -u):$(id -g) php:7.4-cli bash -c "vendor/bin/phpunit; vendor/bin/behat"
# Create the docker container
docker run --rm -d -v $(pwd):/app -w /app --name behat-parallel-old-php php:7.4-cli bash -c "tail -f /dev/null"
# Install deps
docker exec -ti behat-parallel-old-php bash -c "apt-get update && apt-get install -y libzip-dev && docker-php-ext-install zip"
# Install composer for the current user
docker exec -ti -u $(id -u):$(id -g) -e HOME=/tmp behat-parallel-old-php bash -c "curl -s https://getcomposer.org/installer | php -- --install-dir=/tmp; php /tmp/composer.phar update --prefer-lowest --prefer-dist;"
# Run tests
docker exec -ti -u $(id -u):$(id -g) behat-parallel-old-php php /tmp/composer.phar phpunit
docker exec -ti -u $(id -u):$(id -g) behat-parallel-old-php php /tmp/composer.phar behat
# Stop and remove the docker container
docker kill behat-parallel-old-php
```

## Tools and Coding standards
Expand Down
2 changes: 1 addition & 1 deletion behat.yml.dist
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@ default:
- DMarynicz\Tests\Behat\Context\SimulateTestContext
extensions:
DMarynicz\BehatParallelExtension\Extension: ~

DMarynicz\Tests\Behat\Extension\TestsSkipperExtension: ~
132 changes: 132 additions & 0 deletions features/chunk-size.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
@skip_without_multiple_paths
Feature: Parallel chunk size parameter
As a programmer, I want to able to group my tests in chunks to reduce overhead.

Scenario: Test behat tests with --parallel-chunk-size and --parallel-feature option
Given I delete file "tests/fixtures/chunked/chunk_test.json"
When I run "behat --config tests/fixtures/chunked/behat.yml.dist --parallel-feature 20 --parallel-chunk-size 2"
Then it should pass
And I should see progress bar
And the output should contain:
"""
Starting parallel tests with 20 workers and 2 tests per worker
"""
# suite01 has 2 features -> should be 1 chunk containing both
# suite02 has 1 feature -> should be 1 chunk
# Total 2 commands expected
And I should have 2 behat commands in "tests/fixtures/chunked/chunk_test.json"
Then behat commands in "tests/fixtures/chunked/chunk_test.json" should match:
"""
{BEHAT_BIN} --no-interaction --fail-on-undefined-step --config tests/fixtures/chunked/behat.yml.dist tests/fixtures/chunked/suite01/f1.feature tests/fixtures/chunked/suite01/f2.feature
{BEHAT_BIN} --no-interaction --fail-on-undefined-step --config tests/fixtures/chunked/behat.yml.dist tests/fixtures/chunked/suite02/f3.feature
"""

Scenario: Test behat tests with --parallel-chunk-size and --parallel option
Given I delete file "tests/fixtures/chunked/chunk_test.json"
When I run "behat --config tests/fixtures/chunked/behat.yml.dist --parallel 20 --parallel-chunk-size 2"
Then it should pass
And I should see progress bar
And the output should contain:
"""
Starting parallel tests with 20 workers and 2 tests per worker
"""
# suite01: f1 (1 scenario), f2 (1 scenario) -> 2 scenarios total -> 1 chunk
# suite02: f3 (2 scenarios) -> 2 scenarios -> 1 chunk
# Total 2 commands expected
And I should have 2 behat commands in "tests/fixtures/chunked/chunk_test.json"
Then behat commands in "tests/fixtures/chunked/chunk_test.json" should match:
"""
{BEHAT_BIN} --no-interaction --fail-on-undefined-step --config tests/fixtures/chunked/behat.yml.dist tests/fixtures/chunked/suite01/f1.feature:2 tests/fixtures/chunked/suite01/f2.feature:2
{BEHAT_BIN} --no-interaction --fail-on-undefined-step --config tests/fixtures/chunked/behat.yml.dist tests/fixtures/chunked/suite02/f3.feature:2 tests/fixtures/chunked/suite02/f3.feature:4
"""

Scenario: Test behat tests with --parallel-chunk-size 1 (default)
Given I delete file "tests/fixtures/chunked/chunk_test.json"
When I run "behat --config tests/fixtures/chunked/behat.yml.dist --parallel 20"
Then it should pass
And I should see progress bar
And the output should contain:
"""
Starting parallel tests with 20 workers
"""
# 3 features total, 4 scenarios total
# suite01: f1 (1 scenario), f2 (1 scenario)
# suite02: f3 (2 scenarios)
# Total 4 scenarios -> 4 commands
And I should have 4 behat commands in "tests/fixtures/chunked/chunk_test.json"
Then behat commands in "tests/fixtures/chunked/chunk_test.json" should match:
"""
{BEHAT_BIN} --no-interaction --fail-on-undefined-step --config tests/fixtures/chunked/behat.yml.dist tests/fixtures/chunked/suite01/f1.feature:2
{BEHAT_BIN} --no-interaction --fail-on-undefined-step --config tests/fixtures/chunked/behat.yml.dist tests/fixtures/chunked/suite01/f2.feature:2
{BEHAT_BIN} --no-interaction --fail-on-undefined-step --config tests/fixtures/chunked/behat.yml.dist tests/fixtures/chunked/suite02/f3.feature:2
{BEHAT_BIN} --no-interaction --fail-on-undefined-step --config tests/fixtures/chunked/behat.yml.dist tests/fixtures/chunked/suite02/f3.feature:4
"""

Scenario: Test behat tests with --rerun and --parallel-chunk-size 2
Given I delete file "tests/fixtures/chunked/chunk_test.json"
# Rerun should still execute chunked tasks when no previous failures exist.
When I run "behat --config tests/fixtures/chunked/behat.yml.dist --parallel 2 --parallel-chunk-size 2 --rerun"
Then it should pass
And I should see progress bar
And the output should contain:
"""
Starting parallel tests with 2 workers and 2 tests per worker
"""
And I should have 2 behat commands in "tests/fixtures/chunked/chunk_test.json"
Then behat commands in "tests/fixtures/chunked/chunk_test.json" should match:
"""
{BEHAT_BIN} --no-interaction --fail-on-undefined-step --rerun --config tests/fixtures/chunked/behat.yml.dist tests/fixtures/chunked/suite01/f1.feature:2 tests/fixtures/chunked/suite01/f2.feature:2
{BEHAT_BIN} --no-interaction --fail-on-undefined-step --rerun --config tests/fixtures/chunked/behat.yml.dist tests/fixtures/chunked/suite02/f3.feature:2 tests/fixtures/chunked/suite02/f3.feature:4
"""

Scenario: Test behat tests with --rerun and one failing job with --parallel option
Given I delete file "tests/fixtures/chunked_rerun/chunk_test.json"
# First run executes at least two chunks from suite01.
And I run "behat --config tests/fixtures/chunked_rerun/behat.yml.dist --parallel 2 --parallel-chunk-size 2"
Then it should fail
And I should have 2 behat commands in "tests/fixtures/chunked_rerun/chunk_test.json"
And behat commands in "tests/fixtures/chunked_rerun/chunk_test.json" should match:
"""
{BEHAT_BIN} --no-interaction --fail-on-undefined-step --config tests/fixtures/chunked_rerun/behat.yml.dist tests/fixtures/chunked_rerun/suite01/f1.feature:2 tests/fixtures/chunked_rerun/suite01/f2.feature:2
{BEHAT_BIN} --no-interaction --fail-on-undefined-step --config tests/fixtures/chunked_rerun/behat.yml.dist tests/fixtures/chunked_rerun/suite01/f3.feature:2
"""
And I delete file "tests/fixtures/chunked_rerun/chunk_test.json"
# Rerun only the previously failed command from suite01.
When I run "behat --config tests/fixtures/chunked_rerun/behat.yml.dist --parallel 2 --parallel-chunk-size 2 --rerun"
Then it should fail
And I should see progress bar
And the output should contain:
"""
Starting parallel tests with 2 workers and 2 tests per worker
"""
And I should have 1 behat commands in "tests/fixtures/chunked_rerun/chunk_test.json"
And behat commands in "tests/fixtures/chunked_rerun/chunk_test.json" should match:
"""
{BEHAT_BIN} --no-interaction --fail-on-undefined-step --rerun --config tests/fixtures/chunked_rerun/behat.yml.dist tests/fixtures/chunked_rerun/suite01/f1.feature:2 tests/fixtures/chunked_rerun/suite01/f2.feature:2
"""

Scenario: Test behat tests with --rerun and one failing job with --parallel-feature option
Given I delete file "tests/fixtures/chunked_rerun/chunk_test.json"
# First run executes at least two chunks from suite01.
And I run "behat --config tests/fixtures/chunked_rerun/behat.yml.dist --parallel-feature 2 --parallel-chunk-size 2"
Then it should fail
And I should have 2 behat commands in "tests/fixtures/chunked_rerun/chunk_test.json"
And behat commands in "tests/fixtures/chunked_rerun/chunk_test.json" should match:
"""
{BEHAT_BIN} --no-interaction --fail-on-undefined-step --config tests/fixtures/chunked_rerun/behat.yml.dist tests/fixtures/chunked_rerun/suite01/f1.feature tests/fixtures/chunked_rerun/suite01/f2.feature
{BEHAT_BIN} --no-interaction --fail-on-undefined-step --config tests/fixtures/chunked_rerun/behat.yml.dist tests/fixtures/chunked_rerun/suite01/f3.feature
"""
And I delete file "tests/fixtures/chunked_rerun/chunk_test.json"
# Rerun only the previously failed command from suite01.
When I run "behat --config tests/fixtures/chunked_rerun/behat.yml.dist --parallel-feature 2 --parallel-chunk-size 2 --rerun"
Then it should fail
And I should see progress bar
And the output should contain:
"""
Starting parallel tests with 2 workers and 2 tests per worker
"""
And I should have 1 behat commands in "tests/fixtures/chunked_rerun/chunk_test.json"
And behat commands in "tests/fixtures/chunked_rerun/chunk_test.json" should match:
"""
{BEHAT_BIN} --no-interaction --fail-on-undefined-step --rerun --config tests/fixtures/chunked_rerun/behat.yml.dist tests/fixtures/chunked_rerun/suite01/f1.feature tests/fixtures/chunked_rerun/suite01/f2.feature
"""
2 changes: 1 addition & 1 deletion features/environment.feature
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Feature: Environment
Then it should pass
And the output should contain:
"""
Starting parallel scenario tests with 8 workers
Starting parallel tests with 8 workers
"""
And the output should contain:
"""
Expand Down
2 changes: 1 addition & 1 deletion features/parallel-feature-environment.feature
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Feature: Environment
Then it should pass
And the output should contain:
"""
Starting parallel scenario tests with 8 workers
Starting parallel tests with 8 workers
"""
And the output should contain:
"""
Expand Down
2 changes: 1 addition & 1 deletion features/sigint.feature
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,4 @@ Feature: Sigint
Given I start "behat --config tests/fixtures/fail/behat.yml.dist --parallel 2"
And I wait for 3 seconds
And I send a SIGINT signal to behat process
Then it should fail
Then it should fail
39 changes: 25 additions & 14 deletions src/Cli/ParallelController.php
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@

namespace DMarynicz\BehatParallelExtension\Cli;

use Behat\Gherkin\Node\ScenarioLikeInterface;
use Behat\Testwork\Cli\Controller;
use Behat\Testwork\Tester\Cli\ExerciseController;
use DMarynicz\BehatParallelExtension\Event\AfterTaskTested;
Expand Down Expand Up @@ -52,6 +51,9 @@ abstract class ParallelController
/** @var CanDetermineNumberOfProcessingUnits */
protected $numberOfCores;

/** @var TitleListFormatter */
protected $titleListFormatter;

public function __construct(
Controller $decoratedController,
TaskFactory $taskFactory,
Expand Down Expand Up @@ -86,20 +88,20 @@ public function execute(InputInterface $input, OutputInterface $output)
$this->output = $output;
$this->input = $input;

// Unable to get the COLUMNS environment variable to know the width of the terminal
// because Behat overrides it to 9999. Let's use the default TitleListFormatter's maxLength value.
$this->titleListFormatter = new TitleListFormatter();

return $this->parallelExecute();
}

public function beforeTaskTested(BeforeTaskTested $beforeTaskTested): void
{
$task = $beforeTaskTested->getTask();
$featureTitle = sprintf('<info>Feature: %s</info>', $task->getFeature()->getTitle());
$scenarioTitle = '';
if ($task->getScenario() instanceof ScenarioLikeInterface) {
$scenarioTitle = sprintf('<info>Scenario: %s</info>', $task->getScenario()->getTitle());
}
$task = $beforeTaskTested->getTask();
$units = $task->getUnits();

$this->progressBar->setMessage($featureTitle, 'feature');
$this->progressBar->setMessage($scenarioTitle, 'scenario');
$this->progressBar->setMessage($this->titleListFormatter->formatFeatures($units), 'feature');
$this->progressBar->setMessage($this->titleListFormatter->formatScenarios($units), 'scenario');
}

public function afterTaskTested(AfterTaskTested $taskTested): void
Expand Down Expand Up @@ -168,8 +170,17 @@ private function setupTasksWithProgressBar(): void

private function startPoll(): void
{
$poolSize = $this->getMaxPoolSize();
$this->output->writeln(sprintf('Starting parallel scenario tests with %d workers', $poolSize));
$poolSize = $this->getMaxPoolSize();
$chunkSize = (int) $this->input->getOption('parallel-chunk-size');
$message = sprintf(
'Starting parallel tests with %d workers',
$poolSize
);
if ($chunkSize > 1) {
$message .= sprintf(' and %d tests per worker', $chunkSize);
}

$this->output->writeln($message);
$this->poll->setMaxWorkers($poolSize);
$this->poll->start();
$maxSize = $this->getMaxSizeFromParallelOption();
Expand Down Expand Up @@ -204,12 +215,12 @@ private function createTasks()
return $this->taskFactory->createTasks($this->input, null);
}

$tasks = [];
$tasksPerPath = [];
foreach ($paths as $path) {
$tasks = array_merge($tasks, $this->taskFactory->createTasks($this->input, $path));
$tasksPerPath[] = $this->taskFactory->createTasks($this->input, $path);
}

return $tasks;
return array_merge(...$tasksPerPath);
}

/**
Expand Down
7 changes: 7 additions & 0 deletions src/Cli/ParallelFeatureController.php
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,13 @@ public function configure(SymfonyCommand $command): void
'How many scenario jobs run in parallel feature mode? Available values empty or integer',
false
);
$command->addOption(
'parallel-chunk-size',
null,
InputOption::VALUE_OPTIONAL,
'How many features run in one chunk? (Behat >= 3.23)',
1
);
}

/**
Expand Down
7 changes: 7 additions & 0 deletions src/Cli/ParallelScenarioController.php
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,13 @@ public function configure(SymfonyCommand $command): void
)
->addUsage('--parallel 8')
->addUsage('--parallel');
$command->addOption(
'parallel-chunk-size',
null,
InputOption::VALUE_OPTIONAL,
'How many scenarios run in one chunk? (Behat >= 3.23)',
1
);
}

/**
Expand Down
23 changes: 10 additions & 13 deletions src/Cli/RerunController.php
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
namespace DMarynicz\BehatParallelExtension\Cli;

use Behat\Behat\Tester\Cli\RerunController as BehatRerunController;
use Behat\Gherkin\Node\ScenarioLikeInterface;
use Behat\Testwork\Cli\Controller;
use DMarynicz\BehatParallelExtension\Event\AfterTaskTested;
use DMarynicz\BehatParallelExtension\Event\EventDispatcherDecorator;
Expand Down Expand Up @@ -59,20 +58,18 @@ public function collectFailedTask(AfterTaskTested $taskTested): void
return;
}

$suiteName = $taskTested->getTask()->getSuite()->getName();
$featureFile = $taskTested->getTask()->getFeature()->getFile();
$scenarioLine = null;
$scenario = $taskTested->getTask()->getScenario();
if ($scenario instanceof ScenarioLikeInterface) {
$scenarioLine = $scenario->getLine();
}
$task = $taskTested->getTask();
$suiteName = $task->getSuite()->getName();
$units = $task->getUnits();

$line = $featureFile;
if ($scenarioLine !== null) {
$line .= ':' . $scenarioLine;
}
foreach ($units as $unit) {
$path = $unit->getPath();
if (in_array($path, $this->lines[$suiteName] ?? [], true)) {
continue;
}

$this->lines[$suiteName][] = $line;
$this->lines[$suiteName][] = $path;
}
}

/**
Expand Down
Loading