Skip to content

Commit 767ac61

Browse files
authored
Add support for bulk operations (#42)
* Add support for bulk operations * Fix mktemp * Fixup tests * Fix dirty working dir check * Add missing file (for dirty working dir check) * Fix dirty working dir check * Update readme with bulk docs
1 parent d8216fa commit 767ac61

38 files changed

Lines changed: 6848 additions & 299 deletions

.circleci/config.yml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,19 @@ jobs:
2525

2626
command: |
2727
set -ux
28+
find . -name moq_\*.go -delete
2829
go build -o $GOPATH/bin/moqueries github.com/myshkin5/moqueries
30+
export MOQ_BULK_STATE_FILE=$(mktemp --tmpdir= moq-XXXXXX)
31+
moqueries bulk-initialize
2932
go generate ./...
33+
MOQ_DEBUG=true moqueries bulk-finalize
34+
35+
# fail if working directory is dirty
36+
git status --short
37+
if [[ -n $(git status --short) ]]; then
38+
echo "Working directory dirty"
39+
exit 1
40+
fi
3041
3142
- run:
3243
name: Run tests

.gitignore

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,4 +26,3 @@ moq_*_dst.txt
2626
# other
2727
.envrc
2828
/.idea
29-
/.go

.golangci.yaml

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -17,18 +17,22 @@ linters:
1717
- maintidx
1818
- maligned
1919
- nlreturn
20+
- nosnakecase
2021
- paralleltest
2122
- prealloc
2223
- predeclared
23-
- varnamelen
2424
- scopelint
25+
- tagliatelle
26+
- varnamelen
2527
- wsl
2628

2729
# Consider enabling
2830
- gocyclo
29-
- golint
3031
- wrapcheck
3132

33+
# Usually disabled but useful for checking everything has godoc
34+
- golint
35+
3236
linters-settings:
3337
gci:
3438
sections:
@@ -47,6 +51,7 @@ linters-settings:
4751
- "-ID"
4852

4953
issues:
54+
exclude-use-default: false
5055
exclude-rules:
5156
- path: '(.+)_test.go'
5257
linters:
@@ -56,9 +61,6 @@ issues:
5661
- linters:
5762
- lll
5863
source: "^//go:generate "
59-
# Inline nolint directive isn't working (CI/CD fails on nolintlint)
60-
- linters:
61-
- thelper
62-
# This is the actual test so not a helper
63-
path: 'generator/testmoqs/testmoqs_test.go'
64-
source: "testOptionalSuccess := func"
64+
include:
65+
# disable excluding of issues about comments from golint.
66+
- EXC0002

README.md

Lines changed: 38 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -249,6 +249,22 @@ Quite often tests will require several mocks. A `Scene` is a collection of mocks
249249
scene.AssertExpectationsMet()
250250
```
251251

252+
## Bulk generation
253+
Generating mock can be CPU intensive! Additionally, Moqueries only knows the package where to look for a type so the entire package has to be parsed. And to top it off, you will quite often mock several types from the same package. To avoid re-parsing the same package repeatedly, Moqueries has a bulk mode that can best be described by these three steps:
254+
1. Initialize the bulk processing file
255+
2. `go generate ./...`
256+
3. Finalize the bulk processing (and generate all the mocks)
257+
258+
Moqueries [CI/CD pipeline](.circleci/config.yml) accomplishes this with the following few commands:
259+
```shell
260+
export MOQ_BULK_STATE_FILE=$(mktemp --tmpdir= moq-XXXXXX)
261+
moqueries bulk-initialize
262+
go generate ./...
263+
moqueries bulk-finalize
264+
```
265+
266+
The first line creates a new temporary file to hold the state. The second line initializes the file (holds on to some global attributes to ensure consistency). The third line is the standard `go generate` line but because `MOQ_BULK_STATE_FILE` is defined, it only records the intent to generate a new mock. The forth and final line is where the work actually occurs, and it might take some time depending on how many mocks you want to generate. See more details below in the [Command line reference](#command-line-reference).
267+
252268
## More command line options
253269
Below is a loose collection of out-of-the-ordinary command line options for use in out-of-the-ordinary situations.
254270

@@ -285,8 +301,8 @@ writerMoq.OnCall().Write([]byte("3")).ReturnResults(0, fmt.Errorf("couldn't writ
285301
## Command line reference
286302
The Moqueries command line has the following form:
287303

288-
```bash
289-
$ moqueries [options] [interfaces and/or function types to mock] [options]
304+
```shell
305+
moqueries [options] [interfaces and/or function types to mock] [options]
290306
```
291307

292308
Interfaces and function types are separated by whitespace. Multiple types may be specified.
@@ -297,6 +313,7 @@ Interfaces and function types are separated by whitespace. Multiple types may be
297313
| `--destination <file>` | `string` | `./moq_<type>.go` when exported or `./moq_<type>_test.go` when not exported | The file path where mocks are generated relative to directory containing generate directive (or relative to the current directory) |
298314
| `--destination-dir <dir>` | `string` | `.` | The file directory where mocks are generated relative to the directory containing the generate directive (or relative to the current directory) |
299315
| `--export` | `bool` | `false` | If true, generated mocks will be exported and accessible from other packages |
316+
| `-h` or `--help` | `bool` | `false` | Display command help |
300317
| `--import <name>` | `string` | `.` (the directory containing generate directive) | The package containing the type (interface or function type) to be mocked |
301318
| `--package <name>` | `string` | The test package of the destination directory when `--export=false` or the package of the destination directory when `--export=true` | The package to generate code into |
302319
| `--test-import` | `bool` | `false` | Indicates that the types are defined in the test package |
@@ -310,19 +327,34 @@ Options with a value type of `bool` are set (turned on) by specifying the option
310327
### Environment Variables
311328
The Moqueries command line can also be controlled by the following environment variables:
312329

313-
| Name | Usage |
314-
|-------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
315-
| `MOQ_DEBUG` | If set to a "true" value (see [`strconv.ParseBool`](https://pkg.go.dev/strconv#ParseBool)), debugging output will be logged (also see `--debug` in [Command line reference](#command-line-reference) above) |
330+
| Name | Usage |
331+
|-----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
332+
| `MOQ_BULK_STATE_FILE` | If set, defines the bulk state file in which generate requests will be stored for bulk generation. See [Bulk generation](#bulk-generation) above. |
333+
| `MOQ_DEBUG` | If set to a "true" value (see [`strconv.ParseBool`](https://pkg.go.dev/strconv#ParseBool)), debugging output will be logged (also see `--debug` in [Command line reference](#command-line-reference) above) |
316334

317335
### Subcommands
318336

319337
#### Default
320338
The default subcommand generates one or more mocks based on the command specified. As described [above](#generating-mocks), this is typically invoked by a `go:generate` directive. The default subcommand is invoked when no subcommand is specified.
321339

340+
If the `MOQ_BULK_STATE_FILE` environment variable is defined (see [above](#environment-variables)), the default subcommand does not immediately generate the mocks, but instead appends the generate request to the state file. See [Bulk generation](#bulk-generation) above.
341+
342+
#### Bulk initialize
343+
Initializes the bulk state file defined by the `MOQ_BULK_STATE_FILE` environment variable. `MOQ_BULK_STATE_FILE` is required. Note that the bulk state file is overwritten if it exists.
344+
```shell
345+
moqueries bulk-initialize
346+
```
347+
348+
#### Bulk finalize
349+
Finalizes bulk processing by generating multiple mocks at once. The `MOQ_BULK_STATE_FILE` environment variable is required and specifies which mocks to generate.
350+
```shell
351+
moqueries bulk-finalize
352+
```
353+
322354
#### Summarize metrics
323355
The `summarize-metrics` subcommand takes the debug logs from multiple generate runs (using the [default](#default) subcommand), reads metrics from each individual run, and outputs summary metrics. This subcommand takes a single, optional argument specifying the log file to read. If no file is specified or if the file is specified as `-', standard in is read.
324356
325357
The following command generates all mocks specified in `go:generate` directives and summarizes the metrics for all runs:
326358
```shell
327-
$ MOQ_DEBUG=true go generate ./... | moqueries summarize-metrics
359+
MOQ_DEBUG=true go generate ./... | moqueries summarize-metrics
328360
```

ast/cache.go

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
// Package ast provides utilities for working with Go's Abstract Syntax Tree
12
package ast
23

34
import (
@@ -82,7 +83,7 @@ func (c *Cache) Type(id dst.Ident, testImport bool) (*dst.TypeSpec, string, erro
8283
typ, ok := c.typesByIdent[realId]
8384
if !ok {
8485
return nil, "", fmt.Errorf(
85-
"%q (original package %q): %w", realId, id.Path, ErrTypeNotFound)
86+
"%w: %q (original package %q)", ErrTypeNotFound, realId, id.Path)
8687
}
8788

8889
return typ, pkgPath, nil

ast/testpkg/testpkg.go

Lines changed: 0 additions & 1 deletion
This file was deleted.

bulk/bulk.go

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
// Package bulk is used to generate several Moqueries mocks at once
2+
package bulk
3+
4+
import (
5+
"io"
6+
"os"
7+
8+
"github.com/myshkin5/moqueries/bulk/internal"
9+
"github.com/myshkin5/moqueries/generator"
10+
)
11+
12+
// Initialize initializes bulk processing and creates the bulk processing state
13+
// file
14+
func Initialize(stateFile, rootDir string) error {
15+
createFn := func(name string) (io.WriteCloser, error) {
16+
//nolint:gosec // Users can use any file for bulk operations
17+
return os.Create(name)
18+
}
19+
20+
return internal.Initialize(stateFile, rootDir, createFn)
21+
}
22+
23+
// Append appends a mock generate request to the bulk state
24+
func Append(stateFile string, request generator.GenerateRequest) error {
25+
openFileFn := func(name string, flag int, perm os.FileMode) (internal.ReadWriteSeekCloser, error) {
26+
//nolint:gosec // Users can use any file for bulk operations
27+
return os.OpenFile(name, flag, perm)
28+
}
29+
30+
return internal.Append(stateFile, request, openFileFn)
31+
}
32+
33+
// Finalize complete bulk processing by generating all the requested mocks
34+
func Finalize(stateFile, rootDir string) error {
35+
openFn := func(name string) (io.ReadCloser, error) {
36+
//nolint:gosec // Users can use any file for bulk operations
37+
return os.Open(name)
38+
}
39+
40+
return internal.Finalize(stateFile, rootDir, openFn, generator.Generate)
41+
}

bulk/internal/appender.go

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
package internal
2+
3+
import (
4+
"bufio"
5+
"encoding/json"
6+
"errors"
7+
"fmt"
8+
"go/build"
9+
"io"
10+
"os"
11+
"path"
12+
"path/filepath"
13+
"strings"
14+
15+
"github.com/myshkin5/moqueries/generator"
16+
"github.com/myshkin5/moqueries/logs"
17+
)
18+
19+
var (
20+
// ErrBadAppendRequest is returned when a caller passes bad parameters to
21+
// Append
22+
ErrBadAppendRequest = errors.New("bad request")
23+
// ErrBulkState is returned when the bulk state is invalid
24+
ErrBulkState = errors.New("bulk state error")
25+
)
26+
27+
//go:generate moqueries OpenFileFn
28+
29+
// OpenFileFn is the function type of os.OpenFile
30+
type OpenFileFn func(name string, flag int, perm os.FileMode) (ReadWriteSeekCloser, error)
31+
32+
//go:generate moqueries ReadWriteSeekCloser
33+
34+
// ReadWriteSeekCloser is the interface that groups the basic Read, Write,
35+
// Seek and Close methods.
36+
type ReadWriteSeekCloser interface {
37+
io.Reader
38+
io.Writer
39+
io.Seeker
40+
io.Closer
41+
}
42+
43+
// Append appends a mock generate request to the bulk state
44+
func Append(stateFile string, req generator.GenerateRequest, openFileFn OpenFileFn) error {
45+
if !path.IsAbs(req.WorkingDir) {
46+
return fmt.Errorf("%w: the request working directory must be absolute: %s",
47+
ErrBadAppendRequest, req.WorkingDir)
48+
}
49+
50+
f, err := openFileFn(stateFile, os.O_RDWR|os.O_APPEND, 0)
51+
if err != nil {
52+
return fmt.Errorf("error opening state file: %w", err)
53+
}
54+
defer func() {
55+
err := f.Close()
56+
if err != nil {
57+
logs.Error("error closing state file", err)
58+
}
59+
}()
60+
61+
_, err = verifyState(f, stateFile, req.WorkingDir, false)
62+
if err != nil {
63+
return err
64+
}
65+
66+
err = appendRequest(f, stateFile, req)
67+
if err != nil {
68+
return err
69+
}
70+
71+
return nil
72+
}
73+
74+
func verifyState(f io.ReadCloser, stateFile, workingDir string, rootDirOnly bool) (*bufio.Scanner, error) {
75+
scanner := bufio.NewScanner(f)
76+
if !scanner.Scan() {
77+
return nil, fmt.Errorf("%w: state file %s not initialized properly",
78+
ErrBulkState, stateFile)
79+
}
80+
81+
txt := scanner.Text()
82+
err := scanner.Err()
83+
if err != nil {
84+
return nil, fmt.Errorf("error reading state file %s: %w", stateFile, err)
85+
}
86+
87+
var state initialState
88+
err = json.Unmarshal([]byte(txt), &state)
89+
if err != nil {
90+
return nil, fmt.Errorf("error unmarshalling state file %s: %w", stateFile, err)
91+
}
92+
93+
if state.GoPath != build.Default.GOPATH {
94+
return nil, fmt.Errorf("%w: current GOPATH doesn't match GOPATH from state file %s (%s != %s)",
95+
ErrBulkState, stateFile, build.Default.GOPATH, state.GoPath)
96+
}
97+
98+
if rootDirOnly {
99+
if state.RootDir != workingDir {
100+
return nil, fmt.Errorf("%w: finalize root directory %s does"+
101+
" not match root directory %s from state file %s",
102+
ErrBulkState, workingDir, state.RootDir, stateFile)
103+
}
104+
} else {
105+
rel, err := filepath.Rel(state.RootDir, workingDir)
106+
if err != nil {
107+
logs.Panicf("error getting relative path %s from %s: %#v",
108+
state.RootDir, workingDir, err)
109+
}
110+
111+
if strings.HasPrefix(rel, "..") {
112+
return nil, fmt.Errorf("%w: working directory %s is not a"+
113+
" child of root directory %s from state file %s",
114+
ErrBulkState, workingDir, state.RootDir, stateFile)
115+
}
116+
}
117+
118+
return scanner, nil
119+
}
120+
121+
func appendRequest(f ReadWriteSeekCloser, stateFile string, req generator.GenerateRequest) error {
122+
_, err := f.Seek(0, io.SeekEnd)
123+
if err != nil {
124+
return fmt.Errorf("error seeking end of state file %s: %w", stateFile, err)
125+
}
126+
_, err = f.Write(compact(req))
127+
if err != nil {
128+
return fmt.Errorf("error writing state file %s: %w", stateFile, err)
129+
}
130+
_, err = f.Write([]byte("\n"))
131+
if err != nil {
132+
return fmt.Errorf("error finishing writing of state file %s: %w", stateFile, err)
133+
}
134+
return nil
135+
}

0 commit comments

Comments
 (0)