Golang | Embedding a WaitGroup within a custom HTTP client

[atc0005]

There are a number of items to study further, but these stand out:

• embedding a WaitGroup into a type which has multiple methods used for spinning off recursive goroutines
• calling Wait() within a wrapper method for the recursive goroutine generating method

refs https://www.youtube.com/watch?v=zMgpEcFN1og

[atc0005]

Worth noting: This example does not (seem to) apply an upper limit on spawned goroutines. While the client is likely perfectly happy with this arrangement, the remote server being crawled may not be fine with being hammered by by the crawler, especially if the server is not tuned for the load.

See atc0005/check-cert#103 (comment) for an approach using channels with a set queue depth to limit the maximum number of concurrent connections.