Flakey tests - Validate healthy server startup before sending requests

[anniefu]

Problem

The current validation logic starts up the functions framework servers, sleeps for a preconfigured time, and then attempts to send a request to the server.

1. Server start:

   functions-framework-conformance/client/validate.go

   Line 76 in bb58df6

   shutdown, err := v.funcServer.Start()

2. Sleep (buildpack-based):

   functions-framework-conformance/client/buildpacks.go

   Line 135 in 80ce4da

   time.Sleep(time.Duration(*startDelay) * time.Second)

   
   Sleep (local server):

   functions-framework-conformance/client/local.go

   Line 53 in bb58df6

   time.Sleep(time.Duration(*startDelay) * time.Second)

3. Request is sent:

   functions-framework-conformance/client/validate.go

   Line 85 in bb58df6

   if err := v.validate("http://localhost:8080"); err != nil {

The sleep time is configured by the client's startDelay flag and is configured per conformance test. If it's not configured long enough, it's possible that the server is not started up by the time the HTTP request is sent and the test fails or becomes flakey.

startDelay also forces a minimum wait time per test and might be adding unnecessary wait time.

Note that "server start" here is whatever command is passed to the conformance test client's --cmd flag and could be doing more work than just starting up the functions framework server, making it harder to choose the correct startDelay.

Suggestion

It would be better if the runValidation function checked for "health status" by seeing if there is something serving on localhost:8080 before sending the validating HTTP request instead of having each test guess the minimum required startDelay. That would eliminate the need to "guess" the correct start delay needed across different tests and repos.