Refactor provider shutdown and pending flows

[artemredkin]

Fixes some edge cases in shutdown and pending states.

Motivation:
There are two issues this PR addresses:

1. Flaky test, where we are trying to close the client when connection was able to enqueue and get an unexpected error
2. The fact that provider is created with +1 pending state, while it might not be an error, it makes testing difficult, also might lead to races in shutdown.

Modifications:

1. HTTP1ConnectionsProvider is no longer created with +1 pending
2. If we receive close while request is enqueued, it should be failed with a more expected .alreadyShutdown error, since its the only way to get into that state, so test was actually catching the right thing.

Result:
Closes #226
Closes #323

[weissi]

This looks like a great start. I think we can delete some code though.

[weissi]

yippie! Could we assert here that the return value is the expected one?

[artemredkin]

great idea, done!

[weissi]

nice! I left a bunch of stuff like the following comments in the code

        /* BEGIN workaround for #232, this whole block is to be replaced by the commented out line below */
        // not closing the provider here (https://github.com/swift-server/async-http-client/issues/232)
        var state = self.http1ConnectionProvider.state.testsOnly_getInternalState()
        if state.pending == 1, state.waiters.isEmpty, state.leasedConnections.isEmpty, state.openedConnectionsCount == 0 {
            state.pending = 0
            self.http1ConnectionProvider.state.testsOnly_setInternalState(state)
        }
        self.http1ConnectionProvider.closePromise.succeed(())
        /* END workaround for #232 */

and

            // cleanup
            // this cleanup code needs to go and use HTTP1ConnectionProvider's API instead

I believe we can also delete all this code then.

[artemredkin]

I was able to delete the workaround, some clean up code, and I was able to replace some cleanup with API calls instead of direct state manipulation. Currently there are 3-4 tests that manipulate state directly (they all need the state to be near the limit), do you want me to fix it here or when I get to #232?

[weissi]

no, let's fix them in a followup. It's super awesome you could remove the workaround

[weissi]

This looks great! Thank you so much

[weissi]

@artemredkin from what I can tell, no test noticed that this error changed which probably means no test is getting this error ;). I think we should probably add a test case that tests this condition.

[weissi]

filed #244

[artemredkin]

Specific test is a good idea, yes, though this error was picked up by a test, just not reliably :)