Reproducer for https://github.com/eclipse/vert.x/issues/2065

ReproducePausedStreamInPool.java

import io.vertx.core.Vertx;
import io.vertx.core.http.*;
import io.vertx.core.streams.Pump;

public class ReproducePausedStreamInPool {

    public static void main(String[] args) throws Exception {
        final Vertx vertx = Vertx.vertx();

        startBackendServer(vertx); // listen on Port 7020
        startProxy(vertx);         // listen on Port 7013, forwards to 7020

        HttpClientOptions clOpts = new HttpClientOptions().setMaxPoolSize(1).setKeepAlive(true);
        final HttpClient httpClient = vertx.createHttpClient(clOpts);

        for (int i = 1; ; i++) {
            String uri = "id-" + i;
//            String uri = "/houston/services/beeble/index.html?" + i;
            HttpClientRequest req = httpClient.get(7013, "localhost", uri);
            req.handler(resp -> {
                int status=resp.statusCode();
                if (status != 200) {
                    System.out.println("Request " + uri + ": Non-200-Status " + status);
                }
                resp.handler(data -> {
//                    System.out.println("Request #" + id + ": received " + data.length() + " bytes");
//                    resp.pause();
//                    vertx.setTimer(10, timeout -> {
//                        resp.resume();
//                    });
                });
                resp.endHandler(end -> {
                    System.out.println("Request " + uri + ": finished (response complete)");
                });
            });
            req.connectionHandler(conn ->{
                System.out.println("Connection established for " + uri);
            });
            req.end();
            System.out.println("Request " + uri + ": starting");

            // close every 10th request before http-response arrives - this provokes the "paused" stream in Houston
            if (i % 2 == 0) {
                while (true) {
                    // need to wait as the connection is only available sometime in near future...
                    final HttpConnection connection = req.connection();
                    if (connection != null) {
                        System.out.println("closed " + uri);
                        connection.close();
                        break;
                    }
                }
            }


            Thread.sleep(2000);
        }

    }

    private static void startBackendServer(Vertx vertx) {
        final HttpServer httpServer = vertx.createHttpServer();
        httpServer.requestHandler(req -> {
            System.out.println("-------------------------- Backend-Server received request " + req.uri());
            req.response().end("my data");
            System.out.println("-------------------------- Backend-Server finished response " + req.uri());
        });
        httpServer.listen(7020);
    }

    private static void startProxy(Vertx vertx) {
        final HttpServer httpServer = vertx.createHttpServer();
        final HttpClient httpClient = vertx.createHttpClient();

        httpServer.requestHandler(req -> {
            final String uri = req.uri();
            System.out.println("--------------- Proxy received request " + uri);

            final HttpServerResponse resp = req.response();
            final HttpClientRequest cReq = httpClient.request(req.method(), 7020, "localhost", uri, cResp -> {
                System.out.println("--------------- Proxy receives response " + uri);
                resp.setChunked(true);
                resp.setStatusCode(cResp.statusCode());

                // ============================================================
                // TOGGLE HERE BETWEEN PUMP (which leaves paused channels in httpClient's connection pool) AND NON-PUMP (stable)
                Pump.pump(cResp, resp).start();
//                cResp.handler(resp::write);
                // ============================================================

                cResp.endHandler(end -> {
                    resp.end();
                    System.out.println("--------------- Proxy finished response " + uri);
                });
            });
            cReq.exceptionHandler(ex -> {
                ex.printStackTrace();
                resp.setStatusCode(500);
                resp.setStatusMessage(ex.getMessage());
                resp.end();
            });
            cReq.setTimeout(1000);
            cReq.end();
        });
        httpServer.listen(7013);
    }
}

Example:

• Request "id-1" is totally fine (Client->Proxy->BackendServer and back)
• Request "id-2" is closed by the Client after request was submitted. It still goes Client->Proxy->BackendServer and back to Proxy - but does not reach client any more.
• Request "id-3" creates a new connection (as the one and only is closed) - and finally goes fine Client->Proxy->BackendServer and back
• Request "id-4" is interesting: client connection is closed again. Request still goes Client->Proxy->BackendServer and back to Proxy. But the end of the response (BackendServer->Proxy) is not seen by the Proxy (no log "Proxy finished response id-4"). This is now the point where this connection between Proxy and BackendServer is 'half paused' by the Pump
• Request "id-5" then creates a new connection Client - but the Proxy grabs the 'half paused' Backend-connection from "id-4". It still can send the Request Proxy->BackendServer and BackendServer responses to it. But the response handler in the Proxy is never invoked.

See also PCAP file of this test run: https://github.com/oli-h/gateleen/blob/demonstrate504timeout.pcap/demonstrate504timeout.pcap.pcapng
Open with WIreshark and use filter "tcp.stream eq 0" to "tcp.stream eq 4". The "tcp.stream eq 1" is the stream Proxy->BackendServer which is half paused - however you don't see this on the TCP network layer.

Request id-1: starting
Connection established for id-1
--------------- Proxy received request id-1
-------------------------- Backend-Server received request id-1
-------------------------- Backend-Server finished response id-1
--------------- Proxy receives response id-1
--------------- Proxy finished response id-1
Request id-1: finished (response complete)
Request id-2: starting
closed id-2
--------------- Proxy received request id-2
-------------------------- Backend-Server received request id-2
-------------------------- Backend-Server finished response id-2
--------------- Proxy receives response id-2
--------------- Proxy finished response id-2
Feb 14, 2018 4:25:38 PM io.vertx.core.http.impl.HttpClientRequestImpl
SCHWERWIEGEND: io.vertx.core.VertxException: Connection was closed
Request id-3: starting
Connection established for id-3
--------------- Proxy received request id-3
-------------------------- Backend-Server received request id-3
-------------------------- Backend-Server finished response id-3
--------------- Proxy receives response id-3
--------------- Proxy finished response id-3
Request id-3: finished (response complete)
Request id-4: starting
closed id-4
--------------- Proxy received request id-4
-------------------------- Backend-Server received request id-4
-------------------------- Backend-Server finished response id-4
--------------- Proxy receives response id-4
Feb 14, 2018 4:25:42 PM io.vertx.core.http.impl.HttpClientRequestImpl
SCHWERWIEGEND: io.vertx.core.VertxException: Connection was closed
Request id-5: starting
Connection established for id-5
--------------- Proxy received request id-5
-------------------------- Backend-Server received request id-5
-------------------------- Backend-Server finished response id-5
Request id-5: Non-200-Status 500
Request id-5: finished (response complete)
java.util.concurrent.TimeoutException: The timeout period of 1000ms has been exceeded while executing GET id-5 for host localhost
	at io.vertx.core.http.impl.HttpClientRequestBase.timeout(HttpClientRequestBase.java:183)
	at io.vertx.core.http.impl.HttpClientRequestBase.handleTimeout(HttpClientRequestBase.java:168)
	at io.vertx.core.http.impl.HttpClientRequestBase.lambda$setTimeout$0(HttpClientRequestBase.java:126)
	at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:870)
	at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:829)
	at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:344)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
Request id-6: starting
closed id-6
--------------- Proxy received request id-6
Feb 14, 2018 4:25:46 PM io.vertx.core.http.impl.HttpClientRequestImpl
SCHWERWIEGEND: io.vertx.core.VertxException: Connection was closed
-------------------------- Backend-Server received request id-6
-------------------------- Backend-Server finished response id-6
--------------- Proxy receives response id-6