Very Evasive Proxy Error from Nginx and Apache when uploading files that take more than 1 second to upload. However, the problem was not with Proxy Servers, nor the backend application server. The problem came from the router.
TL;DR
Proxy Error from Nginx, and Apache on large uploads
Large file upload breaks;
Large file uploads disconnect;
File Upload fails if upload >
1000ms or 1 second
No true fix, except for switching routers.
All configurations related to searches on these errors did not fix the problem.
Errors
Sample Nginx Errors
2018/04/29 09:41:36 [debug] 5455#5455: *947 http client request body recv -2 2018/04/29 09:41:36 [debug] 5455#5455: *947 http client request body rest 74726165 2018/04/29 09:41:36 [debug] 5455#5455: *947 event timer add: 5: 60000:67967884 2018/04/29 09:41:36 [debug] 5455#5455: *947 event timer del: 20: 77987884 2018/04/29 09:41:36 [debug] 5455#5455: *947 http run request: "/share/proxy/alfresco/api/upload?" 2018/04/29 09:41:36 [debug] 5455#5455: *947 http upstream read request handler 2018/04/29 09:41:36 [debug] 5455#5455: *947 http upstream send request 2018/04/29 09:41:36 [debug] 5455#5455: *947 http upstream send request body 2018/04/29 09:41:36 [debug] 5455#5455: *947 http read client request body 2018/04/29 09:41:36 [debug] 5455#5455: *947 SSL_read: -1 2018/04/29 09:41:36 [debug] 5455#5455: *947 SSL_get_error: 5 2018/04/29 09:41:36 [debug] 5455#5455: *947 peer shutdown SSL cleanly 2018/04/29 09:41:36 [debug] 5455#5455: *947 http client request body recv 0 2018/04/29 09:41:36 [info] 5455#5455: *947 client prematurely closed connection while sending request to upstream, client: 192.168.1.1, server: content.mahabir.org, request: "POST /share/proxy/alfresco/api/upload HTTP/1.1", upstream: "http://[::1]:8080/share/proxy/alfresco/api/upload", host: "192.168.1.123", referrer: "https://192.168.1.123/share/page/context/mine/myfiles" 2018/04/29 09:41:36 [debug] 5455#5455: *947 finalize http upstream request: 400 2018/04/29 09:41:36 [debug] 5455#5455: *947 finalize http proxy request 2018/04/29 09:41:36 [debug] 5455#5455: *947 free rr peer 2 0 2018/04/29 09:41:36 [debug] 5455#5455: *947 close http upstream connection: 20 2018/04/29 09:41:36 [debug] 5455#5455: *947 free: 000055CC3CD48100, unused: 48 2018/04/29 09:41:36 [debug] 5455#5455: *947 reusable connection: 0 2018/04/29 09:41:36 [debug] 5455#5455: *947 http finalize request: 400, "/share/proxy/alfresco/api/upload?" a:1, c:1 2018/04/29 09:41:36 [debug] 5455#5455: *947 http terminate request count:1 2018/04/29 09:41:36 [debug] 5455#5455: *947 http terminate cleanup count:1 blk:0 2018/04/29 09:41:36 [debug] 5455#5455: *947 http posted request: "/share/proxy/alfresco/api/upload?" 2018/04/29 09:41:36 [debug] 5455#5455: *947 http terminate handler count:1 2018/04/29 09:41:36 [debug] 5455#5455: *947 http request count:1 blk:0 2018/04/29 09:41:36 [debug] 5455#5455: *947 http close request 2018/04/29 09:41:36 [debug] 5455#5455: *947 http log handler
Sample Apache Errors
[Sun Apr 29 10:32:50.490278 2018] [mpm_event:notice] [pid 11071:tid 140462232258432] AH00489: Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g configured -- resuming normal operations [Sun Apr 29 10:32:50.490328 2018] [core:notice] [pid 11071:tid 140462232258432] AH00094: Command line: '/usr/sbin/apache2' [Sun Apr 29 10:33:06.914022 2018] [proxy_ajp:error] [pid 11074:tid 140461880575744] (104)Connection reset by peer: [client 192.168.1.1:58335] AH02822: dialog with client 192.168.1.1:58335 failed, ref ...skipping... [Sun Apr 29 10:44:19.586534 2018] [proxy_balancer:debug] [pid 11342:tid 139630102398720] mod_proxy_balancer.c(616): [client 192.168.1.1:60708] AH01172: balancer://acs: worker (ajp://localhost) rewritten to ajp://localhost/share/proxy/alfresco/api/upload, referer: https://192.168.1.123/share/page/context/mine/myfiles [Sun Apr 29 10:44:19.586544 2018] [proxy:debug] [pid 11342:tid 139630102398720] proxy_util.c(1785): AH00924: worker ajp://localhost shared already initialized [Sun Apr 29 10:44:19.586550 2018] [proxy:debug] [pid 11342:tid 139630102398720] proxy_util.c(1832): AH00927: initializing worker ajp://localhost local [Sun Apr 29 10:44:19.586572 2018] [proxy:debug] [pid 11342:tid 139630102398720] proxy_util.c(1867): AH00930: initialized pool in child 11342 for (localhost) min=0 max=25 smax=25 [Sun Apr 29 10:44:19.586581 2018] [proxy:debug] [pid 11342:tid 139630102398720] mod_proxy.c(1160): [client 192.168.1.1:60708] AH01143: Running scheme balancer handler (attempt 0), referer: https://192.168.1.123/share/page/context/mine/myfiles [Sun Apr 29 10:44:19.586590 2018] [proxy_ajp:debug] [pid 11342:tid 139630102398720] mod_proxy_ajp.c(743): [client 192.168.1.1:60708] AH00895: serving URL ajp://localhost/share/proxy/alfresco/api/upload, referer: https://192.168.1.123/share/page/context/mine/myfiles [Sun Apr 29 10:44:19.586632 2018] [proxy:debug] [pid 11342:tid 139630102398720] proxy_util.c(2160): AH00942: AJP: has acquired connection for (localhost) [Sun Apr 29 10:44:19.586643 2018] [proxy:debug] [pid 11342:tid 139630102398720] proxy_util.c(2213): [client 192.168.1.1:60708] AH00944: connecting ajp://localhost/share/proxy/alfresco/api/upload to localhost:8009, referer: https://192.168.1.123/share/page/context/mine/myfiles [Sun Apr 29 10:44:19.586942 2018] [proxy:debug] [pid 11342:tid 139630102398720] proxy_util.c(2422): [client 192.168.1.1:60708] AH00947: connected /share/proxy/alfresco/api/upload to localhost:8009, referer: https://192.168.1.123/share/page/context/mine/myfiles [Sun Apr 29 10:44:19.587032 2018] [proxy:debug] [pid 11342:tid 139630102398720] proxy_util.c(2799): AH02824: AJP: connection established with 127.0.0.1:8009 (localhost) [Sun Apr 29 10:44:19.589149 2018] [proxy_ajp:debug] [pid 11342:tid 139630102398720] mod_proxy_ajp.c(275): [client 192.168.1.1:60708] AH00873: data to read (max 8186 at 4), referer: https://192.168.1.123/share/page/context/mine/myfiles [Sun Apr 29 10:44:19.589187 2018] [proxy_ajp:debug] [pid 11342:tid 139630102398720] mod_proxy_ajp.c(289): [client 192.168.1.1:60708] AH00875: got 8186 bytes of data, referer: https://192.168.1.123/share/page/context/mine/myfiles [Sun Apr 29 10:44:20.575571 2018] [ssl:info] [pid 11342:tid 139630102398720] (104)Connection reset by peer: [client 192.168.1.1:60708] AH01991: SSL input filter read failed. [Sun Apr 29 10:44:20.575588 2018] [proxy_ajp:debug] [pid 11342:tid 139630102398720] mod_proxy_ajp.c(393): (104)Connection reset by peer: [client 192.168.1.1:60708] AH00880: ap_get_brigade failed, referer: https://192.168.1.123/share/page/context/mine/myfiles [Sun Apr 29 10:44:20.575592 2018] [proxy_ajp:debug] [pid 11342:tid 139630102398720] mod_proxy_ajp.c(611): [client 192.168.1.1:60708] AH00890: Processing of request failed backend: 0, client: 1, referer: https://192.168.1.123/share/page/context/mine/myfiles [Sun Apr 29 10:44:20.575594 2018] [proxy_ajp:error] [pid 11342:tid 139630102398720] (104)Connection reset by peer: [client 192.168.1.1:60708] AH02822: dialog with client 192.168.1.1:60708 failed, referer: https://192.168.1.123/share/page/context/mine/myfiles [Sun Apr 29 10:44:20.575602 2018] [proxy:debug] [pid 11342:tid 139630102398720] proxy_util.c(2175): AH00943: AJP: has released connection for (localhost) [Sun Apr 29 10:44:20.575620 2018] [proxy_balancer:debug] [pid 11342:tid 139630102398720] mod_proxy_balancer.c(670): [client 192.168.1.1:60708] AH01176: proxy_balancer_post_request for (balancer://acs), referer: https://192.168.1.123/share/page/context/mine/myfiles [Sun Apr 29 10:44:20.575666 2018] [mpm_event:debug] [pid 11342:tid 139630102398720] event.c(1123): (32)Broken pipe: [client 192.168.1.1:60708] AH00470: network write failure in core output filter
Sample Tomcat Errors
With nginx proxy, there were no errors in tomcat. However Apache with mod_proxy_balancer triggered errors in the Tomcat logs
2018-04-29 10:33:06,922 INFO [webscripts.connector.RemoteClient] [ajp-apr-8009-exec-7] Exception calling (POST) http://localhost:8080/alfresco/s/api/upload?alf_ticket=TICKET_c4174f8b83582e5d2fc1969
43ba393a5ef185017
2018-04-29 10:33:06,922 INFO [webscripts.connector.RemoteClient] [ajp-apr-8009-exec-7] Error status 500 null
org.apache.http.client.ClientProtocolException
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.springframework.extensions.webscripts.connector.RemoteClient.service(RemoteClient.java:1164)
at org.springframework.extensions.webscripts.connector.RemoteClient.call(RemoteClient.java:801)
at org.springframework.extensions.webscripts.connector.HttpConnector.call(HttpConnector.java:143)
at org.springframework.extensions.webscripts.connector.AuthenticatingConnector.call(AuthenticatingConnector.java:293)
at org.springframework.extensions.webscripts.servlet.mvc.EndPointProxyController.handleRequestInternal(EndPointProxyController.java:328)
at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:853)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:743)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:485)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:410)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:337)
at org.tuckey.web.filters.urlrewrite.NormalRewrittenUrl.doRewrite(NormalRewrittenUrl.java:213)
at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:171)
at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:389)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.extensions.webscripts.servlet.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:177)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.springframework.extensions.webscripts.servlet.CSRFFilter.doFilter(CSRFFilter.java:322)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.alfresco.web.site.servlet.SSOAuthenticationFilter.doFilter(SSOAuthenticationFilter.java:474)
at org.alfresco.web.site.servlet.SSOAuthenticationFilter.doFilter(SSOAuthenticationFilter.java:443)
at org.springframework.extensions.webscripts.servlet.BeanProxyFilter.doFilter(BeanProxyFilter.java:80)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.alfresco.web.site.servlet.MTAuthenticationFilter.doFilter(MTAuthenticationFilter.java:81)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:188)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2549)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2538)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.client.NonRepeatableRequestException: Cannot retry request with a non-repeatable request entity
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:107)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
... 63 more
Caused by: java.io.IOException: Socket read failed
at org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:325)
at org.apache.coyote.ajp.AjpAprProcessor.readMessage(AjpAprProcessor.java:421)
at org.apache.coyote.ajp.AjpAprProcessor.receive(AjpAprProcessor.java:380)
at org.apache.coyote.ajp.AbstractAjpProcessor.refillReadBuffer(AbstractAjpProcessor.java:735)
at org.apache.coyote.ajp.AbstractAjpProcessor$SocketInputBuffer.doRead(AbstractAjpProcessor.java:1220)
at org.apache.coyote.Request.doRead(Request.java:438)
at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290)
at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:449)
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:315)
at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:200)
at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:140)
at org.apache.http.impl.execchain.RequestEntityProxy.writeTo(RequestEntityProxy.java:123)
at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:162)
at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
... 65 more
Components (Applications and Hardware) that resulted in this error
Apache/2.4.18 (Ubuntu)
nginx/1.13.12
Alfresco Community 2017; Alfresco Community 2016;
Linksys Router (Smart Wifi)
Unsure if chromecast devices on this specific router causes issue, as chromecast devices are known to cause network problems.
Thanks to this thread, I was able to validate assumptions of network before reconfiguring new router.
Swazzy 