Skip to content

Evasive Errors

Very Evasive Proxy Error from Nginx and Apache when uploading files that take more than 1 second to upload. However, the problem was not with Proxy Servers, nor the backend application server. The problem came from the router.

TL;DR

Proxy Error from Nginx, and Apache on large uploads
Large file upload breaks;
Large file uploads disconnect;
File Upload fails if upload >
1000ms or 1 second
No true fix, except for switching routers.
All configurations related to searches on these errors did not fix the problem.

Errors

Sample Nginx Errors

2018/04/29 09:41:36 [debug] 5455#5455: *947 http client request body recv -2
2018/04/29 09:41:36 [debug] 5455#5455: *947 http client request body rest 74726165
2018/04/29 09:41:36 [debug] 5455#5455: *947 event timer add: 5: 60000:67967884
2018/04/29 09:41:36 [debug] 5455#5455: *947 event timer del: 20: 77987884
2018/04/29 09:41:36 [debug] 5455#5455: *947 http run request: "/share/proxy/alfresco/api/upload?"
2018/04/29 09:41:36 [debug] 5455#5455: *947 http upstream read request handler
2018/04/29 09:41:36 [debug] 5455#5455: *947 http upstream send request
2018/04/29 09:41:36 [debug] 5455#5455: *947 http upstream send request body
2018/04/29 09:41:36 [debug] 5455#5455: *947 http read client request body
2018/04/29 09:41:36 [debug] 5455#5455: *947 SSL_read: -1
2018/04/29 09:41:36 [debug] 5455#5455: *947 SSL_get_error: 5
2018/04/29 09:41:36 [debug] 5455#5455: *947 peer shutdown SSL cleanly
2018/04/29 09:41:36 [debug] 5455#5455: *947 http client request body recv 0
2018/04/29 09:41:36 [info] 5455#5455: *947 client prematurely closed connection while sending request to upstream, client: 192.168.1.1, server: content.mahabir.org, request: "POST /share/proxy/alfresco/api/upload HTTP/1.1", upstream: "http://[::1]:8080/share/proxy/alfresco/api/upload", host: "192.168.1.123", referrer: "https://192.168.1.123/share/page/context/mine/myfiles"
2018/04/29 09:41:36 [debug] 5455#5455: *947 finalize http upstream request: 400
2018/04/29 09:41:36 [debug] 5455#5455: *947 finalize http proxy request
2018/04/29 09:41:36 [debug] 5455#5455: *947 free rr peer 2 0
2018/04/29 09:41:36 [debug] 5455#5455: *947 close http upstream connection: 20
2018/04/29 09:41:36 [debug] 5455#5455: *947 free: 000055CC3CD48100, unused: 48
2018/04/29 09:41:36 [debug] 5455#5455: *947 reusable connection: 0
2018/04/29 09:41:36 [debug] 5455#5455: *947 http finalize request: 400, "/share/proxy/alfresco/api/upload?" a:1, c:1
2018/04/29 09:41:36 [debug] 5455#5455: *947 http terminate request count:1
2018/04/29 09:41:36 [debug] 5455#5455: *947 http terminate cleanup count:1 blk:0
2018/04/29 09:41:36 [debug] 5455#5455: *947 http posted request: "/share/proxy/alfresco/api/upload?"
2018/04/29 09:41:36 [debug] 5455#5455: *947 http terminate handler count:1
2018/04/29 09:41:36 [debug] 5455#5455: *947 http request count:1 blk:0
2018/04/29 09:41:36 [debug] 5455#5455: *947 http close request
2018/04/29 09:41:36 [debug] 5455#5455: *947 http log handler

 

Sample Apache Errors

[Sun Apr 29 10:32:50.490278 2018] [mpm_event:notice] [pid 11071:tid 140462232258432] AH00489: Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g configured -- resuming normal operations
[Sun Apr 29 10:32:50.490328 2018] [core:notice] [pid 11071:tid 140462232258432] AH00094: Command line: '/usr/sbin/apache2'
[Sun Apr 29 10:33:06.914022 2018] [proxy_ajp:error] [pid 11074:tid 140461880575744] (104)Connection reset by peer: [client 192.168.1.1:58335] AH02822: dialog with client 192.168.1.1:58335 failed, ref
...skipping...
[Sun Apr 29 10:44:19.586534 2018] [proxy_balancer:debug] [pid 11342:tid 139630102398720] mod_proxy_balancer.c(616): [client 192.168.1.1:60708] AH01172: balancer://acs: worker (ajp://localhost) rewritten to ajp://localhost/share/proxy/alfresco/api/upload, referer: https://192.168.1.123/share/page/context/mine/myfiles
[Sun Apr 29 10:44:19.586544 2018] [proxy:debug] [pid 11342:tid 139630102398720] proxy_util.c(1785): AH00924: worker ajp://localhost shared already initialized
[Sun Apr 29 10:44:19.586550 2018] [proxy:debug] [pid 11342:tid 139630102398720] proxy_util.c(1832): AH00927: initializing worker ajp://localhost local
[Sun Apr 29 10:44:19.586572 2018] [proxy:debug] [pid 11342:tid 139630102398720] proxy_util.c(1867): AH00930: initialized pool in child 11342 for (localhost) min=0 max=25 smax=25
[Sun Apr 29 10:44:19.586581 2018] [proxy:debug] [pid 11342:tid 139630102398720] mod_proxy.c(1160): [client 192.168.1.1:60708] AH01143: Running scheme balancer handler (attempt 0), referer: https://192.168.1.123/share/page/context/mine/myfiles
[Sun Apr 29 10:44:19.586590 2018] [proxy_ajp:debug] [pid 11342:tid 139630102398720] mod_proxy_ajp.c(743): [client 192.168.1.1:60708] AH00895: serving URL ajp://localhost/share/proxy/alfresco/api/upload, referer: https://192.168.1.123/share/page/context/mine/myfiles
[Sun Apr 29 10:44:19.586632 2018] [proxy:debug] [pid 11342:tid 139630102398720] proxy_util.c(2160): AH00942: AJP: has acquired connection for (localhost)
[Sun Apr 29 10:44:19.586643 2018] [proxy:debug] [pid 11342:tid 139630102398720] proxy_util.c(2213): [client 192.168.1.1:60708] AH00944: connecting ajp://localhost/share/proxy/alfresco/api/upload to localhost:8009, referer: https://192.168.1.123/share/page/context/mine/myfiles
[Sun Apr 29 10:44:19.586942 2018] [proxy:debug] [pid 11342:tid 139630102398720] proxy_util.c(2422): [client 192.168.1.1:60708] AH00947: connected /share/proxy/alfresco/api/upload to localhost:8009, referer: https://192.168.1.123/share/page/context/mine/myfiles
[Sun Apr 29 10:44:19.587032 2018] [proxy:debug] [pid 11342:tid 139630102398720] proxy_util.c(2799): AH02824: AJP: connection established with 127.0.0.1:8009 (localhost)
[Sun Apr 29 10:44:19.589149 2018] [proxy_ajp:debug] [pid 11342:tid 139630102398720] mod_proxy_ajp.c(275): [client 192.168.1.1:60708] AH00873: data to read (max 8186 at 4), referer: https://192.168.1.123/share/page/context/mine/myfiles
[Sun Apr 29 10:44:19.589187 2018] [proxy_ajp:debug] [pid 11342:tid 139630102398720] mod_proxy_ajp.c(289): [client 192.168.1.1:60708] AH00875: got 8186 bytes of data, referer: https://192.168.1.123/share/page/context/mine/myfiles
[Sun Apr 29 10:44:20.575571 2018] [ssl:info] [pid 11342:tid 139630102398720] (104)Connection reset by peer: [client 192.168.1.1:60708] AH01991: SSL input filter read failed.
[Sun Apr 29 10:44:20.575588 2018] [proxy_ajp:debug] [pid 11342:tid 139630102398720] mod_proxy_ajp.c(393): (104)Connection reset by peer: [client 192.168.1.1:60708] AH00880: ap_get_brigade failed, referer: https://192.168.1.123/share/page/context/mine/myfiles
[Sun Apr 29 10:44:20.575592 2018] [proxy_ajp:debug] [pid 11342:tid 139630102398720] mod_proxy_ajp.c(611): [client 192.168.1.1:60708] AH00890: Processing of request failed backend: 0, client: 1, referer: https://192.168.1.123/share/page/context/mine/myfiles
[Sun Apr 29 10:44:20.575594 2018] [proxy_ajp:error] [pid 11342:tid 139630102398720] (104)Connection reset by peer: [client 192.168.1.1:60708] AH02822: dialog with client 192.168.1.1:60708 failed, referer: https://192.168.1.123/share/page/context/mine/myfiles
[Sun Apr 29 10:44:20.575602 2018] [proxy:debug] [pid 11342:tid 139630102398720] proxy_util.c(2175): AH00943: AJP: has released connection for (localhost)
[Sun Apr 29 10:44:20.575620 2018] [proxy_balancer:debug] [pid 11342:tid 139630102398720] mod_proxy_balancer.c(670): [client 192.168.1.1:60708] AH01176: proxy_balancer_post_request for (balancer://acs), referer: https://192.168.1.123/share/page/context/mine/myfiles
[Sun Apr 29 10:44:20.575666 2018] [mpm_event:debug] [pid 11342:tid 139630102398720] event.c(1123): (32)Broken pipe: [client 192.168.1.1:60708] AH00470: network write failure in core output filter

 

Sample Tomcat Errors

With nginx proxy, there were no errors in tomcat. However Apache with mod_proxy_balancer triggered errors in the Tomcat logs

2018-04-29 10:33:06,922  INFO  [webscripts.connector.RemoteClient] [ajp-apr-8009-exec-7] Exception calling (POST) http://localhost:8080/alfresco/s/api/upload?alf_ticket=TICKET_c4174f8b83582e5d2fc1969
43ba393a5ef185017
 2018-04-29 10:33:06,922  INFO  [webscripts.connector.RemoteClient] [ajp-apr-8009-exec-7] Error status 500 null
 org.apache.http.client.ClientProtocolException
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.springframework.extensions.webscripts.connector.RemoteClient.service(RemoteClient.java:1164)
        at org.springframework.extensions.webscripts.connector.RemoteClient.call(RemoteClient.java:801)
        at org.springframework.extensions.webscripts.connector.HttpConnector.call(HttpConnector.java:143)
        at org.springframework.extensions.webscripts.connector.AuthenticatingConnector.call(AuthenticatingConnector.java:293)
        at org.springframework.extensions.webscripts.servlet.mvc.EndPointProxyController.handleRequestInternal(EndPointProxyController.java:328)
        at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
        at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:853)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:743)
        at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:485)
        at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:410)
        at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:337)
        at org.tuckey.web.filters.urlrewrite.NormalRewrittenUrl.doRewrite(NormalRewrittenUrl.java:213)
        at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:171)
        at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
        at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
        at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:389)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.springframework.extensions.webscripts.servlet.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:177)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.springframework.extensions.webscripts.servlet.CSRFFilter.doFilter(CSRFFilter.java:322)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.alfresco.web.site.servlet.SSOAuthenticationFilter.doFilter(SSOAuthenticationFilter.java:474)
        at org.alfresco.web.site.servlet.SSOAuthenticationFilter.doFilter(SSOAuthenticationFilter.java:443)
        at org.springframework.extensions.webscripts.servlet.BeanProxyFilter.doFilter(BeanProxyFilter.java:80)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.alfresco.web.site.servlet.MTAuthenticationFilter.doFilter(MTAuthenticationFilter.java:81)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
         at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
                at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
                at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
                at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:188)
                at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
                at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2549)
                at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2538)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
                at java.lang.Thread.run(Thread.java:748)
        Caused by: org.apache.http.client.NonRepeatableRequestException: Cannot retry request with a non-repeatable request entity
                at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:107)
                at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
                at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
                ... 63 more
        Caused by: java.io.IOException: Socket read failed
                at org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:325)
                at org.apache.coyote.ajp.AjpAprProcessor.readMessage(AjpAprProcessor.java:421)
                at org.apache.coyote.ajp.AjpAprProcessor.receive(AjpAprProcessor.java:380)
                at org.apache.coyote.ajp.AbstractAjpProcessor.refillReadBuffer(AbstractAjpProcessor.java:735)
                at org.apache.coyote.ajp.AbstractAjpProcessor$SocketInputBuffer.doRead(AbstractAjpProcessor.java:1220)
                at org.apache.coyote.Request.doRead(Request.java:438)
                at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290)
                at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:449)
                at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:315)
                at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:200)
                at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:140)
                at org.apache.http.impl.execchain.RequestEntityProxy.writeTo(RequestEntityProxy.java:123)
                at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
                at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:162)
                at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
                at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
                at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
                at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
                at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
                ... 65 more

Components (Applications and Hardware) that resulted in this error

Apache/2.4.18 (Ubuntu)
nginx/1.13.12
Alfresco Community 2017; Alfresco Community 2016;
Linksys Router (Smart Wifi)

Unsure if chromecast devices on this specific router causes issue, as chromecast devices are known to cause network problems.

Thanks to this thread, I was able to validate assumptions of network before reconfiguring new router.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: