tracker issue : CF-4181270

select a category, or use search below
(searches all categories and all time range)
Title:

Websocket Proxy in infinite loop

| View in Tracker

Status/Resolution/Reason: Closed/Fixed/

Reporter/Name(from Bugbase): Jay Kirk / Jay Kirk (Jay Kirk)

Created: 08/15/2016

Components: Web Socket

Versions: 11.0

Failure Type:

Found In Build/Fixed In Build: CF11_Final /

Priority/Frequency: Major / All users will encounter

Locale/System: English / Win 2012 Server x64

Vote Count: 0

Listed in the version 2016.0.03.300466 Issues Fixed doc 
Problem Description: Websockets locking up thread stuck in infinite loop.  What we're seeing is it's receiving a poorly formatted message, and then gets stuck in infinite loop.

Steps to Reproduce: We're seeing this in our production environment.  There is a thread being locked up in the processor and websockets become unresponsive.  See

Actual Result: Websockets locked up and unresponsive.

Expected Result: Message processed or skipped.

Any Workarounds:  Restarting Coldfusion app to free thread.

----------------------------- Additional Watson Details -----------------------------

Watson Bug ID:	4181270

External Customer Info:
External Company:  
External Customer Name: Jay Kirk
External Customer Email:  
External Test Config: My Hardware and Environment details:

Windows Server 2012 x64 (latest)

Intel CPU E5-2680 v2 @ 2.80 GHz

7.50 GB Memory

Coldfusion 11 (latest)

IIS 8.0

Attachments:

  1. August 16, 2016 00:00:00: 1_websocketproxy.txt

Comments:

Here is the thread that is in an infinite loop: "pool-2-thread-84" prio=6 tid=0x0000007897b59800 nid=0x1970 runnable [0x00000078a08bf000] java.lang.Thread.State: RUNNABLE at java.lang.Throwable.fillInStackTrace(Native Method) at java.lang.Throwable.fillInStackTrace(Throwable.java:783) - locked <0x00000007a6b86ed8> (a java.lang.NumberFormatException) at java.lang.Throwable.<init>(Throwable.java:265) at java.lang.Exception.<init>(Unknown Source) at java.lang.RuntimeException.<init>(Unknown Source) at java.lang.IllegalArgumentException.<init>(Unknown Source) at java.lang.NumberFormatException.<init>(Unknown Source) at java.lang.NumberFormatException.forInputString(Unknown Source) at java.lang.Integer.parseInt(Unknown Source) at java.lang.Integer.valueOf(Unknown Source) at coldfusion.tagext.net.websocket.server.proxy.ProxyMessageProcessor.processProxyMessages(ProxyMessageProcessor.java:36) at coldfusion.tagext.net.websocket.server.proxy.WSProxyMessageReader$WebSocketProxyReadCompletionHandler.parseReceivedMessage(WSProxyMessageReader.java:82) at coldfusion.tagext.net.websocket.server.proxy.WSProxyMessageReader$WebSocketProxyReadCompletionHandler.completed(WSProxyMessageReader.java:59) at coldfusion.tagext.net.websocket.server.proxy.WSProxyMessageReader$WebSocketProxyReadCompletionHandler.completed(WSProxyMessageReader.java:47) at sun.nio.ch.Invoker.invokeUnchecked(Unknown Source) at sun.nio.ch.Invoker$2.run(Unknown Source) at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source)
Comment by External U.
2015 | August 15, 2016 03:47:03 PM GMT
Hi Jay, Can you please share a code snippet where you see this issue? Or if you have a repro case? This will help us in identifying the issue faster. Also, is it a particular scenario where this issue is seen? Or do you see it for all proxy cases? Thanks, Kailash
Comment by Kailash B.
2016 | August 16, 2016 12:13:09 AM GMT
Hi Kailash, The bug is in the coldfusion/tagext/net/websocket/server/proxy/ProxyMessageProcessor.class in cfusion.jar. The infinite loop occurs in the following function: public static Object[] processProxyMessages(String msg); If (String msg) past into this function doesn’t have an integer at the beginning, on line 36, the Integer.valueOf() function will throw a Numberformat exception. The function gets in an infinite loop and keeps calling the Integer.valueOf() for the same string. This causes the web sockets become unresponsive. It’s happening about every other day to our server. With millions of web socket messages coming in, we aren’t able to zero in on the exact message that causes but guessing it happens when a partial or malformatted web socket message comes through. The attached log file (websocketproxy.txt) has an example of a message that comes across to cause the loop.
Comment by External U.
2017 | August 16, 2016 10:13:52 AM GMT
Hi Jay, We see it that this is an issue in our code, but we have not been able to repro it. We do have a fix in the form of a patch in place for the same. Can you send a mail to cfinstal@adobe.com asking for the patch? Also, do let us know if this fixes the issue so that we can go ahead and close the bug.
Comment by Kailash B.
2018 | August 22, 2016 12:22:48 AM GMT
Hi Kailash, Great, thank you! I have sent an email requesting the patch. Once we receive it, it will take a few days to test. I will let you know the results! Thanks, Jay
Comment by External U.
2019 | August 22, 2016 09:46:24 AM GMT
Hi Kailash, We pushed the fix into our production environment last night. So far so good, but it can take a few days before we see the error normally. I will keep you updated on the status. Thanks, Jay
Comment by External U.
2020 | August 23, 2016 09:56:17 AM GMT
Thanks Jay. We will wait for a few more days before we close this bug.
Comment by Vamseekrishna N.
2021 | August 24, 2016 09:15:23 PM GMT
Hi Kailash, It has been two full days and we have not seen the issue return. I believe we can consider the bug closed at this point. Thank you so much for your quick response and fix! Thanks, Jay
Comment by External U.
2022 | August 25, 2016 08:26:54 AM GMT
Hi Jay, Thank you for the response. I hope the fix is working perfectly for you? Thanks, Kailash
Comment by Kailash B.
2023 | August 29, 2016 11:17:27 PM GMT
Hi Kailash, Yes, everything is working great with the web sockets now, than you. Thanks, Jay
Comment by External U.
2024 | August 30, 2016 08:08:14 AM GMT