tracker issue : CF-3540118

select a category, or use search below
(searches all categories and all time range)
Title:

ColdFusion not responding after clock change from GMT to BST

| View in Tracker

Status/Resolution/Reason: Closed/Withdrawn/CannotReproduce

Reporter/Name(from Bugbase): Tom Lofts / Tom Lofts (loftx)

Created: 04/11/2013

Components: Language, Functions

Versions: 10.0

Failure Type: Memory Leak

Found In Build/Fixed In Build: Final /

Priority/Frequency: Major / Unknown

Locale/System: English / Win 2003 Server

Vote Count: 10

We run a ColdFusion 10 server with a timezone set to GMT (Dublin, Edinburgh, Lisbon, London) which has started experiencing issues since the switch to British Summer Time on 31/03/2012.

Every day since the change, the server has stopped responding to requests until the ColdFusion service has been restarted. This appears to happen on a fairly (though not exactly) regular cycle every 12 hours or so, so the server has been going down twice a day for the last two weeks.

When this happens, we’re not seeing any unusual activity in terms of CPU/Memory usages on the server (though memory usage of the coldfusion.exe process seems to be quite high anyway) and no other errors – requests simply time out until the service is restarted.

The only errors relating to this I’ve been able to find appear when the server is restarted a few hundred of the following errors appear in coldfusion-error.log:

10-Apr-2013 23:06:50 org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/] created a ThreadLocal with key of type [org.apache.axiom.util.UIDGenerator$1] (value [org.apache.axiom.util.UIDGenerator$1@1926480]) and a value of type [long[]] (value [[J@15f03f5]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
10-Apr-2013 23:06:50 org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/] created a ThreadLocal with key of type [coldfusion.util.DateUtils$1] (value [coldfusion.util.DateUtils$1@1800bfa]) and a value of type [java.util.GregorianCalendar] (value [java.util.GregorianCalendar[time=1365617351930,areFieldsSet=true,areAllFieldsSet=true,lenient=false,zone=sun.util.calendar.ZoneInfo[id="Europe/London",offset=0,dstSavings=3600000,useDaylight=true,transitions=242,lastRule=java.util.SimpleTimeZone[id=Europe/London,offset=0,dstSavings=3600000,useDaylight=true,startYear=0,startMode=2,startMonth=2,startDay=-1,startDayOfWeek=1,startTime=3600000,startTimeMode=2,endMode=2,endMonth=9,endDay=-1,endDayOfWeek=1,endTime=3600000,endTimeMode=2]],firstDayOfWeek=2,minimalDaysInFirstWeek=4,ERA=1,YEAR=2013,MONTH=3,WEEK_OF_YEAR=15,WEEK_OF_MONTH=2,DAY_OF_MONTH=10,DAY_OF_YEAR=100,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=2,AM_PM=1,HOUR=7,HOUR_OF_DAY=19,MINUTE=9,SECOND=11,MILLISECOND=930,ZONE_OFFSET=0,DST_OFFSET=3600000]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. 

These seem to indicate that the problem may relate to ColdFusion’s date handling, but I’ve been unable to find any information about this online with the exception of this single forum thread http://forums.adobe.com/message/5184804


We’ve been experiencing this issue running update 3 and also after upgrading to update 8. We don’t see the issue on our other ColdFusion 10 servers, but these run very different applications with much less traffic.

I can’t be sure these errors relate to the clock change, or even that the errors in the log relate to the server problems, but if you require any more information to investigate this please let me know.

----------------------------- Additional Watson Details -----------------------------

Watson Bug ID:	3540118

External Customer Info:
External Company:  
External Customer Name: loftx
External Customer Email:  
External Test Config: My Hardware and Environment details:



ColdFusion 10 standard edition with update 8 installed running on Windows 2003.



Server timezone is set to  GMT (Dublin, Edinburgh, Lisbon, London)

Attachments:

  1. November 20, 2013 00:00:00: 1_tas_error.PNG

Comments:

I am also seeing this behavior, this is a new install updated to CF10 update 9. I have re-ran the webserver config tool as admin as per suggestions here: http://forums.adobe.com/thread/1085240 but to no effect. The server clock is set to time zone (UTC) Dublin, Edinburgh, Lisbon, London. If i change the clock to (UTC -0800) Pacific Time (US & Canada) i do not get the error. The error in the logs is: Apr 11, 2013 4:02:18 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/] created a ThreadLocal with key of type [coldfusion.util.DateUtils$1] (value [coldfusion.util.DateUtils$1@6bbdcdd5]) and a value of type [java.util.GregorianCalendar] (value [java.util.GregorianCalendar[time=1365690925920,areFieldsSet=true,areAllFieldsSet=true,lenient=false,zone=sun.util.calendar.ZoneInfo[id="Europe/London",offset=0,dstSavings=3600000,useDaylight=true,transitions=242,lastRule=java.util.SimpleTimeZone[id=Europe/London,offset=0,dstSavings=3600000,useDaylight=true,startYear=0,startMode=2,startMonth=2,startDay=-1,startDayOfWeek=1,startTime=3600000,startTimeMode=2,endMode=2,endMonth=9,endDay=-1,endDayOfWeek=1,endTime=3600000,endTimeMode=2]],firstDayOfWeek=2,minimalDaysInFirstWeek=4,ERA=1,YEAR=2013,MONTH=3,WEEK_OF_YEAR=15,WEEK_OF_MONTH=2,DAY_OF_MONTH=11,DAY_OF_YEAR=101,DAY_OF_WEEK=5,DAY_OF_WEEK_IN_MONTH=2,AM_PM=1,HOUR=3,HOUR_OF_DAY=15,MINUTE=35,SECOND=25,MILLISECOND=920,ZONE_OFFSET=0,DST_OFFSET=3600000]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Comment by External U.
15734 | April 11, 2013 09:51:59 AM GMT
I am voting on this as i am unable to install CF in a production environment with severe errors showing in the logs.
Vote by External U.
15744 | April 11, 2013 09:54:13 AM GMT
+1, Ubuntu 10.04 Server on AWS, CF10 Fresh Install. Happens on at least update 7-9 ~ Ben
Comment by External U.
15735 | April 11, 2013 01:33:55 PM GMT
Adding my vote. I placed a note in the notes section. Next time I'll know to put it here instead.
Vote by External U.
15745 | April 11, 2013 01:41:21 PM GMT
Adding my vote. Even with a new JVM and update 11 (CF 10, Windows 2012, IIS 8) the SEVERE errors continue.
Comment by External U.
15736 | September 09, 2013 05:21:13 AM GMT
We see this error in the error logs after a server restart. And I believe it may be the cause of a loss of threads the next time the server comes under modest load. In our case we're getting remote calls are being dropped causing our SAAS service to fail completely. REMOTEOBJECT ERROR Client.Packet.Encoding Error deserializing client message. null The Errors started after a server restart on 5th November ( after the change to winter time ?? ) We made no code changes , updates etc etc - only the server restart on 5th Nov triggered the issue. The server is CF10 update 11 Standard on Windows 2008 ( stared to happen on update 4 too ) Our server clock is GMT etc etc Adobe Team - Please Give this the serious attention it requires
Comment by External U.
15737 | November 13, 2013 01:19:39 PM GMT
We see this error in the error logs after a server restart. And I believe it may be the cause of a loss of threads the next time the server comes under modest load. In our case we're getting remote calls are being dropped causing our SAAS service to fail completely. REMOTEOBJECT ERROR Client.Packet.Encoding Error deserializing client message. null The Errors started after a server restart on 5th November ( after the change to winter time ?? ) We made no code changes , updates etc etc - only the server restart on 5th Nov triggered the issue. The server is CF10 update 11 Standard on Windows 2008 ( stared to happen on update 4 too ) Our server clock is GMT etc etc Adobe Team - Please Give this the serious attention it requires Continued: could it be a problem with scheduled tasks being initialised on start up ? The error seems to happen once for every scheduled task we have in CFADMIN. We're on CF10 Standard which had an issue with schedule tasks after release ??? Nov 13, 2013 5:21:29 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/] created a ThreadLocal with key of type [coldfusion.util.DateUtils$1] (value [coldfusion.util.DateUtils$1@7ce474af]) and a value of type [java.util.GregorianCalendar] (value [java.util.GregorianCalendar[time=1384362572889,areFieldsSet=true,areAllFieldsSet=true,lenient=false,zone=sun.util.calendar.ZoneInfo[id="Europe/London",offset=0,dstSavings=3600000,useDaylight=true,transitions=242,lastRule=java.util.SimpleTimeZone[id=Europe/London,offset=0,dstSavings=3600000,useDaylight=true,startYear=0,startMode=2,startMonth=2,startDay=-1,startDayOfWeek=1,startTime=3600000,startTimeMode=2,endMode=2,endMonth=9,endDay=-1,endDayOfWeek=1,endTime=3600000,endTimeMode=2]],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2013,MONTH=10,WEEK_OF_YEAR=46,WEEK_OF_MONTH=3,DAY_OF_MONTH=13,DAY_OF_YEAR=317,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=2,AM_PM=1,HOUR=5,HOUR_OF_DAY=17,MINUTE=9,SECOND=32,MILLISECOND=889,ZONE_OFFSET=0,DST_OFFSET=0]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. . . . . . . We get 14 of these followed by 4 of SEVERE: The web application [/] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@194ad86c]) and a value of type [org.apache.derby.iapi.services.context.ContextManager] (value [org.apache.derby.iapi.services.context.ContextManager@6f4582b8]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. We're on CF10 Standard Update 11 ( it happened on update 4 too ) Win 2008, IIS7.5 We're on GMT It may be related but we have a very serious issue with the server not responding to flash remote calls under very modest load. as follows - REMOTEOBJECT ERROR Client.Packet.Encoding Error deserializing client message. null This problem started after a restart on November 5th ( GMT - first restart after the clocks changed ?? ) Could this be due to running out of threads / Memory leak due to the date error?
Vote by External U.
15746 | November 13, 2013 01:19:49 PM GMT
We are seeing this behavior as well, intermittent. We have a main server and two instances, this error always occurs in one of the instances, the same one every time. We have to restart the instance to clear the error. It does seem to be related to heavier server load.
Vote by External U.
15747 | November 19, 2013 04:29:20 PM GMT
Hi, Are you still facing this issue? Have you tried with the latest update?
Comment by Suchika S.
15738 | February 26, 2014 06:46:15 AM GMT
I have the same SEVERE ( GRAVE ) entries in my production logs (and also dev logs) Environnement : Windows 2008 server, Apache 2.2.25, Coldfusion 10 with update 12 I tried to use another JRE (the last one), but it does not solve the problem. I use Coldfusion with multiple instances. Thanks in advance Stephane
Vote by External U.
15748 | February 28, 2014 04:42:32 AM GMT
CF10, update 12, Win 2012 R2. Just throwing in a me too over a year later. Tons of that same entry in coldfusion-error.log. Site associated with this instance hangs. CPU, memory, disk IO all well within normal. Restarting CF service restores normal operation.
Vote by External U.
15749 | July 20, 2014 11:23:34 AM GMT
We've been getting this same error since we went to CF10. Doesn't seem to actually cause any problems, but I'm not fond of seeing SEVERE errors in the logs.
Vote by External U.
15750 | November 06, 2014 06:04:33 AM GMT
Seeing in CF11 (11,0,02,291725) deployed on Tomcat 7.0.57, Oracle JDK 1.7.0_71 Main concern is that the tomcat server fails to stop cleanly and the process must be killed. Timezone is AEDT (Australian Eastern Daylight Time)
Vote by External U.
15751 | November 19, 2014 06:18:40 PM GMT
Verified the bug on CF10 HF14 and CF11 HF3 on Windows 2012.
Comment by Suchika S.
15739 | December 03, 2014 04:48:18 AM GMT
Attaching the error log
Comment by Suchika S.
15740 | December 03, 2014 04:50:45 AM GMT
The CF hang issue is not reproducible. In some random cases, the SEVERE log from tomcat does appear when the server is started but that is not related to the timezone change and is unlikely to cause a server hang. Tomcat logs it whenever a webapp does not call remove() on a threadlocal it created to warn about a POSSIBLE memory leak. In our case DateUtils stores a Calendar object in ThreadLocal which is unlikely to cause a memory leak. We can try to prevent the SEVERE log from coming up but that will need some consistent repro case. We need more info to address the CF hang issue.
Comment by Himavanth R.
15741 | January 09, 2015 04:57:31 AM GMT
CF 11 Update 3, seeing same error: SEVERE: The web application [/] created a ThreadLocal with key of type [coldfusion.util.DateUtils$1] (value [coldfusion.util.DateUtils$1@3a8cc2b1]) and a value of type [java.util.GregorianCalendar] (value [java.util.GregorianCalendar[time=1421944727395,areFieldsSet=true,areAllFieldsSet=true,lenient=false,zone=sun.util.calendar.ZoneInfo[id="America/Chicago",offset=-21600000,dstSavings=3600000,useDaylight=true,transitions=235,lastRule=java.util.SimpleTimeZone[id=America/Chicago,offset=-21600000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2015,MONTH=0,WEEK_OF_YEAR=4,WEEK_OF_MONTH=4,DAY_OF_MONTH=22,DAY_OF_YEAR=22,DAY_OF_WEEK=5,DAY_OF_WEEK_IN_MONTH=4,AM_PM=0,HOUR=10,HOUR_OF_DAY=10,MINUTE=38,SECOND=47,MILLISECOND=395,ZONE_OFFSET=-21600000,DST_OFFSET=0]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Vote by External U.
15752 | January 22, 2015 12:23:59 PM GMT
We are still not able to reproduce the server hang issue. The note from Himavanth mentions the reasons for the SEVERE log entry.
Comment by Suchika S.
15742 | June 19, 2015 03:39:45 AM GMT
Since there is no response from the customer, I am closing the bug. Please reply to the bug if you are still facing the issue. We would re-open the bug then. Thanks, Suchika.
Comment by Suchika S.
15743 | September 21, 2015 02:27:53 AM GMT
Seeing this error multiple times in the log whenever I shut down a CF instance. CF 11, update 6, on Windows Server 2012 R2, US eastern time zone. Nov 17, 2015 5:11:12 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks SEVERE: The web application [/] created a ThreadLocal with key of type [coldfusion.util.DateUtils$1] (value [coldfusion.util.DateUtils$1@3f7017c]) and a value of type [java.util.GregorianCalendar] (value [java.util.GregorianCalendar[time=-2208970800000,areFieldsSet=true,areAllFieldsSet=false,lenient=false,zone=sun.util.calendar.ZoneInfo[id="America/New_York",offset=-18000000,dstSavings=3600000,useDaylight=true,transitions=235,lastRule=java.util.SimpleTimeZone[id=America/New_York,offset=-18000000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=?,YEAR=1900,MONTH=0,WEEK_OF_YEAR=?,WEEK_OF_MONTH=?,DAY_OF_MONTH=1,DAY_OF_YEAR=?,DAY_OF_WEEK=?,DAY_OF_WEEK_IN_MONTH=?,AM_PM=?,HOUR=?,HOUR_OF_DAY=0,MINUTE=0,SECOND=0,MILLISECOND=?,ZONE_OFFSET=?,DST_OFFSET=?]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Vote by External U.
15753 | November 17, 2015 05:09:45 PM GMT