tracker issue : CF-4171358

select a category, or use search below
(searches all categories and all time range)
Title:

frequent CFSchedule crashes - failing to update neo-cron.xml

| View in Tracker

Status/Resolution/Reason: Closed/Fixed/Fixed

Reporter/Name(from Bugbase): Tim Parker / Tim Parker (Tim Parker)

Created: 07/08/2016

Components: Scheduler

Versions: 2016,2018

Failure Type: Crash

Found In Build/Fixed In Build: CF2016_Update1 / 2018.0.02.312498,2016.0.08.312505

Priority/Frequency: Minor / Some users will encounter

Locale/System: ALL / Win 2008 Server R2 64 bit

Vote Count: 0

Problem Description: We are seeing frequent crashes when trying to use CFSchedule - sometimes it succeeds, sometimes it fails

Steps to Reproduce:

Actual Result: crash:
             Message: java.io.FileNotFoundException: C:\servers\ColdFusion2016\cfusion\lib\neo-cron.xml (The requested operation cannot be performed on a file with a user-mapped section open)
                Type: Application
              Detail: 

Caused by (class java.io.FileNotFoundException): 
             Message: C:\servers\ColdFusion2016\cfusion\lib\neo-cron.xml (The requested operation cannot be performed on a file with a user-mapped section open)

minimized stack with 179 entries
0: ........java.io.FileOutputStream.open0[fileoutputstream.java:-2]
1: ........java.io.FileOutputStream.open[fileoutputstream.java:270]
2: ........java.io.FileOutputStream.<init>[fileoutputstream.java:213]
3: ........java.io.FileWriter.<init>[filewriter.java:107]
4: ........coldfusion.server.ServiceBase.doSerialize[servicebase.java:247]
5: ........coldfusion.server.ServiceBase.access$100[servicebase.java:37]
6: ........coldfusion.server.ServiceBase$2.run[servicebase.java:204]
7: ........java.security.AccessController.doPrivileged[accesscontroller.java:-2]
8: ........coldfusion.server.ServiceBase.serialize[servicebase.java:200]
9: ........coldfusion.scheduling.CronServiceImpl.store[cronserviceimpl.java:491]
10: ........coldfusion.scheduling.CronServiceImpl.updateAndStore[cronserviceimpl.java:1232]
11: ........coldfusion.scheduling.CronServiceImpl.updateTask[cronserviceimpl.java:1191]
12: ........coldfusion.tagext.lang.ScheduleTag.doActionUpdate[scheduletag.java:1056]
13: ........coldfusion.tagext.lang.ScheduleTag.doStartTag[scheduletag.java:725]
14: ........coldfusion.runtime.CfJspPage._emptyTcfTag[cfjsppage.java:3698]
15: ... (our code - <CFSchedule  ACTION="Update"
		URL="...."
		PORT="80"
		TASK="..."
		OPERATION="HTTPRequest"
		PUBLISH="yes"
		PATH="...."
		FILE="jobX.log"
		INTERVAL="once"
		STARTDATE="2016-07-08"
		STARTTIME="14:00:01"
		REQUESTTIMEOUT="7200">)


Expected Result: the task gets scheduled without error

Any Workarounds:

----------------------------- Additional Watson Details -----------------------------

Watson Bug ID:	4171358

Reason:	PRHaveInfo

External Customer Info:
External Company:  
External Customer Name: Tim Parker
External Customer Email:  
External Test Config: My Hardware and Environment details:

 CFML Engine Version: ColdFusion Server 2016.0.02.299200 (Apache Tomcat/8.0.32)

         JVM version: 1.8.0_92-b14 - [64 bits, Windows Server 2008 R2]

     JVM memory (MB):  max:961.5 total:961.5 free:425.65

 CFML Engine Started: 2016-07-08 09:26:59.468 (running for 4 hours and 33 minutes)

>> Server is a VM with 3GB total memory - only other processes running are a MySQL instance and a SQL Server instance - task manager reports 77% physical memory usage, indicating that paging is probably not a factor

>> virus scans disabled

>> JVM args:

java.args=-server  -Xms1024m -Xmx1024m -XX:MetaspaceSize=192m -XX:MaxMetaspaceSize=512m -XX:+UseParallelGC -Xbatch

-Dcoldfusion.home={application.home} -Duser.language=en -Dcoldfusion.rootDir={application.home}

-Dcoldfusion.dotnet.disableautoconversion=true -Dcoldfusion.libPath={application.home}/lib

-Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random

-Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.JavaUtilLog

-Djava.util.logging.config.file={application.home}/lib/logging.properties

{classpath is default plus our private Java libraries}

Attachments:

Comments:

As described... this doesn't always happen - the error indicates a resource contention problem which isn't adequately handled. Previously we've seen (with CF10 corruption of neo-cron.xml (see #CF-3681626), so we suspect that either the problem wasn't completely fixed, or there's a regression in CF2016. Note also that this has not been observed with the same code on a similar Linux (CentOS 7) instance (same JVM version, same CF patch level)
Comment by External U.
2204 | July 08, 2016 01:30:43 PM GMT
one other environment detail... this VM has one core allocated - the Linux server where the problem is not occurring.. has two
Comment by External U.
2205 | July 08, 2016 01:31:35 PM GMT
Hi Tim, We tried to repro the scenario but were unable to do so. We would like to check on few things: 1. Does the task actually execute for you and just logs exception after executing? 2. Is neo-cron.xml corrupted? 3. Is it possible for you to send us your neo-cron.xml & error logs? It would be great if you help us to reproduce the issue. You can either attach the files on this bug or email it to suchsing@adobe.com.
Comment by Suchika S.
2206 | September 08, 2016 12:22:14 AM GMT
1) the task does not execute 2) neo-cron.xml is not corrupted 3) we are looking for relevant logs now, and will forward what we find. We have tried to reproduce the issue with no luck - this appears to be a collision between two tasks which must be in exactly the right state for the problem to occur. My guess is that the scheduler is updating neo-cron.xml at the same instant the CFSchedule tag is running, and that there's no provision in the CFSchedule logic to handle this case... This could be as simple as a loop "if file in use, wait 50ms and try again", combined with verification in the scheduler that the file is not being kept open any longer than absolutely required.
Comment by External U.
2207 | September 08, 2016 09:47:26 AM GMT
clarification - attempts to reproduce the issue in a controlled way (isolated from our CMS) have failed - however, we do still see the problem in the context of our production code
Comment by External U.
2208 | September 08, 2016 09:48:38 AM GMT
@Tim, Can you please send an email to cf.install@adobe.com ? We can schedule a call to look into the issue.
Comment by Rahul U.
2209 | October 04, 2016 12:40:02 PM GMT