tracker issue : CF-4204021

To fix urgently by Adobe team (please). And tnx to Pete Freitag.

Vote by T. O.

There is no way we can install this update on any of the servers that we maintain. This bug would sink us with our clients if they lost their scheduled tasks.

Comment by Wil G.

Just a note that the neo-cron.bak file created by the update is also empty. I'm guessing that it gets overwritten when the unistall of the update is used, but either way my tasks our gone and the backup file is useless.

Comment by John S.

@ John S. - If you do have a valid backup file (check the contents of the neo-cron.cml file): 1) Uninstall the update, 2) Stop de CF service, 3) put the backup file in place, 4) restart the CF service, 5) login as admin and your scheduled tasks are there again.

Comment by T. O.

@T.O. thanks, restoring that file from a backup now.

Comment by John S.

We had to recreate all scheduled tasks on all servers.

Comment by George B.

Hoping Adobe has a correction soon. In the meantime, there is a work around (which worked for me) , share by Keith tussing in a comment on the cf team blog post about ehr the update (https://coldfusion.adobe.com/2019/02/coldfusion-2018-release-update-2-coldfusion-2016-release-update-8-coldfusion-11-update-16-released/) : "Workaround – Manually edit a backup copy of the neo-crons.xml adding <var name=’clustered’><boolean value=’false’/></var> to each job and then replace the wiped out copy with this one and restart the services. If it gets wiped out again you missed a job. You can also rebuild all of your jobs by hand too."

Comment by Charlie A.

Hoping Adobe has a correction soon. In the meantime, there is a work around (which worked for me) , share by Keith tussing in a comment on the cf team blog post about ehr the update (https://coldfusion.adobe.com/2019/02/coldfusion-2018-release-update-2-coldfusion-2016-release-update-8-coldfusion-11-update-16-released/) : "Workaround – Manually edit a backup copy of the neo-crons.xml adding <var name=’clustered’><boolean value=’false’/></var> to each job and then replace the wiped out copy with this one and restart the services. If it gets wiped out again you missed a job. You can also rebuild all of your jobs by hand too."

Comment by Charlie A.

After recovering a pre-update neo-crons.xml I was able to recover my scheduled tasks by adding the following to each task. This is a little different than what was posted on the official blog and above. <var name='clustered'> <string>false</string> </var>

Comment by John S.

Happened to our task server, thankfully had a backup of neo-cron.xml, rolled back the patch and restored the file....

Vote by John C.

It would be great if Adobe (or others experiencing this problem) could confirm one way or the other whether it was that each task (to work after the update) need to have: <var name=’clustered’><boolean value=’false’/></var> or <var name='clustered'><string>false</string></var> We have people here reporting both. My recollection in working with a client was that only one way DID work, and I could have sworn it was the first way (as a boolean, and only that way). But I see John saying it worked for him with string. We just all need to know, for sure, if it's one or the other (or perhaps both) that work. :-) And I do hope Adobe (or others) will get to the bottom of why this happens for some but not ALL (losing tasks with the update). From what I can gather (though not fully researched), it seems to be affecting those who had tasks that were existing from prior migrations (or CAR imports) from older CF versions, or at least since some older version of CF. So that people who created all their tasks in the UI from scratch (in later versions of CF) did NOT have the problem, as they had NOT migrated or imported any. (To be clear, when I refer to migrate or import, I do mean using the features enabled by the CF Admin, rather than someone just copying the neo-cron.xml from one instance to another. That is never wise, if the versions of CF are different which created that file. By import I do mean via the CAR mechanism. And by "migrate", I mean the feature that runs automatically at first launch of the admin, if the installer had detected that you had a previous CF version on the same machine, or that can be triggered to run again after the fact by tweaking the adminconfig.xml file.)

Comment by Charlie A.

Charlie: We had this problem on a brand new server where CF 2016 was the first and only version installed and we set up scheduled tasks through the CF Admin. -bg

Comment by Bill G.

OK, Bill. Thanks for sharing that. I'm simply trying to help us all gather evidence and determine what's really amiss. So I would wonder: do you have a backup of your neo-cron.xml from before the update? Does it have ANY clustered elements in it at all? It could be simply that that (post the update) CF is mistakenly requiring those be there, where it did not before. And in that case, did you add the elements as above (and if so, which), and did it then work? One more possible point of evidence: is your CF running Standard? Since the feature to cluster scheduled tasks is new in CF10 Enterprise, I would expect that CF would NOT have put that clustered element in there before. And perhaps the issue is that they (the CF code, per the update) didn't think to accomodate that situation (of no clustered elements present in each task.) And it may be that even those on CF Enterprise--who had imported their previous tasks--had the SAME problem, where those tasks did not have the clustered element, if perhaps they were carrying forward (perhaps over multiple releases) tasks that had been created even in CF9 or earlier, before clustering of tasks was offered. Again, just trying to help think this through. I suspect (hope) that Adobe is doing even more diligent research.

Comment by Charlie A.

We don't have a backup, but the web host restored the xml file from a backup run the night before the update was run to get us going again. Would the current xml file be useful to you? We have CF Enterprise version 2016.0.07.311392. The scheduled tasks had not been imported.

Comment by Bill G.

The word "clustered" does not appear in our neo-cron.xml file.

Comment by Bill G.

Just a few more details to provide in regards to Charlie's comments. Our server is CF 2016 Standard, we previously upgraded from CF 10 Standard, but I don't remember for sure if I manually recreated our tasks when we went to CF 2016 or if we imported them from our previous installation. We are also running on Windows Server 2012 R2 if that plays a role in this. I tried to recover our tasks by adding <var name=’clustered’><boolean value=’false’/></var> to each, but upon a restart of the server, they would all be removed. I manually created a task to see how the XML was formatted and that is where I found the <var name='clustered'><string>false</string></var> entry on the one task created. Adding this to all of my previous jobs did allow me to recover my scheduled tasks successfully.

Comment by John S.

major problem. Lost all our scheduled jobs.

Vote by Adam G.

This comment doesn't help those who have already updated, but to those who haven't yet, and don't have hundreds of scheduled tasks defined, it's possible to add the missing 'clustered' var name to neo-cron.xml through the CF Administrator BEFORE updating. 1. Open the scheduled task in CF Admin 2. Click the 'Show Additional Settings' button at the bottom 3. Check the 'Cluster' checkbox at the bottom and click 'Submit' 4. You should receive an error message about enabling clustering (if it's not enabled) 5. Ignore the error message, uncheck the 'Cluster' checkbox and click 'Submit' 6. The task should save successfully, and <var name='clustered'><string>false</string></var> should've been added to the XML config file. 7. Rinse and repeat for all tasks. Note: doing this also appears to remove from the task what I imagine are obsolete settings from neo-cron.xml (disabled, eventhandlerrp) left over from previous versions. Tested on CF 2016 Update 7, which has previously been upgraded from 10 to 11 to 2016,

Comment by Michael C.

Actually, forget the check/error/uncheck steps in my previous comment, just open the task in CF Admin and click 'Submit' to save it, has exactly the same effect!

Comment by Michael C.

Trying to find a backup neo-cron file. Why doesn't the updater save a backup??

Vote by Eric B.

I have no backup. Have to re-create all my tasks by hand from memory. This blows.

Comment by Eric B.

I tried to use <var name='clustered'><string>false</string></var> instead of <var name=’clustered’><boolean value=’false’/></var> as the workaround Adobe reported and that worked for me

Comment by Cedric D.

Eric asks in a vote comment (on the right), "Trying to find a backup neo-cron file. Why doesn't the updater save a backup??" My answer will be of little consolation, but FWIW, it's not the updater that lost the neo-cron.back file. It was the first launch of CF, which when it found your neo-cron.xml didn't have what it liked (because of a mistake in the updater), it was THEN that your neo-cron files got messed up. (I saw this with a client, though I didn't notice if it first saved off the existing neo-cron.xml over the bak, or it saved what it now modified, removing what it saw as "bad" tasks, over the bak.) Anyway, I know that's a technicality. You wish CF would have done a better job to save the tasks. (That's really a bigger discussion, because CF also overwrites the bak file WHENEVER a task simply RUNS, not just when tasks are added/edited/deleted. That's because it holds not only the DEFINITION of the tasks but info about the last EXECUTION of any.) I'm simply pointing out that technically there was nothing in the updater that was SUPPOSED to mess with the neo-cron.xml, so there was no reason therefore for them to have thought to back it up as part of the update. And I can confirm, there are NO neo files in the cfusion\hf-updates\hf-2018-00002-313961\backup\lib folder (nor have their been in the lib folders of backups for MOST CF updates, as the updater only backs up the files it IS knowingly changing).

Comment by Charlie A.

Following up on my last comment, I recalled that I had in fact opened a ticket on the issue of how CF keeps only a single generation backup of the neo-cron.xml, and worse that it's overwritten whenever a task RUNS, not just when tasks are added or changed. That ticket is here: https://tracker.adobe.com/#/view/CF-4200009 Please go add votes to it. And also, notice that Adobe had CLOSED the ticket last April, for reasons that I don't agree with and I just commented on in the blog. While adding comments there may be helpful, I do wonder if they don't bother to read comments of tickets that they have closed months ago. So you may want to add a comment here, to get their attention. Indeed, I would argue (as I do in my comment on that other ticket just now) that if CF kept both SEPARATED out tracking the DEFINITION and EXECUTION of tasks to separate files, and better still kept MORE THAN ONE backup generation, any of us in this boat (due to this latest bug in the Feb 19 updates) would have more likely been able to RECOVER from the problem. Like Eric said, some people will have no good bak file, and may not readily have a way to recover any from backups, etc. (Sure, one could argue that folks need better backups on their own. But that doesn't take away from a reasonable hope that CF would save for us more backups of these relatively tiny neo*.xml files.)

Comment by Charlie A.

I appreciate your help and comments Charlie. I voted on your bug - seems so simple and logical to me. It took me 9+ hours, but I was able to recreate all of my scheduled tasks by hand by poring through the scheduler logs and matching up each task with the cfm page that runs. I've updated my CF update plan to include a manual backup of the neo-cron.xml file as well as running a full CAR archive before installing.

Comment by Eric B.

I restored a backup of neo-cron.xml after it was reset by this patch. On the next startup of ColdFusion the file was reset again. I had to uninstall the CF11u16 patch and restore the backup of neo-cron.xml to get the scheduled tasks back.

Vote by Jason L.

Thanks, Eric. And to Jason, who voted and offered a comment about how he could only revert to the previous update, he seems to have missed the discussions here about how he could get it working with that backed up neo-cron.xml, by editing it to have each task have the needed "archived" element, which the recent update is mistakenly requiring. Just pointing this out in case somehow someone else may come here and maybe is not inclined to read through ALL the prior comments, but may notice this more recent one. (But no question: we need Adobe to fix the updates to NOT have this bug.)

Comment by Charlie A.

major issue. Hotfix 8 is NOT productive ready!!

Vote by Jonathan T.

Awful. Just lots my Scheduled Tasks. From what I can tell, the backup does not have the file either. This is really sad.

Comment by Michael T.

seems to be fixed with update 9 (2016), https://helpx.adobe.com/coldfusion/kb/bugs-fixed-coldfusion-2016-update-9.html#knownissues but update 10 was also released https://helpx.adobe.com/coldfusion/kb/coldfusion-2016-update-10.html

Comment by Jonathan T.

Adobe folks, could you tell us why this has been changed (this morning) in priority from "normal" to "critical" and now to "blocker"? (We who are subscribed get an email on any change to the ticket.) What I mean is, since the problem was said to have been fixed in the updates for CF11 and 2016 that were released Feb 22, after the ones on Feb 12 which had caused the problem, why this change now, 3 weeks later? I see no new comments here (or vote comments) to suggest why this would still be an issue. Is there new info you have, to suggest this is still hitting people? Can you share it with us? And I never noticed if the "fixed in" value had ever been filled in back with those Feb 22 updates, so I'm not sure what to make of it being blank now. Thanks.

Comment by Charlie A.

Can anyone from Adobe address the questions in the last comment?

Comment by Charlie A.