tracker issue : CF-4207069

select a category, or use search below
(searches all categories and all time range)
Title:

Very slow startup time CF2018 update 6 and above - compiler issue?

| View in Tracker

Status/Resolution/Reason: Closed/Withdrawn/CannotReproduce

Reporter/Name(from Bugbase): Doug C. / ()

Created: 02/13/2020

Components: General Server, Performance

Versions: 2018

Failure Type: Performance Issue

Found In Build/Fixed In Build: 2018 update 6 and above /

Priority/Frequency: Normal / All users will encounter

Locale/System: English / Win 2016

Vote Count: 2

Problem Description:

It looks like u6 has introduced a significant overhead to the compiler, I would guess with the new fat arrow parsing (props to John Whish for that thought).
 
2018 u6 – 550,000 + ms startup
2018 u4 – 130,000 ms startup
2016 u13 – 110,000 ms startup
2011 u19 – 115,000 ms startup
 
U6 has nearly a 5x slowdown!

Image is a stack trace of an initial request being processed on startup

Steps to Reproduce:

John Whish has helpfully created a test suite that highlights the issue. He has added it git hub here:
 
https://github.com/aliaspooryorik/cf_compiler_test
 
If you grab the source and follow the instructions in the README.md it will use command box to grab test box and run a simple set of tests – these are purely for this exercise and simply create a lot of cfc’s with functions that don’t really do anything.

We suspect this is a compiler issue related to the introduction of fat arrow syntax in update 6

Actual Result:
If I run this on my setup I see cf2018 u6 takes around 80,000ms on its first run

Expected Result:
On cf2018 u4 it takes 40,000ms on first run.

Any Workarounds:
Use cf2018 u4 - but this is unacceptable as u6+ includes several security fixes

Attachments:

Comments:

I've noted slower startup times as well, but I didn't measure them before and after. I was attributing them to a the server having minimal resources. I am running CF18u6 on a m5.large at AWS and getting about 7 minute startup times. This has been causing issues with an auto-scaling infrastructure.
Comment by Wil G.
33115 | February 13, 2020 04:31:16 PM GMT
Adding my vote. I created the test suite as we wanted to be able to prove it wasn't environmental.
Vote by John W.
33116 | February 13, 2020 04:46:53 PM GMT
This is an issue, as Wil described, not only with auto-scaling EC2 instances at AWS but in general. We should expect fast start up times for an Enterprise level service.
33117 | February 13, 2020 05:42:22 PM GMT
Doug, I understand, the issue is the time CF takes to process the first cfm request post server startup, and not really the time it takes for the server to start up. I've tried the test cf_compiler_test-master/run_tests.cfm on a commandbox based CF. I don't see the issue. not sure what if I'm missing something in the steps in the readme doc. How are you recording the time taken to process the request. If you are going by the timestamps logged in the CF logs.. can you share the log as well. my browser records a time of ~176 ms to load cf_compiler_test-master/run_tests.cfm post startup. and ~1s if the template cache is cleared (using the test cfm, not CF admin) Below are some details from the setup I tried this on: CommandBox:ApSvrs> version CommandBox 4.8.0+00059 CommandBox:ApSvrs> server info cf2018 cf2018 (stopped) 127.0.0.1:49508 --> C:\ApSvrs\ CF Engine: adobe 2018.0.05+315699
Comment by Piyush K.
33188 | February 25, 2020 05:03:33 PM GMT
Piyush, Try using update 6 or 7 as noted above this issue seems to have appeared from update 6. Update 4 was much quicker. I tested it by simply using the browsers network timing for the first request, subsequent requests are not affected. Doug
Comment by Doug C.
33229 | March 03, 2020 09:20:46 AM GMT
Doug, The page load time mentioned in my last note was on u6 (server info command outputs "CF Engine: adobe 2018.0.05+315699", not the update level, apparently) It tried the workflow again on update 7. The page load time for cf_compiler_test-master/run_tests.cfm was ~900ms, still uder a sec. Can you confirm the following: - the start up time mentioned in the CF server logs, before you hit a CFM. - Is the page load time mentioned in your report for cf_compiler_test-master/run_tests.cfm - Do you observe the issue with any cfm. - can you observe this on a CF instance that is not commandbox based. I tried to reproduce the issue outside the context of commandBox, by generating 10k class files. restarting CF, hitting a helloworld CFM. It took 400ms to load. server startup time was 14318 ms. I'm attaching the test code I used, for your reference.
Comment by Piyush K.
33239 | March 04, 2020 09:18:47 AM GMT
I see this has been updated for HF9 - any idea what sort of timeframe that is?
Comment by Doug C.
33253 | March 06, 2020 11:34:30 AM GMT
@Piyush - you said: > I'm attaching the test code I used, for your reference. I can't see your attachment - can you make it visible please. BTW the version of ColdFusion that you are running will be shown in the Testbox HTML runner.
Comment by John W.
33254 | March 06, 2020 12:30:42 PM GMT
Doug, no timelines yet.... won't be immediate future... since its a couple of updates away. the bug tracker's a bit quirky when it comes to externalizing attachments, I reckon ... so putting in the test code at the end of this note. waiting to hear from you on the questions I put up in my last note. Thanks for the bit on HTML runner, not sure what that is though.. assure you the updates were in place.. checked the update jars in CF sys dir and admin console. an oddity, I noticed though was that the on applying the updates on CBx based CF, the older update jar were not removed from the <cfroot>/lib/updates dir.. so on applying hf7 I had cfh5, cfh6, chf7 jars in there... no anomalies were recorded on the update logs, though. --------------------- cfc_files_generator ----------------- <cfloop index = "i" from = "1" to = "10000" > <cffile action = "write" file = "#Expandpath('./classTemplates/')##i#.cfc" output = '<cfcomponent> <cffunction name="getLocalTime" access="remote"> <cfreturn TimeFormat(now())> </cffunction> <cffunction name="getUTCTime" access="remote"> <cfscript> serverTime=now(); utcTime=GetTimeZoneInfo(); utcStruct=structNew(); utcStruct.Hour=DatePart("h", serverTime); utcStruct.Minute=DatePart("n", serverTime); utcStruct.Hour=utcStruct.Hour + utcTime.utcHourOffSet; utcStruct.Minute=utcStruct.Minute + utcTime.utcMinuteOffSet; if (utcStruct.Minute LT 10) utcStruct.Minute = "0" & utcStruct.Minute; </cfscript> <cfreturn utcStruct.Hour & ":" & utcStruct.Minute> </cffunction> </cfcomponent>' addNewLine = "yes" attributes = "normal" > </cfloop> <cfoutput> Created CFC files to invoke from class_file_generator.cfm </cfoutput> ------------ class_file_generator --------------------- <cfloop index = "i" from = "1" to = "10000" > <!--- Create component instance. ---> <cfobject component="classTemplates/#i#" name="tellTimeObj"> <!--- Invoke the methods. ---> <cfinvoke component="#tellTimeObj#" method="getLocalTime" returnvariable="localTime"> <cfinvoke component="#tellTimeObj#" method="getUTCTime" returnvariable="UTCTime"> <!--- Display the results. ---> <h3>Time Display Page</h3> <cfoutput> Server's Local Time: #localTime#<br> Calculated UTC Time: #UTCTime# </cfoutput> </cfloop> <cfoutput> Invoked all files. This generates cfclass files </cfoutput>
Comment by Piyush K.
33255 | March 06, 2020 05:42:35 PM GMT
Doug, Can you confirm if the test code I shared, is a valid approach towards reproducing the issue. Do you have anything that can used on a standard CF installation, outside the context of CommandBox.
Comment by Piyush K.
33294 | March 16, 2020 09:41:18 AM GMT
Guys, I'm closing this since I haven't been able to reproduce it in either a Command Box based CF or a standalone CF. Feel free to check out the test case I shared or if you can come up with something I can try on a standalone CF please share it here or write to me at pnayak@adobe.com. For a commandbox based environment, I had some questions in my previous comment that I think went unanswered.
Comment by Piyush K.
33320 | March 23, 2020 07:08:21 AM GMT
Hi Piyush, I took your scripts cfc_files_generator.cfm and class_file_generator.cfm and added them to the repo at https://github.com/aliaspooryorik/cf_compiler_test. I generated the files using cfc_files_generator.cfm then stopped the CF server. I started up 2018 u4 hit index.cfm, cleared the template cache. Then ran class_file_generator.cfm which took 2.6 mins I started up 2018 u6 hit index.cfm, cleared the template cache. Then ran class_file_generator.cfm which took 2.3 mins However the test suite takes much longer to run on u6 than it does on u4, so not sure quite what it is but there is something being exercised in the test suite than is slower on u6. The testsuite does make use of closures which the files you generate do not so it could be that maybe? You do not need to use CommandBox at all to try running the testsuite, it's just handy for switching versions and as a package manager. We do not use commandbox at all in production where we first noticed the slow down.
Comment by John W.
33321 | March 25, 2020 10:45:51 PM GMT
We are seeing slowness as well after updating our servers past "update 4". We went directly to "update 8" so not sure which one causes it. But are noticing slowness on all requests, not just the first one. Subsequent requests may be a bit faster but still way slower than when the server was on update 4. Our server is running Mura CMS and we see the slowness on those pages. Please look into this.
Comment by Miguel F.
33322 | April 01, 2020 12:41:48 PM GMT
Miguel, since your issue is on all requests, it seems quite different than the others above. Just saying to be careful, as you problem may be quite different. And FWIW, I can add that yes, I have seen the problem with some folks saying that they have problems with Mura on CF2018. No one has been able to pin it down, and I have offered to help such folks directly to find out what's amiss but by then they have given up and either moved down to CF2016 or over to Lucee, where they don't see the problem. There can be SO many potential explanations, whether related to the code used, or perhaps unique to some aspect of configuration. You might really want to create a new ticket about it, since it is indeed different than the one here (which seems only about the first call to a request.)
Comment by Charlie A.
33323 | April 01, 2020 03:58:59 PM GMT