tracker issue : CF-3750467

select a category, or use search below
(searches all categories and all time range)
Title:

ORM Indexing of Large Recordsets Fail to Complete

| View in Tracker

Status/Resolution/Reason: Closed/Withdrawn/NotABug

Reporter/Name(from Bugbase): Jonathan Clausen / Jonathan Clausen (Jonathan Clausen)

Created: 04/25/2014

Components: ORM Support

Versions: 10.0

Failure Type: Crash

Found In Build/Fixed In Build: Final /

Priority/Frequency: Critical / Most users will encounter

Locale/System: English / Platforms All

Vote Count: 0

Problem Description:  When attempting to index a persistent entity with joined properties with greater than 10,000 records, the function ormIndex('EntityName') hangs the server, even if there is only one property in the entity which is being indexed

Steps to Reproduce:

1) Create persistent CFC associated with table of greater than 10,000 records with join relationships to other components.

2) set all properties as indexable=false except for one

3) Run ormIndex('EntityName') on component

Actual Result:
Server response with java.lang.OutOfMemoryError: GC overhead limit exceeded

Expected Result:
Index of persistent entity

Any Workarounds:

Looping the records in small chunks will succeed, however when attemping to loop the full collection, the server crashes with:  pipe error: Too many open files in system errors

----------------------------- Additional Watson Details -----------------------------

Watson Bug ID:	3750467

External Customer Info:
External Company:  
External Customer Name: JJClausen
External Customer Email:  
External Test Config: My Hardware and Environment details:



OSX:

OSX v 10.9.1

Proc: 2.9 GHz Intel Core i7

Memory: 8GB 



PostgreSQL:

9.2.4 on x86_64-apple-darwin-12.5.0



CF Server:

Server Details

Server Product	 ColdFusion

Version	 ColdFusion 10,287689

Edition	 Developer 

Operating System	 Mac OS X 

OS Version	 10.9.1 

Update Level	 /Applications/ColdFusion10/cfusion/lib/updates/chf10000013.jar 

Adobe Driver Version	 4.1 (Build 0001) 





JVM Details

Java Version	 1.7.0_45 

Java Vendor	 Oracle Corporation 

Java Vendor URL	http://java.oracle.com/ 

Java Home	 /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/jre 

Java File Encoding	 UTF8 

Java Default Locale	 en_US 

File Separator	 / 

Path Separator	 : 

Line Separator	 Chr(10) 

User Name	 jonclausen 

User Home	 /Users/username 

User Dir	 /Applications/ColdFusion10/cfusion/bin 

Java VM Specification Version	 1.7 

Java VM Specification Vendor	 Oracle Corporation 

Java VM Specification Name	 Java Virtual Machine Specification 

Java VM Version	 24.45-b08 

Java VM Vendor	 Oracle Corporation 

Java VM Name	 Java HotSpot(TM) 64-Bit Server VM 

Java Specification Version	 1.7 

Java Specification Vendor	 Oracle Corporation 

Java Specification Name	 Java Platform API Specification 

Java Class Version	 51.0



JVM Config:

Min Heap: 512MB

Max Heap: 1024MB

JVM Args:

-XX:MaxPermSize=1024m -XX:+UseParallelGC -XX:+DisableExplicitGC -Xbatch -Dcoldfusion.home={application.home} -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random

Attachments:

Comments:

Feel free to email for example code.
Comment by External U.
12550 | April 25, 2014 04:41:18 PM GMT
http://wiki.apache.org/lucene-java/LuceneFAQ#Why_am_I_getting_an_IOException_that_says_.22Too_many_open_files.22.3F Must be related.
Comment by Himavanth R.
12551 | May 09, 2014 04:06:56 AM GMT
I tried reproducing the issue with a persistent entity referring to DB table having around 30k records, but did not get any exception. Table "testEmployees" has 30k records join to the table "address" cemployees.cfc: <cfcomponent persistent="true" table="testEmployees" indexable="true"> <cfproperty name="employee_id" fieldtype="id" generator="native" indexable="false"> <cfproperty name="lastname" indexable="true"> <cfproperty name="firstname" indexable="false"> <cfproperty name="title" indexable="false"> <cfproperty name="titleOfCourtesy" indexable="false"> <cfproperty name="address_id" fieldtype="one-to-one" cfc="caddress" fkcolumn="address_id" indexable="false"> </cfcomponent> caddress.cfc <cfcomponent persistent="true" table="address"> <cfproperty name="address_id" fieldType="id"> <cfproperty name="Line1"> <cfproperty name="Line2"> <cfproperty name="City"> </cfcomponent> test.cfm <cfscript> ormindex("cemployees"); </cfscript> Executing test.cfm works as expected without any exception. (Comment added from ex-user id:nawani)
Comment by Adobe D.
12552 | December 09, 2014 08:00:41 AM GMT
Hi Jonathan, We are closing this bug as we have not received any information. Please feel free to write back to us if you are still experiencing this issue and we will re-open this bug.
Comment by Nimit S.
12553 | August 03, 2015 08:40:18 AM GMT
Weird. I didn't receive email notifications on your comments. My apologies. The issue is present in CF11 (as well as Railo/Lucee), but seems to be related to the PostgreSQL JDBC driver and connection pooling. Increasing the max allowed connections will help up to a certain number of records. MySQL doesn't exhibit the same behavior. My workaround has been to create the indexes manually and set index=false on all persistent properties. Closing the is probably the right call on your end, as the bug should be submitted to the driver development team.
Comment by External U.
12554 | August 03, 2015 09:20:44 AM GMT
Thank you so much Jonathan for confirming this. Please let us know if there is any assistance required from our side.
Comment by Nimit S.
12555 | August 03, 2015 09:33:54 AM GMT