Title:
JVM 1.7 corrupts XML objects resulting in StackOverFlows when performing XmlSearch()
| View in TrackerStatus/Resolution/Reason: Closed/Withdrawn/Duplicate
Reporter/Name(from Bugbase): Brian Ghidinelli / Brian Ghidinelli (Brian Ghidinelli)
Created: 04/22/2013
Components: Language
Versions: 10.0
Failure Type: Crash
Found In Build/Fixed In Build: Final /
Priority/Frequency: Trivial / Unknown
Locale/System: English / Linux RH Enterprise 6
Vote Count: 8
Duplicate ID: CF-3739102
Problem Description:
You can read my full process here: https://groups.google.com/forum/?fromgroups=#!topic/transfer-dev/82Nx3lbNEyE
I am in the process of trying to migrate from CF8 enterprise to CF10 enterprise. I have a large Model-Glue/Transfer ORM/Coldspring application. All of the frameworks are persisted in the Application scope. After the application would run for some period of time (hours generally), under almost zero load (1 or 2 users in a pre-deployment test environment), xmlSearch()s run via Transfer would begin failing and leading to StackOverFlow errors ultimately leading to total application failure.
Steps to Reproduce:
I do not have a simple test case nor a repeatable method. However, the commonality is that Transfer would load from a file an XML object. Each time you request an object from Transfer, it performs an xmlSearch() to obtain the definition of the object. These searches would suddenly stop working and begin failing on certain nodes in the XML leading to DOM walker errors and endless loops.
Actual Result:
Lots of never-before-seen errors (and ones that don't turn up in Google, which is a bad sign):
Stack Trace
java.lang.StackOverflowError at net.sf.saxon.serialize.XMLEmitter.writeCharSequence(XMLEmitter.java:652) at net.sf.saxon.serialize.XMLEmitter.writeEscape(XMLEmitter.java:745) at net.sf.saxon.serialize.XMLEmitter.characters(XMLEmitter.java:591) at net.sf.saxon.serialize.UncommittedSerializer.characters(UncommittedSerializer.java:67) at net.sf.saxon.event.ProxyReceiver.characters(ProxyReceiver.java:186) at net.sf.saxon.dom.DOMSender.walkNode(DOMSender.java:172) at net.sf.saxon.dom.DOMSender.outputElement(DOMSender.java:235) at net.sf.saxon.dom.DOMSender.walkNode(DOMSender.java:150) at net.sf.saxon.dom.DOMSender.outputElement(DOMSender.java:235) at net.sf.saxon.dom.DOMSender.walkNode(DOMSender.java:150) at (And the walkNode just repeats to infinity and the request fails.)
java.lang.StackOverflowError at org.apache.xerces.dom.ParentNode.item(Unknown Source) at net.sf.saxon.dom.NodeWrapper$ChildEnumeration.skipFollowingTextNodes(NodeWrapper.java:1166) at net.sf.saxon.dom.NodeWrapper$ChildEnumeration.next(NodeWrapper.java:1194) at net.sf.saxon.tree.util.Navigator$EmptyTextFilter.next(Navigator.java:918) at net.sf.saxon.tree.util.Navigator$DescendantEnumeration.advance(Navigator.java:1052) at net.sf.saxon.tree.util.Navigator$DescendantEnumeration.advance(Navigator.java:1111) at net.sf.saxon.tree.util.Navigator$BaseEnumeration.next(Navigator.java:949) at net.sf.saxon.tree.util.Navigator$DescendantEnumeration.advance(Navigator.java:1043) at (which repeats until the end)
Most importantly, I was able to narrow it down to xmlSearch() by doing some brute force debugging taking stack traces at 20 and 50ms intervals while this was blowing up. I captured the Saxon parser getting stuck in an endless loop in what is a perfectly valid XML file. Keep in mind, this Transfer app has been in production for 5 years. It's currently in production on CF8 Enterprise/64-bit linux, it's in development on CF8 and CF10 Windows and I have it working just fine in a pre-production test bed on CF10 Enterprise/64-bit linux with JDK 1.6.
Expected Result:
xmlSearch() should be able to successfully search the XML object persisted in memory and return a consistent empty or valid result.
Any Workarounds:
Downgrading to the EOL Java 1.6 (u43, specifically) has eliminated the problem.
Alternatively, restart the server every time it happens and wait.
I was able to mitigate this partially by using try/catch when an entity was not found in the XML file and then on-the-fly reload and overwrite the XML object in memory with a fresh copy. This would work for awhile until the pattern would repeat.
----------------------------- Additional Watson Details -----------------------------
Watson Bug ID: 3546959
External Customer Info:
External Company:
External Customer Name: brianatvfive
External Customer Email:
External Test Config: My Hardware and Environment details:
ColdFusion 10 Enterprise on 64-bit RHEL 6, patched up to Update 8 (at time of bug).
JDK 1.7 with errors, JDK 1.6 u43 withoutAttachments:
- April 23, 2013 00:00:00: 1_xmlsearch-blowup-stacktrace.txt
Comments: