tracker issue : CF-3941059

select a category, or use search below
(searches all categories and all time range)
Title:

Incorrect behaviour of SerializeJSON() on strings that contain U+xxxx in CF10 Update 15

| View in Tracker

Status/Resolution/Reason: Closed/Fixed/

Reporter/Name(from Bugbase): Jonas Meller / Jonas Meller (Jonas Meller)

Created: 02/17/2015

Components: Language, Serialization

Versions: 10.0

Failure Type: Data Corruption

Found In Build/Fixed In Build: Final / CF11 Update7,CF10 Update18

Priority/Frequency: Major / Some users will encounter

Locale/System: English / Windows 7 SP1 64-bit

Vote Count: 9

Listed in the version 11.0.07.296330 Issues Fixed doc 
Problem Description:
Incorrect behaviour of SerializeJSON() on strings that contain U+xxxx in CF10 Update 15. It worked correctly in CF10 Update 13.

Steps to Reproduce:
<cfset input = "xU+a600x" />
<cfoutput>
SerializeJSON("#input#") = #SerializeJSON(input)#
DESerializeJSON(#SerializeJSON(input)#) = #DESerializeJSON(SerializeJSON(input))#
</cfoutput>

Actual Result:
SerializeJSON("xU+a600x") = "x\ua600x"
DESerializeJSON("x\ua600x") = x?x 

Expected Result:
SerializeJSON("xU+a600x") = "xU+a600x"
DESerializeJSON("xU+a600x") = xU+a600x 

Any Workarounds:
Write your own serialization code. Adobe will clearly never get it right.

----------------------------- Additional Watson Details -----------------------------

Watson Bug ID:	3941059

External Customer Info:
External Company:  
External Customer Name: Jonas
External Customer Email:  
External Test Config: My Hardware and Environment details:

Attachments:

Comments:

I can confirm this, worked in update 13.
Vote by External U.
8433 | February 17, 2015 04:59:35 AM GMT
Also can confirm this, it breaks base64 strings that get serialized. you end up with invalid base64 in your json if there is ever the sequence u+
Vote by External U.
8434 | March 04, 2015 01:18:13 PM GMT
I transfer base64 encoded image data in structures encoded as JSON strings. The improper coding of u+xxxx strings renders the image data unusable for images that randomly contain this sequence of characters.
Vote by External U.
8435 | March 17, 2015 10:07:23 AM GMT
This is an expected behavior for Unicode control characters, as there went in a fix wrt to the bug #CF-3561029 with CF10 update14 regarding the same. Hence the behavior has changed from returning "xU+a600x" to returning "x\u600x" . Also wrt to the issue with base64 strings, there has been a bug logged for this issue with the bug id #CF-3837347. Do let us know if you still have any issues with this.
Comment by S P.
8405 | March 18, 2015 04:13:01 AM GMT
The fix in #CF-3561029 is incorrect. SerializeJSON("xU+a600x") should result in "xU+a600x". SerializeJSON("x?x") should result in "x\ua600x".
Comment by External U.
8406 | March 18, 2015 04:23:56 AM GMT
Jonas, I do agree with you that the fix have broken the base64 encoded string while serializing an unicode char. But I think there is nothing wrong and we are doing it a per the spec http://www.ietf.org/rfc/rfc4627.txt (section 2.5) Any character may be escaped. If the character is in the Basic Multilingual Plane (U+0000 through U+FFFF), then it may be represented as a six-character sequence: a reverse solidus, followed by the lowercase letter u, followed by four hexadecimal digits that encode the character's code point. The hexadecimal letters A though F can be upper or lowercase. So, for example, a string containing only a single reverse solidus character may be represented as "\u005C".
Comment by Awdhesh K.
8407 | March 30, 2015 04:40:47 AM GMT
No you are not following the spec. "U+a600" is string consisting of 6 characters: * Latin Capital Letter U (U+0055) * Plus Sign (U+002B) * Latin Small Letter A (U+0061) * Digit Six (U+0036) * Digit Zero (U+0030) * Digit Zero (U+0030) It could potentially be serialized as (but please don't do this): "\u0055\u002b\u0061\u0036\u0030\u0030". "?" is string consisting of 1 characters: * Vai Syllable Je (U+A600) If serialized the result could be "\ua600".
Comment by External U.
8408 | March 30, 2015 05:48:38 AM GMT
[subscribe]
Comment by External U.
8409 | March 30, 2015 07:32:33 AM GMT
Same error with CF 10 Update 14 or CF 10 Update 15. Please fix this!
Vote by External U.
8436 | March 30, 2015 08:17:57 AM GMT
Please look at this implementation from the creator of JSON: https://github.com/douglascrockford/JSON-java/blob/master/JSONObject.java#L1194-L1273
Comment by External U.
8410 | March 31, 2015 12:57:42 AM GMT
I created some unit tests: https://github.com/rellem/cfml-test-suite/blob/master/cfmltestsuite/tests/cfml/functions/SerializeJSON_StringsTest.cfc
Comment by External U.
8411 | April 05, 2015 10:25:43 AM GMT
This appears to be the same issue causing multiple serialization issues in ColdFusion Build 14 and later, as well as CF11. Please see bug report CF-3837347 as well. We just spent DAYS trying to track down what was causing our caching to fail on random pages only to find out that it's a bug that Adobe has known about and yet not bothered to fix. This "excuse" that it is "per spec" is ridiculous, you need to just FIX THE PROBLEM.
Comment by External U.
8412 | May 12, 2015 11:38:56 AM GMT
Awdhesh, you are reading the spec wrong. When they say this: "Any character may be escaped. If the character is in the Basic Multilingual Plane (U+0000 through U+FFFF) [etc]" they are talking about *individual characters*. The characters they're talking about there are represented there in plain text as U+0000 etc. They're not saying any *string* that is actually - literally! - "U+a600" that is a six character string, comprising characters that *don't need to be encoded*. This is the difference: A character represented by U+C000: "?" A string that is U+C000: "U+C000" (from http://la.remifa.so/unicode/unicode.php?start=C000&end=CFFF) You've messed it up. Just fix it, pls, and stop messing your clients around. Cheers. -- Adam
Comment by External U.
8413 | May 12, 2015 12:20:39 PM GMT
+1 This should be more than 25 characters.
Vote by External U.
8437 | May 12, 2015 04:12:05 PM GMT
When will Adobe finally fix this bug?
Comment by External U.
8414 | June 15, 2015 05:25:51 AM GMT
+1 =========================
Vote by External U.
8438 | June 29, 2015 11:28:02 PM GMT
Awdhesh?
Comment by External U.
8415 | June 30, 2015 12:49:41 AM GMT
Please fix, it's terrible to have to hack my code when handling Base64 strings of images because the spec is being incorrectly interpreted.
Vote by External U.
8439 | August 13, 2015 09:57:12 AM GMT
Please be sure to review the history of this duplicate. It has some test cases in it. https://bugbase.adobe.com/index.cfm?event=bug&id=CF-3837347
Vote by External U.
8440 | August 18, 2015 09:00:25 AM GMT
I see history in https://bugbase.adobe.com/index.cfm?event=bug&id=CF-3837347 that indicates a dupe. But there is much more info there than here. I see the status is "ToFix". Do we have a target date?
Comment by External U.
8416 | August 18, 2015 09:01:37 AM GMT
The latest comment on the other bug is that they are evaluating a "possible" fix. Not exactly a comment to engender much faith in their getting this fixed soon.
Comment by External U.
8417 | August 19, 2015 01:23:32 AM GMT
Hi, will fixing in soon.
Comment by Awdhesh K.
8418 | August 20, 2015 03:03:52 AM GMT
The fix will be made available as part of the next update.
Comment by Vamseekrishna N.
8419 | August 20, 2015 03:21:04 AM GMT
The fix will be in both CF 10 and CF 11?
Comment by External U.
8420 | August 20, 2015 09:48:38 AM GMT
Yes
Comment by Vamseekrishna N.
8421 | August 20, 2015 09:27:54 PM GMT
How to get the Latest version of flash Player so I can play my games on Facebook
Comment by External U.
8422 | September 14, 2015 11:11:42 AM GMT
The fix will be available in the next ColdFusion version. Thanks!
Comment by S P.
8423 | September 21, 2015 08:02:54 AM GMT
S Preethi- can you clarify "next ColdFusion version" please? Do you mean CF12? If so, that directly contradicts the assurance given by Vamseekrishna Nanneboina on August 20th 2015 that the fix would be available in an update to CF10 and CF11.
Comment by External U.
8424 | September 21, 2015 03:31:39 PM GMT
Sorry for the double comment. Glitch on submitting from smartphone.
Comment by External U.
8425 | September 21, 2015 03:32:33 PM GMT
The fix would indeed be there in ColdFusion 10 and 11 update as well.
Comment by Rupesh K.
8426 | September 22, 2015 12:08:02 AM GMT
Rakshith - thank you for clarifying that.
Comment by External U.
8427 | September 22, 2015 08:55:15 AM GMT
OOPS. Sorry again. *Thanks Rupesh*!
Comment by External U.
8428 | September 22, 2015 08:56:05 AM GMT
+1 Same with ColdFusion 10 Update 16
Vote by External U.
8441 | November 10, 2015 10:36:42 AM GMT
Last week I updated to CF10 update17 and the problem was fixes. This morning I updated to CF10 Update 18 and I'm now getting Java Heap Space error on serializeJson.
Comment by External U.
8429 | November 18, 2015 03:57:28 AM GMT
@Jork, could you pls post a gist URL of a minimal stand-alone repro case for this?
Comment by External U.
8430 | November 18, 2015 04:04:45 AM GMT
@Adam, I went back an forth the update 17 and 18, so that I could create the case. Strangely enough the last time I went to 18 it seems to work better. It appears to me that the problem is just the sheer size of the data I'm serializing and having concurrent/multiple requests. So it appear to me it has nothing to do with this bug, but it a different issue. https://gist.github.com/jorkzijlstra/bb6cDVAPR-18362345b926a1a3
Comment by External U.
8431 | November 18, 2015 06:57:33 AM GMT
Verified this is fixed in CF11 Update 7 (build 11,0,07,296330). Thanks!, -Aaron
Comment by External U.
8432 | December 03, 2015 06:47:17 PM GMT