tracker issue : CF-3402581

select a category, or use search below
(searches all categories and all time range)
Title:

INT(value*100) produces (value*100)-1 for many values with one or two decimal places

| View in Tracker

Status/Resolution/Reason: Closed/Withdrawn/

Reporter/Name(from Bugbase): David Holsinger / David Holsinger (David Holsinger)

Created: 12/07/2012

Components: Language

Versions: 9.0.1

Failure Type:

Found In Build/Fixed In Build: 9.0.1 /

Priority/Frequency: Major / Most users will encounter

Locale/System: English / Win XP All

Vote Count: 0

Problem Description:
INT(value*100) produces (value*100)-1 for many values with one or two decimal places.

Consecutive subset of values that cause this behavior:
0.29, 0.57, 0.58, 1.13, 1.14, 1.15, 1.16, 2.01, 2.03, 2.05, 2.07, 2.26, 2.28, 2.3, 2.32, 2.51, 2.53, 2.55, 4.02, 4.06, 4.1, 4.14, 4.27, 4.31, 4.35, 4.39, 4.52, 4.56, 4.6, 4.64, 4.77, 4.81, 4.85, 4.89, 5.02, 5.06, 5.1, 8.03, 8.04, 8.12, 8.2, 8.28, 8.29, 8.37, 8.45, 8.53, 8.54, 8.62, 8.7, 8.78, 8.79, 8.87, 8.95, 9.03, 9.04, 9.12, 9.2, 9.28, 9.29, 9.37, 9.45, 9.53, 9.54, 9.62, 9.7, 9.78, 9.79, 9.87, 9.95, 10.03, 10.04, 10.12, 10.2, 16.06, 16.08, 16.15, 16.24, 16.31, 16.33, 16.4, 16.49, 16.56, 16.58, 16.65, 16.74, 16.81, 16.83, 16.9, 16.99, 17.06, 17.08...

Steps to Reproduce:

int(2.05*100)

Actual Result:
204
Expected Result:
205
Any Workarounds:
Use numberformat?

----------------------------- Additional Watson Details -----------------------------

Watson Bug ID:	3402581

Deployment Phase:	Release Candidate

External Customer Info:
External Company:  
External Customer Name: DGH
External Customer Email:  
External Test Config: My Hardware and Environment details:

Win 2000, CF 9.01

Attachments:

  1. December 07, 2012 00:00:00: 1_INT_test_code_and_values.txt

Comments:

Test code attached. The really bizarre one is: set x=16.4 [or any of these values] , set y= x*100, output int(y) and you'll get 1639. Discovered when debugging a transaction processing app that submitted values in pennies to a processing gateway. There are tens of thousands of decimal values between 1 and 100000 that cause this behavior.
Comment by External U.
16998 | December 07, 2012 08:05:52 AM GMT
I was wrong about the OS - it's actually Windows Server 2003, Standard edition.
Comment by External U.
16999 | December 08, 2012 03:25:24 PM GMT
The problem is not with the INT function but with the floating point arithmetic expression and the way floating point numbers are represented in memory. http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems If you need precision, you should use PrecisionEvaluate function. So if you use INT(PrecisionEvaluate(value*100)), you will get the result you want.
Comment by Rupesh K.
17000 | December 18, 2013 03:24:23 AM GMT