Some un-Real Precision?
Recently I was helping a fellow developer out on an issue. He was working with some basic code that added Real variables together. The precision for these variables was at the point 8th position of the decimal level. (ex: 99.99999999). When working with Real in X++ it is suppose to be able to handle a precision depth of 12 positions. Well as he took 99.99999999 and added it to 99.99999999 the result of the Real variable that stored the sum of these numbers showed up as: 100.
Very strange this was, and confusing, frustrating. So we took this a step further, and added another 99.99999999 to the given now 100 value variable. From this point the variable's value became 200?
After further investigation, it seems that X++ loses it's precision on Real variables after the 6th position. This can be a problem when trying to do exact percentages, but for what we were trying to do, shorting the precision to the 4th position made the sumaztion work correctly. So word to the wise, don't trust the precision of Real's in X++ past the 6th position. Maybe this will change with the next release, and until then, hopefully this will help another developer wasting hours on an issue that should not exist.
Find a job at: www.DynamicsAXJobs.com
Very strange this was, and confusing, frustrating. So we took this a step further, and added another 99.99999999 to the given now 100 value variable. From this point the variable's value became 200?
After further investigation, it seems that X++ loses it's precision on Real variables after the 6th position. This can be a problem when trying to do exact percentages, but for what we were trying to do, shorting the precision to the 4th position made the sumaztion work correctly. So word to the wise, don't trust the precision of Real's in X++ past the 6th position. Maybe this will change with the next release, and until then, hopefully this will help another developer wasting hours on an issue that should not exist.
Find a job at: www.DynamicsAXJobs.com
posted by brandon | 1:56 PM
2 Comments:
I have come across the rounding and precision issue before. I think the real problem is not the number of digits after the decimal sign, it’s the overall number of digits. Axapta handles precision up to 16 digits. I have posted a job here to demonstrate the point.
Thanks for your comment and posting axgeek!
If you would like I would love to invite you to join this blog, and let you post your code, etc. here. I like your work, it looks very good!
Post a Comment
<< Home