-
Help Understanding Precision
Can you please help me understand the below and why that number is converting to 99999 at end
q)P 0
q)79347181.71
79347181.709999993q)P 16
q)79347181.71
79347181.70999999q)P 15
q)79347181.71
79347181.71 -
1 Reply
-
That number cannot be represented exactly as a floating point number i.e. it has a infinitely repeating representation (if you look at the attached pic you’ll see the binary representation is infinite). Therefore, by adjusting display precision, you’ll see different outputs. Some numbers can be represented as a floating point number e.g.
q)P 7 q)1.125 1.126 1.125 1.126 q)P 0 q)1.125 1.126 1.125 1.1259999999999999
https://www.exploringbinary.com/why-0-point-1-does-not-exist-in-floating-point/
Log in to reply.