KX Community

Find answers, ask questions, and connect with our KX Community around the world.
KX Community Guidelines

Home Forums kdb+ Help Understanding Precision Re: Help Understanding Precision

  • jason_fealy

    Member
    August 25, 2021 at 12:00 am

    That number cannot be represented exactly as a floating point number i.e. it has a infinitely repeating representation (if you look at the attached pic you’ll see the binary representation is infinite). Therefore, by adjusting display precision, you’ll see different outputs. Some numbers can be represented as a floating point number e.g.

     

    q)P 7 
    q)1.125 1.126 1.125 1.126 
    q)P 0 
    q)1.125 1.126 1.125 1.1259999999999999

     

     

     

    https://www.exploringbinary.com/why-0-point-1-does-not-exist-in-floating-point/