Questions tagged [floating-point]
5 questions
3
votes
1 answer
Excel rounding errors
In a new worksheet (I'm using Excel 2013), starting in the top left, I type:
A
B
C
=18.6 + 67.2
=A1-85.8 = 0
=(A1-85.8) * 1000000000000
Expectation
A
B
C
85.8
TRUE
0
Actual Result
A
B
C
85.8
FALSE
0.014210855
I'm…
Bob Sammers
- 206
- 2
- 8
2
votes
1 answer
Excel floating point and boolean unexplained behavior
Thanks for checking out my question.
I was putzing around in Excel today for work, and encountered some anomalous behavior involving Excel's floating-point arithmetic and boolean tests. Here is a link to the Excel file test case I made.
In order…
Patrick
- 121
- 2
1
vote
0 answers
PostgreSQL sum function resurns different result when run multiple of times with psql
I have this query :
SELECT SUM(median) FROM my_table WHERE id = 100
-- median is of type float4
When I run from psql, it returns slightly different result for multiple of runs (For example 1000.08, 1001.02, ...)
The number of entries contributing…
Philippe
- 125
- 5
0
votes
4 answers
Excel modulus function MOD(0.9, 0.3) returns unexpected result
MOD function in Excel should return the Modulo Remainder in Excel
I believe the function =MOD(0.9, 0.3) should give 0 since basically 0.3*3 is 0.9
But it gives 5.55112E-17
Any hints to what's wrong?
0
votes
2 answers
Excel not parsing very small floating point numbers from text to value
From a calculation I get results in text form that I want to paste into Excel.
Some of the numbers are absolute small, down to the minimum IEEE 754 double value larger zero, 2^-1074, e.g. about 10^-324.
When I paste this into Excel, Excel only…
Jens
- 111
- 5