If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. |
|
|
Thread Tools | Display Modes |
#1
|
|||
|
|||
Error calculation
Dear Support,
I was tryting to calculate the total variance with expected result equal to 0. But we received a very strange result. We subtract Figure1 data from Figure 2 and receive variances. After this we used Sum function to calculate the total amount and see strange number see below. While if we calculated Totals of Figure 1 and 2 and then subtract Figure 1 from Figurt 2 we receive correct result. Could you please clarify this issue. Regards, Sergey. # Figure 1 1 32400.000000000000000 2 14400.000000000000000 3 24256.400000000000000 4 57.600000000000000 5 21600.000000000000000 6 10800.000000000000000 103514.000000000000000 Figure 2 1 30960.000000000000000 2 14400.000000000000000 3 40010.000000000000000 4 144.000000000000000 5 7200.000000000000000 6 10800.000000000000000 103514.000000000000000 Variance on Totals 0.000000000000000 Variance by line (Figure 2 - Figure1) 1 -1440.000000000000000 2 0.000000000000000 3 15753.600000000000000 4 86.400000000000000 5 -14400.000000000000000 6 0.000000000000000 Sum of Variance -0.000000000012733 |
#2
|
|||
|
|||
Error calculation
ITs a binary computation/rounding problem..
http://www.tech-archive.net/Archive/.../msg01520.html TFT "Vyacheslav" wrote: Dear Support, I was tryting to calculate the total variance with expected result equal to 0. But we received a very strange result. We subtract Figure1 data from Figure 2 and receive variances. After this we used Sum function to calculate the total amount and see strange number see below. While if we calculated Totals of Figure 1 and 2 and then subtract Figure 1 from Figurt 2 we receive correct result. Could you please clarify this issue. Regards, Sergey. # Figure 1 1 32400.000000000000000 2 14400.000000000000000 3 24256.400000000000000 4 57.600000000000000 5 21600.000000000000000 6 10800.000000000000000 103514.000000000000000 Figure 2 1 30960.000000000000000 2 14400.000000000000000 3 40010.000000000000000 4 144.000000000000000 5 7200.000000000000000 6 10800.000000000000000 103514.000000000000000 Variance on Totals 0.000000000000000 Variance by line (Figure 2 - Figure1) 1 -1440.000000000000000 2 0.000000000000000 3 15753.600000000000000 4 86.400000000000000 5 -14400.000000000000000 6 0.000000000000000 Sum of Variance -0.000000000012733 |
#3
|
|||
|
|||
Error calculation
Look at
=(ROUND(fig1,2)-ROUND(fig2,2)) You will find that the issue is that 24256.4-40010 differs from -15753.6 beyond the 15th digit. As Troy indicated, his happens because the binary representation of most decimal fractions is not exact. The math is right, given the unavoidable binary approximations to the numbers. Since you are only adding an subtracting numbers with a limited number of dcimal places, the simplest solution is to round results. Jerry "Vyacheslav" wrote: Dear Support, I was tryting to calculate the total variance with expected result equal to 0. But we received a very strange result. We subtract Figure1 data from Figure 2 and receive variances. After this we used Sum function to calculate the total amount and see strange number see below. While if we calculated Totals of Figure 1 and 2 and then subtract Figure 1 from Figurt 2 we receive correct result. Could you please clarify this issue. Regards, Sergey. # Figure 1 1 32400.000000000000000 2 14400.000000000000000 3 24256.400000000000000 4 57.600000000000000 5 21600.000000000000000 6 10800.000000000000000 103514.000000000000000 Figure 2 1 30960.000000000000000 2 14400.000000000000000 3 40010.000000000000000 4 144.000000000000000 5 7200.000000000000000 6 10800.000000000000000 103514.000000000000000 Variance on Totals 0.000000000000000 Variance by line (Figure 2 - Figure1) 1 -1440.000000000000000 2 0.000000000000000 3 15753.600000000000000 4 86.400000000000000 5 -14400.000000000000000 6 0.000000000000000 Sum of Variance -0.000000000012733 |
Thread Tools | |
Display Modes | |
|
|