A Microsoft Office (Excel, Word) forum. OfficeFrustration

If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.

Go Back   Home » OfficeFrustration forum » Microsoft Excel » General Discussion
Site Map Home Register Authors List Search Today's Posts Mark Forums Read  

Error calculation



 
 
Thread Tools Display Modes
  #1  
Old November 12th, 2009, 07:21 AM posted to microsoft.public.excel.misc
Vyacheslav
external usenet poster
 
Posts: 1
Default Error calculation

Dear Support,
I was tryting to calculate the total variance with expected result equal to
0. But we received a very strange result. We subtract Figure1 data from
Figure 2 and receive variances. After this we used Sum function to calculate
the total amount and see strange number see below. While if we calculated
Totals of Figure 1 and 2 and then subtract Figure 1 from Figurt 2 we receive
correct result. Could you please clarify this issue. Regards, Sergey.
# Figure 1
1 32400.000000000000000
2 14400.000000000000000
3 24256.400000000000000
4 57.600000000000000
5 21600.000000000000000
6 10800.000000000000000
103514.000000000000000


Figure 2
1 30960.000000000000000
2 14400.000000000000000
3 40010.000000000000000
4 144.000000000000000
5 7200.000000000000000
6 10800.000000000000000
103514.000000000000000

Variance on Totals 0.000000000000000

Variance by line (Figure 2 - Figure1)
1 -1440.000000000000000
2 0.000000000000000
3 15753.600000000000000
4 86.400000000000000
5 -14400.000000000000000
6 0.000000000000000

Sum of Variance -0.000000000012733

  #2  
Old November 12th, 2009, 08:00 AM posted to microsoft.public.excel.misc
TroyT
external usenet poster
 
Posts: 4
Default Error calculation

ITs a binary computation/rounding problem..

http://www.tech-archive.net/Archive/.../msg01520.html

TFT

"Vyacheslav" wrote:

Dear Support,
I was tryting to calculate the total variance with expected result equal to
0. But we received a very strange result. We subtract Figure1 data from
Figure 2 and receive variances. After this we used Sum function to calculate
the total amount and see strange number see below. While if we calculated
Totals of Figure 1 and 2 and then subtract Figure 1 from Figurt 2 we receive
correct result. Could you please clarify this issue. Regards, Sergey.
# Figure 1
1 32400.000000000000000
2 14400.000000000000000
3 24256.400000000000000
4 57.600000000000000
5 21600.000000000000000
6 10800.000000000000000
103514.000000000000000


Figure 2
1 30960.000000000000000
2 14400.000000000000000
3 40010.000000000000000
4 144.000000000000000
5 7200.000000000000000
6 10800.000000000000000
103514.000000000000000

Variance on Totals 0.000000000000000

Variance by line (Figure 2 - Figure1)
1 -1440.000000000000000
2 0.000000000000000
3 15753.600000000000000
4 86.400000000000000
5 -14400.000000000000000
6 0.000000000000000

Sum of Variance -0.000000000012733

  #3  
Old November 14th, 2009, 01:56 PM posted to microsoft.public.excel.misc
Jerry W. Lewis
external usenet poster
 
Posts: 491
Default Error calculation

Look at
=(ROUND(fig1,2)-ROUND(fig2,2))
You will find that the issue is that 24256.4-40010 differs from -15753.6
beyond the 15th digit. As Troy indicated, his happens because the binary
representation of most decimal fractions is not exact. The math is right,
given the unavoidable binary approximations to the numbers.

Since you are only adding an subtracting numbers with a limited number of
dcimal places, the simplest solution is to round results.

Jerry

"Vyacheslav" wrote:

Dear Support,
I was tryting to calculate the total variance with expected result equal to
0. But we received a very strange result. We subtract Figure1 data from
Figure 2 and receive variances. After this we used Sum function to calculate
the total amount and see strange number see below. While if we calculated
Totals of Figure 1 and 2 and then subtract Figure 1 from Figurt 2 we receive
correct result. Could you please clarify this issue. Regards, Sergey.
# Figure 1
1 32400.000000000000000
2 14400.000000000000000
3 24256.400000000000000
4 57.600000000000000
5 21600.000000000000000
6 10800.000000000000000
103514.000000000000000


Figure 2
1 30960.000000000000000
2 14400.000000000000000
3 40010.000000000000000
4 144.000000000000000
5 7200.000000000000000
6 10800.000000000000000
103514.000000000000000

Variance on Totals 0.000000000000000

Variance by line (Figure 2 - Figure1)
1 -1440.000000000000000
2 0.000000000000000
3 15753.600000000000000
4 86.400000000000000
5 -14400.000000000000000
6 0.000000000000000

Sum of Variance -0.000000000012733

 




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump


All times are GMT +1. The time now is 01:47 PM.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 OfficeFrustration.
The comments are property of their posters.