If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. 


Thread Tools  Display Modes 
#1




Fill/Extend Series adding decimal points to some numbers
Hi,
I set up a worksheet for a Lookup Function that contains numbers increasing by 0.1 from 100.0 to 50.0. I used the drag and fill to set up the numbers. I noticed some incorrect entries in the results from the look up and found that on certain numbers it added many decimal spots to the number for example 66.5 look like 66.5000000000001 when I click into the cell. I can not locate a pattern for why this is occurring (i.e. does not occur in the 48 or 1015 but does in the 9 and 16 series). Formatting is set to numerical with 1 decimal point. Any help? 
Ads 
#2




Fill/Extend Series adding decimal points to some numbers
This is a limitation of any computer that uses binary representation
of decimal numbers  some fractions cannot be represented with complete accuracy. A binary fraction is made up of digits to represent 1/2, 1/4, 1/8, 1/16 etc, but some simple decimal fractions (like 1/10 or 0.1) cannot be represented completely in binary as they are recurring numbers (rather like 1/3 or 2/3 in decimal). Another thing to consider is that Excel can only represent up to 15 digits of precision, so a recurring fraction will be truncated to this precision and when it is added repeatedly then errors will accumulate. I'm sure Joe will come along soon to give a more indepth explanation. Hope this helps. Pete On Apr 26, 4:44*pm, awalker wrote: Hi, I set up a worksheet for a Lookup Function that contains numbers increasing by 0.1 from 100.0 to 50.0. I used the drag and fill to set up the numbers.. * * I noticed some incorrect entries in the results from the look up and found that on certain numbers it added many decimal spots to the number for example 66.5 look like 66.5000000000001 when I click into the cell. *I can not locate a pattern for why this is occurring (i.e. does not occur in the 48 or 1015 but does in the 9 and 16 series). *Formatting is set to numerical with 1 decimal point. * Any help? 
Thread Tools  
Display Modes  

