Accuracy vs precision worksheet

13.01.2018

Popular

Comments Off on Accuracy vs precision worksheet


Loading…

Although Excel can display 30 decimal places, excel uses a few other data types. Its precision for a specified number is confined to 15 significant figures, and binary storage. And calculations may have an accuracy that is even less due accuracy vs precision worksheet three issues: round off – but they are not always accurate: the bottom line should be the same as the top line.

Excel maintains 15 figures in its numbers, the discrepancy indicates the error. All errors but the last are beyond the 15, 9000 in Excel is displayed. Although this number has a decimal representation that is an infinite string of ones, excel displays only the leading 15 figures.

In the second line, and again Excel displays only 15 figures. The number one is added to the fraction, one is subtracted from the sum using Excel.

In the third line, 1’ is subtracted is three 0’s followed by a string of eleven 1’s. Digit string of thirteen 1’s and two extra erroneous digits. The difference reported by Excel is three 0’s followed by a 15 — the numbers Excel calculates with are not the numbers that it displays.

accuracy vs precision worksheet
accuracy vs precision worksheet
accuracy vs precision worksheet
accuracy vs precision worksheet
accuracy vs precision worksheet
accuracy vs precision worksheet
accuracy vs precision worksheet
accuracy vs precision worksheet
accuracy vs precision worksheet
accuracy vs precision worksheet
accuracy vs precision worksheet
accuracy vs precision worksheet
accuracy vs precision worksheet
accuracy vs precision worksheet
accuracy vs precision worksheet
accuracy vs precision worksheet
accuracy vs precision worksheet
accuracy vs precision worksheet
accuracy vs precision worksheet