Posted by
Knut Kvaal on
Mar 19, 2010; 11:53am
URL: http://imagej.273.s1.nabble.com/Precision-of-numbers-in-plot-and-results-tp3688902.html
Hello
I have a confusing problem when writing a plugin. When I put numbers into the results table and plot as well, the list in plot dialog shows numbers with different precision than the results.
I have in this example set decimal places to 9 and enabled Scientific Notation.
Here is what I get:
Results table 2. Column and List from Plot 3.rd column
1 1.695265623E0 1.695265651E0
2 1.741630384E0 1.741630435E0
3 1.996989389E0 1.996989369E0
4 1.756611630E0 1.756611586E0
5 1.594811512E0 1.594811559E0
6 1.329282159E0 1.329282165E0
7 1.114680459E0 1.114680409E0
8 9.434604256E-1 9.434604049E-1
9 9.000309046E-1 9.000309110E-1
10 7.652984144E-1 7.652984262E-1
Do anyone have any explanation? The correct values should be the 2. Column as I have checked by direct print.
I have no statements to set the precision an let ImageJ default rule.
Best regards
Knut Kvaal
_____________________________
Knut Kvaal, Associate Professor
Department of Mathematical Sciences and Technology
P.O.Box 5003
Norwegian University of Life Sciences (UMB) N - 1432 Ås NORWAY
Tel: +47 64965394 (Norway)
Cell: +47 90920332 (Norway)
Personal Homepage:
http://arken.umb.no/~kkvaal