Re: Precision of numbers in plot and results

Posted by Michael Schmid on Mar 19, 2010; 12:37pm
URL: http://imagej.273.s1.nabble.com/Precision-of-numbers-in-plot-and-results-tp3688902p3688903.html

Hi Knut,

ImageJ has no image data type with double precision (approx. 14  
digit) accuracy, only float (approx. 7 digits) - at least for  
standard microscopy and light imaging techniques, there are no image  
sensors that would require more than float accuracy. Thus, also  
functions like profile plot use float accuracy.
The Results table has full double precision accuracy.

Michael
________________________________________________________________

On 19 Mar 2010, at 12:53, Knut Kvaal wrote:

> Hello
> I have a confusing problem when writing a plugin. When I put  
> numbers into the results table and plot as well, the list in plot  
> dialog shows numbers with different precision than the results.
> I have  in this example set decimal places to 9 and enabled  
> Scientific Notation.
> Here is what I get:
> Results table 2. Column and List from Plot 3.rd column
> 1    1.695265623E0 1.695265651E0
> 2    1.741630384E0 1.741630435E0
> 3    1.996989389E0 1.996989369E0
> 4    1.756611630E0  1.756611586E0
> 5    1.594811512E0  1.594811559E0
> 6    1.329282159E0 1.329282165E0
> 7    1.114680459E0 1.114680409E0
> 8    9.434604256E-1 9.434604049E-1
> 9    9.000309046E-1 9.000309110E-1
> 10    7.652984144E-1  7.652984262E-1
>
> Do anyone have any explanation? The correct values should be the 2.  
> Column as I have checked by direct print.
> I have no statements to set the precision an let ImageJ default rule.
>
> Best regards
> Knut Kvaal
>
> _____________________________
> Knut Kvaal, Associate Professor
> Department of Mathematical Sciences and Technology
> P.O.Box 5003
> Norwegian University of Life Sciences (UMB) N - 1432 Ås NORWAY
>
> Tel:  [phone] (Norway)
> Cell: [phone] (Norway)
>
> Personal Homepage: http://arken.umb.no/~kkvaal