set null values
Locked
 
|
I have images with outliers that should be set to null and not taken into
account for further processing. How do I set these values to null? For example, I want to set as null values lower than 0 and higher than 2000. Also, once I have them defined as null, would any processing ignore these values? In particular, if I use an FFT band filter, I do not want these values to expand from isolated pixels to little patches (as occurs now). An alternative would be to guess the values for the pixels that have been set as null as the median from the 3x3 surrounding window and the filter out using FFT. Thanks! Agus -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
|
Hi Agus,
FFT does not ignore NaN pixels (it does not work at all with images containing NaNs). Two possiblities: Process>Noise>Remove Outliers (replaces everything that deviates from the median of the surrounding by the median) Threshold the image to select the outliers, creating a selection (Edit>Selection>Create Selection) with the outliers and running a median thereafter - it will affect the selected pixels only. If you want to have NaNs for other purposes (e.g., 'Measure' ignores NaN pixels), threshold the image, create a selection and use Process>Math>Set to NaN. Michael ________________________________________________________________ On Apr 5, 2013, at 09:45, Agustin Lobo wrote: > I have images with outliers that should be set to null and not taken into > account for further processing. How do I set these values to null? > For example, I want to set as null values lower than 0 > and higher than 2000. > Also, once I have them defined as null, would any processing ignore > these values? > In particular, if I use an FFT band filter, I do not want these values > to expand from > isolated pixels to little patches (as occurs now). > An alternative would be to guess the values for the pixels > that have been set as null as the median from the 3x3 surrounding window > and the filter out using FFT. > > Thanks! > > Agus > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
|
Thanks, but I need a better way to define outliers. I think that I can create
a macro to calculate mean and sd (or mad) and define outliers a outside mean+/- x*sd with a variable value of x (i.e. 2, 2.5, 3 etc). Then calculate the actual value in DN and use the Remove outliers function. Does anyone know if this has already been implemented? Agus On Fri, Apr 5, 2013 at 1:44 PM, Michael Schmid <[hidden email]> wrote: > Hi Agus, > > FFT does not ignore NaN pixels (it does not work at all with images containing NaNs). > > Two possiblities: > Process>Noise>Remove Outliers (replaces everything that deviates from the median of the surrounding by the median) > > Threshold the image to select the outliers, creating a selection (Edit>Selection>Create Selection) with the outliers and running a median thereafter - it will affect the selected pixels only. > > If you want to have NaNs for other purposes (e.g., 'Measure' ignores NaN pixels), threshold the image, create a selection and use Process>Math>Set to NaN. > > Michael > ________________________________________________________________ > On Apr 5, 2013, at 09:45, Agustin Lobo wrote: > >> I have images with outliers that should be set to null and not taken into >> account for further processing. How do I set these values to null? >> For example, I want to set as null values lower than 0 >> and higher than 2000. >> Also, once I have them defined as null, would any processing ignore >> these values? >> In particular, if I use an FFT band filter, I do not want these values >> to expand from >> isolated pixels to little patches (as occurs now). >> An alternative would be to guess the values for the pixels >> that have been set as null as the median from the 3x3 surrounding window >> and the filter out using FFT. >> >> Thanks! >> >> Agus >> >> -- >> ImageJ mailing list: http://imagej.nih.gov/ij/list.html > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
|
Augustin,
in Fiji, you can find an Outlier removal plugin under Plugins > Integral Image Filters The plugin detects outliers by calculating mean and SD for each pixel in a block of some given radius as those beyond mean +- x*SD, just as you suggested. Outlier pixels are replaced by smearing in their non-outlier neighbors. As the plugins use integral images to calculate block mean and SD, they are extraordinarily fast, independent of the block-radius. In case you are not using Fiji, find the plugins jar here: http://fly.mpi-cbg.de/saalfeld/download/integral_image.jar Best, Stephan On Fri, 2013-04-12 at 12:56 +0200, Agustin Lobo wrote: > Thanks, but I need a better way to define outliers. I think that I can create > a macro to calculate mean and sd (or mad) and define outliers a > outside mean+/- x*sd > with a variable value of x (i.e. 2, 2.5, 3 etc). Then calculate the > actual value in DN > and use the Remove outliers function. > Does anyone know if this has already been implemented? > > Agus > > On Fri, Apr 5, 2013 at 1:44 PM, Michael Schmid <[hidden email]> wrote: > > Hi Agus, > > > > FFT does not ignore NaN pixels (it does not work at all with images containing NaNs). > > > > Two possiblities: > > Process>Noise>Remove Outliers (replaces everything that deviates from the median of the surrounding by the median) > > > > Threshold the image to select the outliers, creating a selection (Edit>Selection>Create Selection) with the outliers and running a median thereafter - it will affect the selected pixels only. > > > > If you want to have NaNs for other purposes (e.g., 'Measure' ignores NaN pixels), threshold the image, create a selection and use Process>Math>Set to NaN. > > > > Michael > > ________________________________________________________________ > > On Apr 5, 2013, at 09:45, Agustin Lobo wrote: > > > >> I have images with outliers that should be set to null and not taken into > >> account for further processing. How do I set these values to null? > >> For example, I want to set as null values lower than 0 > >> and higher than 2000. > >> Also, once I have them defined as null, would any processing ignore > >> these values? > >> In particular, if I use an FFT band filter, I do not want these values > >> to expand from > >> isolated pixels to little patches (as occurs now). > >> An alternative would be to guess the values for the pixels > >> that have been set as null as the median from the 3x3 surrounding window > >> and the filter out using FFT. > >> > >> Thanks! > >> > >> Agus > >> > >> -- > >> ImageJ mailing list: http://imagej.nih.gov/ij/list.html > > > > -- > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
|
Hi Agus,
it seems that I haven't replied to this---sorry for the delay... Great to hear that the plugin has been of some use for you. To your questions: The plugin implements outlier removal based on local block statistics. The purpose of the plugin is to do that extremely fast by exploiting that particular statistics can be calculated with the help of integral images. That, however, naturally limits it to what you can do with integral images efficiently, e.g. mean and std but not median. NaN-values cannot be accumulated and thus the plugin will fail in their presence. The effect that you're observing is due to the plugin internally setting outlier detections to NaN and then replacing the NaN-values. If there were NaN values initially, all outlier tests will be negative and the existing NaNs will be replaced. One could fix that as you did it now, by filling NaNs initially and then processing as usual, I may do that as soon as I find the time. To the suggestions: 1. The plugin uses the radius that you set for detecting outliers only. The value is calculated from the outlier pixel's directly adjacent pixels, that is the effective radius for replacing values is 1. This is the intended behavior 2. Median cannot be calculated efficiently with integral images and is thus not target of this plugin. Use something else instead, e.g. ImageJ's median filter. Best, Stephan On Mon, 2013-09-09 at 20:18 +0200, Agustin Lobo wrote: > Dear Stephan, > > I've been using your plugin Outlier Removal and I've found that > in case there are NaN values in the image your plugin only correct those values, > regardless the size of the window and the factor of the SD. This > happens even on those > zones of the image for which there are no NaN values within the defined radius. > If NaN values are previously eliminated (i.e, with > Process/Noise/Remove NaN), your plugin works > as expected. This is no problem, just wanted to confirm this behavior > is in accordance with your intended design. > > I also would like to make a couple of suggestions: > 1. Two different radii would be an advantage. Normally, you want a > relatively large radius to detect outliers > but an smaller radius to calculate the value to be set in lieu of the outlier. > 2. Using the mad (median absolute values) instead of the sd would be a > better estimate of the real sd (the one in absence > of outliers). As outliers are included in the sd calculation, the > calculated value is easily inflated by the presence of the outlier > itself > and it can thus be left undetected. mad is less prone to exaggeration. > > Thanks a lot for your plugin, > > Agus > > On Fri, Apr 12, 2013 at 2:59 PM, Stephan Saalfeld <[hidden email]> wrote: > > Augustin, > > > > in Fiji, you can find an Outlier removal plugin under > > > > Plugins > Integral Image Filters > > > > The plugin detects outliers by calculating mean and SD for each pixel in > > a block of some given radius as those beyond mean +- x*SD, just as you > > suggested. Outlier pixels are replaced by smearing in their non-outlier > > neighbors. > > > > As the plugins use integral images to calculate block mean and SD, they > > are extraordinarily fast, independent of the block-radius. > > > > In case you are not using Fiji, find the plugins jar here: > > > > http://fly.mpi-cbg.de/saalfeld/download/integral_image.jar > > > > Best, > > Stephan > > > > > > > > > > On Fri, 2013-04-12 at 12:56 +0200, Agustin Lobo wrote: > >> Thanks, but I need a better way to define outliers. I think that I can create > >> a macro to calculate mean and sd (or mad) and define outliers a > >> outside mean+/- x*sd > >> with a variable value of x (i.e. 2, 2.5, 3 etc). Then calculate the > >> actual value in DN > >> and use the Remove outliers function. > >> Does anyone know if this has already been implemented? > >> > >> Agus > >> > >> On Fri, Apr 5, 2013 at 1:44 PM, Michael Schmid <[hidden email]> wrote: > >> > Hi Agus, > >> > > >> > FFT does not ignore NaN pixels (it does not work at all with images containing NaNs). > >> > > >> > Two possiblities: > >> > Process>Noise>Remove Outliers (replaces everything that deviates from the median of the surrounding by the median) > >> > > >> > Threshold the image to select the outliers, creating a selection (Edit>Selection>Create Selection) with the outliers and running a median thereafter - it will affect the selected pixels only. > >> > > >> > If you want to have NaNs for other purposes (e.g., 'Measure' ignores NaN pixels), threshold the image, create a selection and use Process>Math>Set to NaN. > >> > > >> > Michael > >> > ________________________________________________________________ > >> > On Apr 5, 2013, at 09:45, Agustin Lobo wrote: > >> > > >> >> I have images with outliers that should be set to null and not taken into > >> >> account for further processing. How do I set these values to null? > >> >> For example, I want to set as null values lower than 0 > >> >> and higher than 2000. > >> >> Also, once I have them defined as null, would any processing ignore > >> >> these values? > >> >> In particular, if I use an FFT band filter, I do not want these values > >> >> to expand from > >> >> isolated pixels to little patches (as occurs now). > >> >> An alternative would be to guess the values for the pixels > >> >> that have been set as null as the median from the 3x3 surrounding window > >> >> and the filter out using FFT. > >> >> > >> >> Thanks! > >> >> > >> >> Agus > >> >> > >> >> -- > >> >> ImageJ mailing list: http://imagej.nih.gov/ij/list.html > >> > > >> > -- > >> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html > >> > >> -- > >> ImageJ mailing list: http://imagej.nih.gov/ij/list.html > > > > -- > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
«
Return to ImageJ
|
1 view|%1 views
| Free forum by Nabble | Edit this page |