TrackMate: LAP Tracker not enough memory

[kb011]

Hi all,
I am using TrackMate to track my cells. I am now using the max memory of my
PC (14GB) and still ran into a problem that LAP tracker has not enough
memory to run my analysis. I know the obvious thing to do is to increase
the memory but would there be any way except this?

Thanks.

Cheers,
Karen

--
*Karen Grace V. Bondoc | Ph.D. Student*
*International Max Planck Research School *
*Institute for Inorganic and Analytical Chemistry *
*Bioorganic Analytics, Friedrich-Schiller-Universität Jena*
*Lessingstr. 8 (R 228)*
*D-07743 Jena*

*Tel.: [phone]*

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Jean-Yves Tinevez]

Hi Karen,

I am sorry you met this problem. TrackMate is clearly not optimized for an efficient memory management, and chose to sacrifice memory over speed. It looks like this choice does not work for you.

In your specific case, it looks like the problem is caused by the LAP tracker. In the current TrackMate implementation, solving the LAP requires generating a gigantic 2D matrix that computes the cost to link each spot to all other spots of the model. This happens if you chose to enable merging and splitting events (you have to be open to the possibility that a spot links to anywhere in other tracks). As soon as you have a lot of spots in your model (about 1 million), this becomes prohibitive in memory, and this caused most likely the error you saw.

Here is what I suggest:
- If you don't need to detect merging and splitting event, just stick to the simple LAP tracker, that forbids these events, and therefore allow for a lower memory requirements.
- If your tracking problem is easy, stick to the Nearest Neighbor tracker, the dumbest tracker ever.

On a developer side-note, this poor memory management is caused by our LAP implementation *not* taking advantages of sparse matrices (matrices where only a few elements are significant, this is the case for us since most spot-to-spot links are irrelevant). We need a solver that can deal with these matrices. An issue is open in the TrackMate forge (https://github.com/fiji/TrackMate/issues/14), but we have no concrete plans to do it on short term, and it won't be there in the forthcoming v2.2.0.

The original implementation of the LAP framework by Khulud Jaqaman offers this nice feature http://www.utsouthwestern.edu/labs/jaqaman/software/ (the TrackMate LAP is a stripped-down, oversimplified version of this).

best
jy

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[kb011]

Hi Jean,
Thanks for the explanation on how LAP tracker do the linking of each spot.
However I need the merging events as my cells sort of "touch" each other
while moving. I also have a long video (10min, 1fps) with many cells
(300-400 per frame)

Since its the spots in the matrix that the LAP tracker computes, then
lowering the number of pixels and image size would not help in the memory
problem of tracking, am I right?

Best,
Karen
On Mar 19, 2014 11:39 AM, "Jean-Yves Tinevez" <[hidden email]> wrote:

> Hi Karen,
>
> I am sorry you met this problem. TrackMate is clearly not optimized for an
> efficient memory management, and chose to sacrifice memory over speed. It
> looks like this choice does not work for you.
>
> In your specific case, it looks like the problem is caused by the LAP
> tracker. In the current TrackMate implementation, solving the LAP requires
> generating a gigantic 2D matrix that computes the cost to link each spot to
> all other spots of the model. This happens if you chose to enable merging
> and splitting events (you have to be open to the possibility that a spot
> links to anywhere in other tracks). As soon as you have a lot of spots in
> your model (about 1 million), this becomes prohibitive in memory, and this
> caused most likely the error you saw.
>
> Here is what I suggest:
> - If you don't need to detect merging and splitting event, just stick to
> the simple LAP tracker, that forbids these events, and therefore allow for
> a lower memory requirements.
> - If your tracking problem is easy, stick to the Nearest Neighbor tracker,
> the dumbest tracker ever.
>
> On a developer side-note, this poor memory management is caused by our LAP
> implementation *not* taking advantages of sparse matrices (matrices where
> only a few elements are significant, this is the case for us since most
> spot-to-spot links are irrelevant). We need a solver that can deal with
> these matrices. An issue is open in the TrackMate forge (
> https://github.com/fiji/TrackMate/issues/14), but we have no concrete
> plans to do it on short term, and it won't be there in the forthcoming
> v2.2.0.
>
> The original implementation of the LAP framework by Khulud Jaqaman offers
> this nice feature http://www.utsouthwestern.edu/labs/jaqaman/software/(the TrackMate LAP is a stripped-down, oversimplified version of this).
>
> best
> jy
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Jean-Yves Tinevez]

Hi Karen.
>
> Hi Jean,

Jean-Yves :)

> Thanks for the explanation on how LAP tracker do the linking of each spot.
> However I need the merging events as my cells sort of "touch" each other
> while moving. I also have a long video (10min, 1fps) with many cells
> (300-400 per frame)

This is the kind of use-case that stretch TrackMate to its limits. For now.
If investigating another solution (ICY, Imaris, u-track, …) is impossible, a workaround would be to forbid merging event (having only linear tracks) and correcting the problems with touching cells manually. But I guess with a movie of that size this is impracticable.
>
> Since its the spots in the matrix that the LAP tracker computes, then
> lowering the number of pixels and image size would not help in the memory
> problem of tracking, am I right?

You are right.

best
jy


--
Jean-Yves Tinevez
PFID - Imagopole
Institut Pasteur
25-28, rue du Docteur Roux
75724 Paris cedex 15
France
mob: [phone]
tel: [phone]
fax: [phone]


--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Zhan Chen]

Hi, Karen,
Could you tell me who is doing the test of Biotinylation of cell surface proteins? Could you post it on our postdoc web? Thank you so much.
Zhan

-----Original Message-----
From: ImageJ Interest Group [mailto:[hidden email]] On Behalf Of Jean-Yves Tinevez
Sent: 2014年3月19日 3:16
To: [hidden email]
Subject: Re: TrackMate: LAP Tracker not enough memory

Hi Karen,

I am sorry you met this problem. TrackMate is clearly not optimized for an efficient memory management, and chose to sacrifice memory over speed. It looks like this choice does not work for you.

In your specific case, it looks like the problem is caused by the LAP tracker. In the current TrackMate implementation, solving the LAP requires generating a gigantic 2D matrix that computes the cost to link each spot to all other spots of the model. This happens if you chose to enable merging and splitting events (you have to be open to the possibility that a spot links to anywhere in other tracks). As soon as you have a lot of spots in your model (about 1 million), this becomes prohibitive in memory, and this caused most likely the error you saw.

Here is what I suggest:
- If you don't need to detect merging and splitting event, just stick to the simple LAP tracker, that forbids these events, and therefore allow for a lower memory requirements.
- If your tracking problem is easy, stick to the Nearest Neighbor tracker, the dumbest tracker ever.

On a developer side-note, this poor memory management is caused by our LAP implementation *not* taking advantages of sparse matrices (matrices where only a few elements are significant, this is the case for us since most spot-to-spot links are irrelevant). We need a solver that can deal with these matrices. An issue is open in the TrackMate forge (https://github.com/fiji/TrackMate/issues/14), but we have no concrete plans to do it on short term, and it won't be there in the forthcoming v2.2.0.

The original implementation of the LAP framework by Khulud Jaqaman offers this nice feature http://www.utsouthwestern.edu/labs/jaqaman/software/ (the TrackMate LAP is a stripped-down, oversimplified version of this).

best
jy

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html