making RandomAccessStream use seek(long)

[Mario Emmenlauer-3]

Hi all,

I was working on extending the LSM Toolbox
(http://imagejdocu.tudor.lu/Members/ppirrotte/lsmtoolbox)
to be able to read Zeiss LSM files of more than 4GB filesize. For
that it would be good if the RandomAccessStream class would use
long (64bit) pointers for file offsets, especially for seek().
Below patch solved the issue for me, however I didn't check
if it's completely sane.

BTW: I was surprised to see seek(int) anyways, especially since
java nowadays uses long. Is it a remainder from old times or did
I overlook anything?

Cheers,

    Mario



--- RandomAccessStream.java 2007-08-16 23:45:02.000000000 +0200
+++ myRandomAccessStream.java 2007-08-16 23:48:06.000000000 +0200
@@ -1,3 +1,4 @@
+package ij.io;
  import java.io.*;
  import java.util.Vector;

@@ -15,7 +16,7 @@

      private InputStream src;
      private RandomAccessFile ras;
-    private int pointer;
+    private long pointer;
      private Vector data;
      private int length;
      private boolean foundEOS;
@@ -35,7 +36,7 @@
          this.ras = ras;
      }

-    public int getFilePointer() throws IOException {
+    public long getFilePointer() throws IOException {
          if (ras!=null)
              return (int)ras.getFilePointer();
          else
@@ -45,8 +46,8 @@
      public int read() throws IOException {
          if (ras!=null)
              return ras.read();
-        int l = pointer + 1;
-        int l1 = readUntil(l);
+        long l = pointer + 1;
+        long l1 = readUntil(l);
          if(l1 >= l) {
              byte abyte0[] = (byte[])data.elementAt((int)(pointer>>BLOCK_SHIFT));
              return abyte0[(int)(pointer++ & BLOCK_MASK)] & 0xff;
@@ -63,7 +64,7 @@
              throw new IndexOutOfBoundsException();
          if(len == 0)
              return 0;
-        int l = readUntil(pointer+len);
+        long l = readUntil(pointer+len);
          if(l<=pointer)
              return -1;
          else {
@@ -88,7 +89,7 @@
          } while (read<len);
      }

-    private int readUntil(int l) throws IOException {
+    private long readUntil(long l) throws IOException {
          if(l<length)
              return l;
          if(foundEOS)
@@ -116,10 +115,10 @@
          return length;
      }

-    public void seek(int loc) throws IOException {
+    public void seek(long loc) throws IOException {
          //if (ij.IJ.debugMode) ij.IJ.log("seek: "+loc);
          if (ras!=null)
-            {ras.seek(loc&0xffffffff); return;}
+            {ras.seek(loc); return;}
          if(loc < 0)
              pointer = 0;
          else


--
Mario Emmenlauer                             Phone: [phone]
Institute for Computer Science               Fax:   [phone]
Chair of Pattern Recognition and Image Processing
Albert-Ludwigs-University
Georges-Koehler-Allee 052, room 01-022
79110 Freiburg             mailto:emmenlau * informatik.uni-freiburg.de
Germany          http://lmb.informatik.uni-freiburg.de/people/emmenlau/

[ctrueden]

Hi Mario,

I am a little confused. I downloaded the LSM Toolbox but was unable to
find a class called RandomAccessStream inside the JAR file. However,
the Bio-Formats library has a RandomAccessStream class very similar to
what you describe.

Patrick: did you adapt our RandomAccessStream class for use in LSM
Toolbox? If so, I suggest updating the class to the latest version in
Bio-Formats, since many bugs have been fixed and we now support >2GB
files (meaning seek uses longs, etc.).

Mario, you could also try Bio-Formats for reading very large LSMs. It
does not have as many nifty LSM-related features as LSM Toolbox does,
but could be a stopgap measure until LSM Toolbox has been updated to
handle larger files.

-Curtis

On 8/16/07, Mario Emmenlauer <[hidden email]> wrote:

> Hi all,
>
> I was working on extending the LSM Toolbox
> (http://imagejdocu.tudor.lu/Members/ppirrotte/lsmtoolbox)
> to be able to read Zeiss LSM files of more than 4GB filesize. For
> that it would be good if the RandomAccessStream class would use
> long (64bit) pointers for file offsets, especially for seek().
> Below patch solved the issue for me, however I didn't check
> if it's completely sane.
>
> BTW: I was surprised to see seek(int) anyways, especially since
> java nowadays uses long. Is it a remainder from old times or did
> I overlook anything?
>
> Cheers,
>
>     Mario
>
>
>
> --- RandomAccessStream.java     2007-08-16 23:45:02.000000000 +0200
> +++ myRandomAccessStream.java   2007-08-16 23:48:06.000000000 +0200
> @@ -1,3 +1,4 @@
> +package ij.io;
>   import java.io.*;
>   import java.util.Vector;
>
> @@ -15,7 +16,7 @@
>
>       private InputStream src;
>       private RandomAccessFile ras;
> -    private int pointer;
> +    private long pointer;
>       private Vector data;
>       private int length;
>       private boolean foundEOS;
> @@ -35,7 +36,7 @@
>           this.ras = ras;
>       }
>
> -    public int getFilePointer() throws IOException {
> +    public long getFilePointer() throws IOException {
>           if (ras!=null)
>               return (int)ras.getFilePointer();
>           else
> @@ -45,8 +46,8 @@
>       public int read() throws IOException {
>           if (ras!=null)
>               return ras.read();
> -        int l = pointer + 1;
> -        int l1 = readUntil(l);
> +        long l = pointer + 1;
> +        long l1 = readUntil(l);
>           if(l1 >= l) {
>               byte abyte0[] = (byte[])data.elementAt((int)(pointer>>BLOCK_SHIFT));
>               return abyte0[(int)(pointer++ & BLOCK_MASK)] & 0xff;
> @@ -63,7 +64,7 @@
>               throw new IndexOutOfBoundsException();
>           if(len == 0)
>               return 0;
> -        int l = readUntil(pointer+len);
> +        long l = readUntil(pointer+len);
>           if(l<=pointer)
>               return -1;
>           else {
> @@ -88,7 +89,7 @@
>           } while (read<len);
>       }
>
> -    private int readUntil(int l) throws IOException {
> +    private long readUntil(long l) throws IOException {
>           if(l<length)
>               return l;
>           if(foundEOS)
> @@ -116,10 +115,10 @@
>           return length;
>       }
>
> -    public void seek(int loc) throws IOException {
> +    public void seek(long loc) throws IOException {
>           //if (ij.IJ.debugMode) ij.IJ.log("seek: "+loc);
>           if (ras!=null)
> -            {ras.seek(loc&0xffffffff); return;}
> +            {ras.seek(loc); return;}
>           if(loc < 0)
>               pointer = 0;
>           else
>
>
> --
> Mario Emmenlauer                             Phone: [phone]
> Institute for Computer Science               Fax:   [phone]
> Chair of Pattern Recognition and Image Processing
> Albert-Ludwigs-University
> Georges-Koehler-Allee 052, room 01-022
> 79110 Freiburg             mailto:emmenlau * informatik.uni-freiburg.de
> Germany          http://lmb.informatik.uni-freiburg.de/people/emmenlau/
>

[Mario Emmenlauer-3]

Hi,

I might have put this a little confusing, since I didn't think anyone
would actually care :-)

Curtis Rueden wrote:
> Hi Mario,
>
> I am a little confused. I downloaded the LSM Toolbox but was unable to
> find a class called RandomAccessStream inside the JAR file. However,
> the Bio-Formats library has a RandomAccessStream class very similar to
> what you describe.

The LSM Toolbox uses ij.io.RandomAccessStream, which AFAIK extends
javas InputStream but uses an RandomAccessFile to do the loading and storing.
The weired part is that javas RandomAccessFile fully supports long seek
offsets at least since java 1.4.2 (see
http://java.sun.com/j2se/1.4.2/docs/api/java/io/RandomAccessFile.html),
but ij.io.RandomAccessStream explicitly masks the offset to 32 bit.

I didn't even bother thinking about good style and just fixed
ij.io.RandomAccessStream (removed the masking, changed the pointers to
long) and put it with the LSM Toolbox sources.

> Patrick: did you adapt our RandomAccessStream class for use in LSM
> Toolbox? If so, I suggest updating the class to the latest version in
> Bio-Formats, since many bugs have been fixed and we now support >2GB
> files (meaning seek uses longs, etc.).

He didn't, I did. You can find my version (for testing) here:
   http://lmb.informatik.uni-freiburg.de/people/emmenlau/LSM_Toolbox.jar
but it's only temporary. once it's tested I hope the patch will be
part of the official Toolbox.

> Mario, you could also try Bio-Formats for reading very large LSMs. It
> does not have as many nifty LSM-related features as LSM Toolbox does,
> but could be a stopgap measure until LSM Toolbox has been updated to
> handle larger files.

Would I have known I wouldn't have gone through all the work :-(
Now the LSM Toolbox seems to do the job, but I might try Bio-Formats
anyways.

Thanks!

    Mario

> -Curtis
>
> On 8/16/07, Mario Emmenlauer <[hidden email]> wrote:
>> Hi all,
>>
>> I was working on extending the LSM Toolbox
>> (http://imagejdocu.tudor.lu/Members/ppirrotte/lsmtoolbox)
>> to be able to read Zeiss LSM files of more than 4GB filesize. For
>> that it would be good if the RandomAccessStream class would use
>> long (64bit) pointers for file offsets, especially for seek().
>> Below patch solved the issue for me, however I didn't check
>> if it's completely sane.
>>
>> BTW: I was surprised to see seek(int) anyways, especially since
>> java nowadays uses long. Is it a remainder from old times or did
>> I overlook anything?
>>
>> Cheers,
>>
>>     Mario
>>
>>
>>
>> --- RandomAccessStream.java     2007-08-16 23:45:02.000000000 +0200
>> +++ myRandomAccessStream.java   2007-08-16 23:48:06.000000000 +0200
>> @@ -1,3 +1,4 @@
>> +package ij.io;
>>   import java.io.*;
>>   import java.util.Vector;
>>
>> @@ -15,7 +16,7 @@
>>
>>       private InputStream src;
>>       private RandomAccessFile ras;
>> -    private int pointer;
>> +    private long pointer;
>>       private Vector data;
>>       private int length;
>>       private boolean foundEOS;
>> @@ -35,7 +36,7 @@
>>           this.ras = ras;
>>       }
>>
>> -    public int getFilePointer() throws IOException {
>> +    public long getFilePointer() throws IOException {
>>           if (ras!=null)
>>               return (int)ras.getFilePointer();
>>           else
>> @@ -45,8 +46,8 @@
>>       public int read() throws IOException {
>>           if (ras!=null)
>>               return ras.read();
>> -        int l = pointer + 1;
>> -        int l1 = readUntil(l);
>> +        long l = pointer + 1;
>> +        long l1 = readUntil(l);
>>           if(l1 >= l) {
>>               byte abyte0[] = (byte[])data.elementAt((int)(pointer>>BLOCK_SHIFT));
>>               return abyte0[(int)(pointer++ & BLOCK_MASK)] & 0xff;
>> @@ -63,7 +64,7 @@
>>               throw new IndexOutOfBoundsException();
>>           if(len == 0)
>>               return 0;
>> -        int l = readUntil(pointer+len);
>> +        long l = readUntil(pointer+len);
>>           if(l<=pointer)
>>               return -1;
>>           else {
>> @@ -88,7 +89,7 @@
>>           } while (read<len);
>>       }
>>
>> -    private int readUntil(int l) throws IOException {
>> +    private long readUntil(long l) throws IOException {
>>           if(l<length)
>>               return l;
>>           if(foundEOS)
>> @@ -116,10 +115,10 @@
>>           return length;
>>       }
>>
>> -    public void seek(int loc) throws IOException {
>> +    public void seek(long loc) throws IOException {
>>           //if (ij.IJ.debugMode) ij.IJ.log("seek: "+loc);
>>           if (ras!=null)
>> -            {ras.seek(loc&0xffffffff); return;}
>> +            {ras.seek(loc); return;}
>>           if(loc < 0)
>>               pointer = 0;
>>           else
>>
>>
>> --
>> Mario Emmenlauer                             Phone: [phone]
>> Institute for Computer Science               Fax:   [phone]
>> Chair of Pattern Recognition and Image Processing
>> Albert-Ludwigs-University
>> Georges-Koehler-Allee 052, room 01-022
>> 79110 Freiburg             mailto:emmenlau * informatik.uni-freiburg.de
>> Germany          http://lmb.informatik.uni-freiburg.de/people/emmenlau/
>>
>

--
Mario Emmenlauer                             Phone: [phone]
Institute for Computer Science               Fax:   [phone]
Chair of Pattern Recognition and Image Processing
Albert-Ludwigs-University
Georges-Koehler-Allee 052, room 01-022
79110 Freiburg             mailto:emmenlau * informatik.uni-freiburg.de
Germany          http://lmb.informatik.uni-freiburg.de/people/emmenlau/

[Mario Emmenlauer-3]

Hi again,

Mario Emmenlauer wrote:
>> Patrick: did you adapt our RandomAccessStream class for use in LSM
>> Toolbox? If so, I suggest updating the class to the latest version in
>> Bio-Formats, since many bugs have been fixed and we now support >2GB
>> files (meaning seek uses longs, etc.).
>
> He didn't, I did. You can find my version (for testing) here:
>   http://lmb.informatik.uni-freiburg.de/people/emmenlau/LSM_Toolbox.jar
> but it's only temporary. once it's tested I hope the patch will be
> part of the official Toolbox.

I'm at this very instant testing a new ImageJ-version (1.39d) I got from
Wayne Rasband, that uses an overloaded RandomAccessStream which supports 32
and 64bit offsets (depending on what you supply to it).

In case you have an improved version it might still be good to share
it with him (and me, in any case:-)).

Wayne, thanks for the fixes, it works perfectly. Sorry for the confusion
earlier, I had ImageJ still running in the background :-(

I will send Patrick a new patch against current LSM Toolbox and hope it
will be included in the official release soon. Anyone able to test the
LSM Toolbox is more than welcome to send me error reports.

Cheers,

   Mario


BTW: here the url again:
http://lmb.informatik.uni-freiburg.de/people/emmenlau/LSM_Toolbox.jar

[Liu, Dongfang (NIH/NIAID) [F]]

Dear All,

 

Could anyone provide spinning disc-confocal microscopy to test a pilot experiment in NIH? It is great if your spinning disc-confocal microscopy is combined with TIRF microscopy. Then I can switch TIRF to spinning disc confocal microscopy to acquire some images.


We are trying to visualize some endocytosis events in live cells, combining TIRF and spinning-disc confocal microscopy. Any suggestions would be greatly appreciated!!!

 

Best regards,

Dongfang

My phone: [phone]