| Re: Reading binary files made by Python [message #49264] |
Tue, 11 July 2006 00:14 |
![Maarten[1] is currently offline](theme/default/images/xoffline.png.pagespeed.ic.XRkd1fkXye.png "Maarten[1] is currently offline")
Maarten[1]
Messages: 176
Registered: November 2005
|
Senior Member |
|
|
Rick Towler wrote:
> FWIW, I would consider XDR if you can do that from the Python side. It
> is *much* easier to use than HDF, CDF, etc. and sounds like it is
> exactly what you are looking for. The only potential issue is that I
> don't think MATLAB reads it so if that is a requirement I would go back
> to your original approach.
Yes, that is probably even better, and part of the standard python
library:
http://docs.python.org/lib/module-xdrlib.html
I usually work with hierarchical data sets, or data that is best stored
in that way, hence my suggestion to use HDF.
Maarten
|
|
|
|
| Re: Reading binary files made by Python [message #49266 is a reply to message #49264] |
Mon, 10 July 2006 13:22  |
Rick Towler
Messages: 821
Registered: August 1998
|
Senior Member |
|
|
kuyper@wizard.net wrote:
> Rick Towler wrote:
> ...
>> FWIW, I would consider XDR if you can do that from the Python side. It
>> is *much* easier to use than HDF, CDF, etc. and sounds like it is
>> exactly what you are looking for. The only potential issue is that I
>> don't think MATLAB reads it so if that is a requirement I would go back
>> to your original approach.
>
> Keep in mind that HDF and CDF use XDR internally.
>
I'm trying to K-I-S-S. Set the XDR keyword on OPENR/OPENW and you just
read/write the data... Easier than storing data in HDF/CDF and it seems
to fit the limited needs of the OP.
-Rick
|
|
|
|
| Re: Reading binary files made by Python [message #49267 is a reply to message #49266] |
Mon, 10 July 2006 11:51 |
James Kuyper
Messages: 425
Registered: March 2000
|
Senior Member |
|
|
Rick Towler wrote:
...
> FWIW, I would consider XDR if you can do that from the Python side. It
> is *much* easier to use than HDF, CDF, etc. and sounds like it is
> exactly what you are looking for. The only potential issue is that I
> don't think MATLAB reads it so if that is a requirement I would go back
> to your original approach.
Keep in mind that HDF and CDF use XDR internally.
|
|
|
|
| Re: Reading binary files made by Python [message #49276 is a reply to message #49267] |
Mon, 10 July 2006 08:55 |
Rick Towler
Messages: 821
Registered: August 1998
|
Senior Member |
|
|
Sheldon wrote:
> Maarten wrote:
>> Sheldon wrote:
>>
>>> I have created som binary files using python and tried to read them
>>> with IDL. I used the binary_template as well but IDL completely
>>> misinterprets the data and returns wrong values and values that were
>>> never written in the file.
>>
>> I would recommend to write out the data in a portable file format, like
>> HDF (pytables is an easy way to do that). You can then use the idl HDF
>> routines to ingest the data, avoiding the larger part of the headaches.
>>
>> Maarten
>>
>> pytables: http://www.pytables.org/
>
> This is a very good idea. I think that I will do just that. The data
> was written and read on the same machine. Silly me, I thought that
> binary is just a binary but I guess not. Will try the HDF format.
Binary is just binary :) Try to keep it simple and just use OPENR/READU
and your knowledge of the file. Do it interactively a few bytes at a
time. You'll at least know exactly when things go wrong...
FWIW, I would consider XDR if you can do that from the Python side. It
is *much* easier to use than HDF, CDF, etc. and sounds like it is
exactly what you are looking for. The only potential issue is that I
don't think MATLAB reads it so if that is a requirement I would go back
to your original approach.
-Rick
|
|
|
|
| Re: Reading binary files made by Python [message #49277 is a reply to message #49276] |
Mon, 10 July 2006 06:41 |
Liberum
Messages: 48
Registered: September 2005
|
Member |
|
|
Maarten wrote:
> Sheldon wrote:
>
>> I have created som binary files using python and tried to read them
>> with IDL. I used the binary_template as well but IDL completely
>> misinterprets the data and returns wrong values and values that were
>> never written in the file. Now Matlab is able to read these files
>> without problem. My question is why is IDL failing to do this? I know
>> that the binaries are Float32 and the dimensions are correct. I am
>> doing something wrong?
>
> Without actually seeing the code you used to write the files, it seems
> a bit hard to answer this question. One obvious issue is the endianess
> of the data, I don't know what python, IDL or matlab assume about that
> (I guess you use the same machine to write and read the data).
>
> I would recommend to write out the data in a portable file format, like
> HDF (pytables is an easy way to do that). You can then use the idl HDF
> routines to ingest the data, avoiding the larger part of the headaches.
>
> Maarten
>
> pytables: http://www.pytables.org/
This is a very good idea. I think that I will do just that. The data
was written and read on the same machine. Silly me, I thought that
binary is just a binary but I guess not. Will try the HDF format.
Thanks,
/Sheldon
|
|
|
|
| Re: Reading binary files made by Python [message #49278 is a reply to message #49277] |
Mon, 10 July 2006 06:29 |
Maarten[1]
Messages: 176
Registered: November 2005
|
Senior Member |
|
|
Sheldon wrote:
> I have created som binary files using python and tried to read them
> with IDL. I used the binary_template as well but IDL completely
> misinterprets the data and returns wrong values and values that were
> never written in the file. Now Matlab is able to read these files
> without problem. My question is why is IDL failing to do this? I know
> that the binaries are Float32 and the dimensions are correct. I am
> doing something wrong?
Without actually seeing the code you used to write the files, it seems
a bit hard to answer this question. One obvious issue is the endianess
of the data, I don't know what python, IDL or matlab assume about that
(I guess you use the same machine to write and read the data).
I would recommend to write out the data in a portable file format, like
HDF (pytables is an easy way to do that). You can then use the idl HDF
routines to ingest the data, avoiding the larger part of the headaches.
Maarten
pytables: http://www.pytables.org/
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
