| IDLv5.4 [message #22942] |
Tue, 19 December 2000 12:58  |
Jeffrey Newmark
Messages: 8
Registered: April 1996
|
Junior Member |
|
|
Hi All,
I was wondering if anyone has had problems reading save sets
made with IDL V5.4 on older versions? Is this a feature? I
can read old save sets on the new version. If indeed the savesets
are not backwards compatible, that is very annoying!
IDL> print,!version
{ alpha OSF unix 5.4 Sep 25 2000 64 64}
thanks,
Jeff Newmark
SOHO NASA/GSFC
|
|
|
|
| Re: IDLv5.4 [message #23037 is a reply to message #22942] |
Sun, 31 December 2000 07:26  |
Craig Markwardt
Messages: 1869
Registered: November 1996
|
Senior Member |
|
|
davidf@dfanning.com (David Fanning) writes:
> Craig Markwardt (craigmnet@cow.physics.wisc.edu) writes:
>
>> On the political side, I think all RSI guaranteed was that SAVE files
>> would be *forward* compatible, so they haven't broken any promises.
>
> This is what I remember thinking was the case, too. But
> here is what the documentation for the SAVE command says:
>
> "Note also that save files containing routines may not be
> compatible between different versions of IDL, but that
> files containing data are always backwards-compatible."
>
> Do you suppose "backwards" really means "forward", which
> would mean...
After looking a little further, I think this may indeed be a bug in
IDL, or at least a misimplementation of a design spec.
So far the only difference I found was that they have changed one
record type to allow for 64-bit file offsets, as opposed to 32-bit.
Presumably this is to allow people to save files larger than 4
gigabytes. However there are several tragic flaws in the
implementation.
Every record in an IDL SAVE file of the past had a 16-byte header.
Files are composed of a sequence of records and each one has a small
header with the record type, and a pointer to the next record in the
file. Sometimes this structure is called a "tagged" format (like TIFF
is the Tagged Image File Format). This is an easy way for even old
versions of IDL to read newer files: if they encounter a record type
they don't recognize, they just skip to the next one. Since the
header always had the same layout, this ensured compatibility.
In IDL 5.4, the header layout has changed incompatibly, *sort of*.
The new record header has *20* bytes instead of 16 bytes, to
accomodate 64-bit instead of 32-bit file pointers. However the IDL
people seem to have recognized this might be incompatible, so they
implemented a new record type (type 17) which signals a format change.
Before record 17 appears in the file, the old 16-byte header layout is
used; afterwards the new 20-byte layout is used.
This is tragic for several reasons:
* Record 17 appeared even in the short files that David and Jeff sent
me. They could have easily only written this record for large
datasets, ones that were known to exceed 4 gigabyte limit. That at
least would have limited the incompatibility to large files that IDL
5.3 couldn't store anyway.
* Even more tragic is the fact that the old 16-byte layout appeared to
have eight unused bytes. They could have allocated some of these
existing bytes, and kept the header size unchanged that way. The
current approach has variable-sized headers, which guarantees
incompatibility.
* The original IDL SAVE protocol was so elegant because it was largely
state-free. That is, one record really didn't depend on another.
This allowed old versions of IDL to ignore records they didn't
understand, with little or no consequences. This isn't true
anymore.
With all this said, I think it is fair to say now that RSI had the
opportunity to make this new format backward compatible, but they
dropped the ball somehow. Perhaps they intended to maintain
compatibility but didn't test enough to discover the problem. In that
case there is hope it will be corrected in a new release. Another
remote possibility is that they are pushing a new file format ala
Microsoft to encourage upgrades. Let's hope this isn't the case.
Craig
--
------------------------------------------------------------ --------------
Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
------------------------------------------------------------ --------------
|
|
|
|
| Re: IDLv5.4 [message #23038 is a reply to message #22942] |
Sat, 30 December 2000 20:53 |
davidf
Messages: 2866
Registered: September 1996
|
Senior Member |
|
|
Craig Markwardt (craigmnet@cow.physics.wisc.edu) writes:
> On the political side, I think all RSI guaranteed was that SAVE files
> would be *forward* compatible, so they haven't broken any promises.
This is what I remember thinking was the case, too. But
here is what the documentation for the SAVE command says:
"Note also that save files containing routines may not be
compatible between different versions of IDL, but that
files containing data are always backwards-compatible."
Do you suppose "backwards" really means "forward", which
would mean...
Oh, never mind, I'm confused enough chasing events
around at the moment. :-(
Cheers,
David
P.S. Let's just say I'm not even going to mention
the words "Column Major". :-)
--
David Fanning, Ph.D.
Fanning Software Consulting
Phone: 970-221-0438 E-Mail: davidf@dfanning.com
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Toll-Free IDL Book Orders: 1-888-461-0155
|
|
|
|
| Re: IDLv5.4 [message #23040 is a reply to message #22942] |
Sat, 30 December 2000 19:56 |
Craig Markwardt
Messages: 1869
Registered: November 1996
|
Senior Member |
|
|
Greetings again--
Thanks to David and Jeffrey for sending me examples of SAVE files from
IDL version 5.4.
I can confirm that these files are not in a format compatible with
versions earlier than IDL 5.4. Both files that were sent to me by
David and Jeffrey had a format version of "6". For comparison, other
IDL 5's produce a file version of "5" and IDL 4 doesn't store a file
version. There is at least one new record type, and the format of
some existing record types seems to have changed. More research on
this is needed.
Therefore I can pretty safely conclude that this change was a
*conscious* one by RSI, and not an incidental bug. It is virtually
guaranteed that files produced by IDL 5.4 will not be backward
compatible.
On the political side, I think all RSI guaranteed was that SAVE files
would be *forward* compatible, so they haven't broken any promises.
However they really should have withheld this change until the next
major revision, and especially not a minor revision. The IDL 4 -> 5
conversion didn't even break backward compatibility I'm pretty sure.
And finally on the more positive side, I am still working on
documenting my code. It can read and write SAVE files in the older
format, and in principle can be extended to support both formats.
Craig
Craig Markwardt <craigmnet@cow.physics.wisc.edu> writes:
> I've been following this discussion with some interest. I've been
> slowly developing and documenting some tools to open, read,
> interrogate, and write IDL save files, separate from IDL itself. For
> example I have an IDL procedure while can print a "directory" of the
> contents of a save file. This might also be nice for programmers who
> would want to save and restore variables with SAVE files, and don't
> want to be stuck with the SAVE and RESTORE commands.
>
> I would imagine that people would be interested in trying this
> out. Is that true?
>
> I would also like to see some examples of this IDL 5.4 behavior.
> Maybe I can figure out where it's going wrong. Email of some short
> files would be fine.
>
> Craig
>
>
> Jeffrey Newmark <newmark@eitv2.nascom.nasa.gov> writes:
>> Hi Ben and others,
>>
>> I am having this problem with data, any array at all! I am
>> aware of problems previously with restoring complex structures, code
>> etc., but for many versions now I have not had the problem of
>> restoring a simple data array with an older version of IDL.
>>
>> Ben Tupper wrote:
>>>
>>> Hello,
>>>
>>> I'm probably reminding you of something you already know, but only
>>> data in SAVEd files are IDL version independent. Code is not version
>>> independent. This behavior is noted in the online documentation for
>>> SAVE.
>>>
>>> Are your saved sets of code or data? If it's data that you can't
>>> RESTORE, then we all better be listening. (How many funky
>>> colortables have I saved as *.pal?)
>>>
>>> Ben
>>>
>>> Jeffrey Newmark wrote:
>>>
>>>> Hi All,
>>>>
>>>> I was wondering if anyone has had problems reading save sets
>>>> made with IDL V5.4 on older versions? Is this a feature? I
>>>> can read old save sets on the new version. If indeed the savesets
>>>> are not backwards compatible, that is very annoying!
>>>>
>>>> IDL> print,!version
>>>> { alpha OSF unix 5.4 Sep 25 2000 64 64}
>>>>
>>>> thanks,
>>>> Jeff Newmark
>>>> SOHO NASA/GSFC
>>>
>>> --
>>> Ben Tupper
>>> 248 Lower Round Pond Road
>>> POB 106
>>> Bristol, ME 04539
>>>
>>> Tel: (207) 563-1048
>>> Email: PemaquidRiver@tidewater.net
>>
>> --
>> ------------------------------------------------------------ ------
>>
>> Dr. Jeffrey Newmark
>> SOHO - EIT Team Scientist
>
> --
> ------------------------------------------------------------ --------------
> Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
> Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
> ------------------------------------------------------------ --------------
--
------------------------------------------------------------ --------------
Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
------------------------------------------------------------ --------------
|
|
|
|
|
|
| Re: IDLv5.4 [message #23049 is a reply to message #22942] |
Wed, 27 December 2000 14:04 |
Craig Markwardt
Messages: 1869
Registered: November 1996
|
Senior Member |
|
|
I've been following this discussion with some interest. I've been
slowly developing and documenting some tools to open, read,
interrogate, and write IDL save files, separate from IDL itself. For
example I have an IDL procedure while can print a "directory" of the
contents of a save file. This might also be nice for programmers who
would want to save and restore variables with SAVE files, and don't
want to be stuck with the SAVE and RESTORE commands.
I would imagine that people would be interested in trying this
out. Is that true?
I would also like to see some examples of this IDL 5.4 behavior.
Maybe I can figure out where it's going wrong. Email of some short
files would be fine.
Craig
Jeffrey Newmark <newmark@eitv2.nascom.nasa.gov> writes:
> Hi Ben and others,
>
> I am having this problem with data, any array at all! I am
> aware of problems previously with restoring complex structures, code
> etc., but for many versions now I have not had the problem of
> restoring a simple data array with an older version of IDL.
>
> Ben Tupper wrote:
>>
>> Hello,
>>
>> I'm probably reminding you of something you already know, but only
>> data in SAVEd files are IDL version independent. Code is not version
>> independent. This behavior is noted in the online documentation for
>> SAVE.
>>
>> Are your saved sets of code or data? If it's data that you can't
>> RESTORE, then we all better be listening. (How many funky
>> colortables have I saved as *.pal?)
>>
>> Ben
>>
>> Jeffrey Newmark wrote:
>>
>>> Hi All,
>>>
>>> I was wondering if anyone has had problems reading save sets
>>> made with IDL V5.4 on older versions? Is this a feature? I
>>> can read old save sets on the new version. If indeed the savesets
>>> are not backwards compatible, that is very annoying!
>>>
>>> IDL> print,!version
>>> { alpha OSF unix 5.4 Sep 25 2000 64 64}
>>>
>>> thanks,
>>> Jeff Newmark
>>> SOHO NASA/GSFC
>>
>> --
>> Ben Tupper
>> 248 Lower Round Pond Road
>> POB 106
>> Bristol, ME 04539
>>
>> Tel: (207) 563-1048
>> Email: PemaquidRiver@tidewater.net
>
> --
> ------------------------------------------------------------ ------
>
> Dr. Jeffrey Newmark
> SOHO - EIT Team Scientist
--
------------------------------------------------------------ --------------
Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
------------------------------------------------------------ --------------
|
|
|
|
|
|
| Re: IDLv5.4 [message #23058 is a reply to message #22942] |
Wed, 27 December 2000 07:09 |
Jeffrey Newmark
Messages: 8
Registered: April 1996
|
Junior Member |
|
|
Hi Ben and others,
I am having this problem with data, any array at all! I am
aware of problems previously with restoring complex structures, code
etc., but for many versions now I have not had the problem of
restoring a simple data array with an older version of IDL.
Ben Tupper wrote:
>
> Hello,
>
> I'm probably reminding you of something you already know, but only
> data in SAVEd files are IDL version independent. Code is not version
> independent. This behavior is noted in the online documentation for
> SAVE.
>
> Are your saved sets of code or data? If it's data that you can't
> RESTORE, then we all better be listening. (How many funky
> colortables have I saved as *.pal?)
>
> Ben
>
> Jeffrey Newmark wrote:
>
>> Hi All,
>>
>> I was wondering if anyone has had problems reading save sets
>> made with IDL V5.4 on older versions? Is this a feature? I
>> can read old save sets on the new version. If indeed the savesets
>> are not backwards compatible, that is very annoying!
>>
>> IDL> print,!version
>> { alpha OSF unix 5.4 Sep 25 2000 64 64}
>>
>> thanks,
>> Jeff Newmark
>> SOHO NASA/GSFC
>
> --
> Ben Tupper
> 248 Lower Round Pond Road
> POB 106
> Bristol, ME 04539
>
> Tel: (207) 563-1048
> Email: PemaquidRiver@tidewater.net
--
------------------------------------------------------------ ------
Dr. Jeffrey Newmark
SOHO - EIT Team Scientist
|
|
|
|
|
|
| Re: IDLv5.4 [message #23172 is a reply to message #23038] |
Sat, 06 January 2001 09:19 |
William Daffer
Messages: 34
Registered: February 1999
|
Member |
|
|
davidf@dfanning.com (David Fanning) writes:
<snip>
> P.S. Let's just say I'm not even going to mention
> the words "Column Major". :-)
Oh *nooooo*. Not the Column Major Thread!
whd
--
Outside of a dog a book is man's best friend.
Inside of a dog it's too dark to read
Groucho Marx
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
