| Reading large files with restore [message #85942] |
Fri, 20 September 2013 08:18  |
Sir Loin Steak
Messages: 42
Registered: January 2012
|
Member |
|
|
Hi all,
I have a script (which I didn't write) which loops through different months and reads in atmospheric variables from .sav files, with a different file for each month. These files contain lots of data, and hence are around 200-500 MB in size.
When I run the script it takes a long time to restore the data on each loop (obviously!), but it makes my entire computer go slow, and on occasion it completely freezes and I have no choice but to actually turn the computer off and on again.
I am assuming this is a memory issue, and was wondering if anyone knew a way around these problems (re-writing the script and save files isn't really an option I want to consider). Is there perhaps some way of 'de-allocating' the memory after the completion of each loop?
Thanks,
Liam
|
|
|
|
| Re: Reading large files with restore [message #85943 is a reply to message #85942] |
Fri, 20 September 2013 08:25  |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
ljs15@fsmail.net writes:
> I have a script (which I didn't write) which loops through different months and reads in atmospheric variables from .sav files, with a different file for each month. These files contain lots of data, and hence are around 200-500 MB in size.
>
> When I run the script it takes a long time to restore the data on each loop (obviously!), but it makes my entire computer go slow, and on occasion it completely freezes and I have no choice but to actually turn the computer off and on again.
>
> I am assuming this is a memory issue, and was wondering if anyone knew a way around these problems (re-writing the script and save files isn't really an option I want to consider). Is there perhaps some way of 'de-allocating' the memory after the completion of each loop?
Wait, you have a program that does something crazy, but you don't want
to change it? Do I have that right? I would say off-hand that this
hampers our ability to provide a solution. (Even prayer is unlikely to
work in this situation.)
Is this "script" a main-level program by any chance?
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
Sepore ma de ni thue. ("Perhaps thou speakest truth.")
|
|
|
|
| Re: Reading large files with restore [message #85944 is a reply to message #85943] |
Fri, 20 September 2013 08:30 |
Sir Loin Steak
Messages: 42
Registered: January 2012
|
Member |
|
|
On Friday, 20 September 2013 16:25:36 UTC+1, David Fanning wrote:
> ljs15@fsmail.net writes:
>
>
>
>> I have a script (which I didn't write) which loops through different months and reads in atmospheric variables from .sav files, with a different file for each month. These files contain lots of data, and hence are around 200-500 MB in size.
>
>>
>
>> When I run the script it takes a long time to restore the data on each loop (obviously!), but it makes my entire computer go slow, and on occasion it completely freezes and I have no choice but to actually turn the computer off and on again.
>
>>
>
>> I am assuming this is a memory issue, and was wondering if anyone knew a way around these problems (re-writing the script and save files isn't really an option I want to consider). Is there perhaps some way of 'de-allocating' the memory after the completion of each loop?
>
>
>
> Wait, you have a program that does something crazy, but you don't want
>
> to change it? Do I have that right? I would say off-hand that this
>
> hampers our ability to provide a solution. (Even prayer is unlikely to
>
> work in this situation.)
>
>
>
> Is this "script" a main-level program by any chance?
>
>
>
> Cheers,
>
>
>
> David
>
>
>
>
>
>
>
> --
>
> David Fanning, Ph.D.
>
> Fanning Software Consulting, Inc.
>
> Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
>
> Sepore ma de ni thue. ("Perhaps thou speakest truth.")
Ha! Well, what I mean is that the .sav files have already been created, and I don't have access to any of the scripts or the original data to re-produce them. So whatever I do, I'm still going to have to read the .sav files in.
The script is a main-level program which restores the save files, gets the relevant data it wants from the complete data set (e.g. temperature and cloud opacities) and then writes this subset of data to a file.
As I say, I can't (easily or quickly) do anything about the save files, so just thought there may be some memory thing which can help me.
Cheers,
Liam
|
|
|
|
| Re: Reading large files with restore [message #85945 is a reply to message #85944] |
Fri, 20 September 2013 08:33 |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
ljs15@fsmail.net writes:
> Ha! Well, what I mean is that the .sav files have already been created, and I don't have access to any of the scripts or the original data to re-produce them. So whatever I do, I'm still going to have to read the .sav files in.
>
> The script is a main-level program which restores the save files, gets the relevant data it wants from the complete data set (e.g. temperature and cloud opacities) and then writes this subset of data to a file.
>
> As I say, I can't (easily or quickly) do anything about the save files, so just thought there may be some memory thing which can help me.
Try adding a .Reset at the end of your loop.
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
Sepore ma de ni thue. ("Perhaps thou speakest truth.")
|
|
|
|
| Re: Reading large files with restore [message #85946 is a reply to message #85945] |
Fri, 20 September 2013 08:50 |
Sir Loin Steak
Messages: 42
Registered: January 2012
|
Member |
|
|
On Friday, 20 September 2013 16:33:46 UTC+1, David Fanning wrote:
> ljs15@fsmail.net writes:
>
>
>
>> Ha! Well, what I mean is that the .sav files have already been created, and I don't have access to any of the scripts or the original data to re-produce them. So whatever I do, I'm still going to have to read the .sav files in.
>
>>
>
>> The script is a main-level program which restores the save files, gets the relevant data it wants from the complete data set (e.g. temperature and cloud opacities) and then writes this subset of data to a file.
>
>>
>
>> As I say, I can't (easily or quickly) do anything about the save files, so just thought there may be some memory thing which can help me.
>
>
>
> Try adding a .Reset at the end of your loop.
>
>
>
> Cheers,
>
>
>
> David
>
>
>
>
>
>
>
> --
>
> David Fanning, Ph.D.
>
> Fanning Software Consulting, Inc.
>
> Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
>
> Sepore ma de ni thue. ("Perhaps thou speakest truth.")
Thanks David,
However, while I can enter .reset into the command line no problem, the code won't compile with this added anywhere in the script. Am I doing something idiotic? A search of the web has offered no help. :-(
|
|
|
|
| Re: Reading large files with restore [message #85947 is a reply to message #85946] |
Fri, 20 September 2013 09:00 |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
ljs15@fsmail.net writes:
> However, while I can enter .reset into the command line no problem, the code won't compile with this added anywhere in the script. Am I doing something idiotic? A search of the web has offered no help.
Yeah, I guess not. :-(
Humm. You could undefine your variables at the end of the loop (UNDEFINE
command from the Coyote Library), but this presupposes you know the
names of the variables you are creating, and I suppose there could be a
great many of them. You could also try to track down the SOB who created
the script you are using. Not sure that would help all that much,
though. :-)
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
Sepore ma de ni thue. ("Perhaps thou speakest truth.")
|
|
|
|
|
|
| Re: Reading large files with restore [message #85979 is a reply to message #85948] |
Mon, 23 September 2013 05:48 |
Sir Loin Steak
Messages: 42
Registered: January 2012
|
Member |
|
|
On Friday, 20 September 2013 17:46:42 UTC+1, Phillip Bitzer wrote:
> Let me see if I understand:
>
>
>
> 1) You have a save file with a *lot* of data.
>
> 2) You'd like to restore and work with some of the data.
>
> 3) It takes a long time to restore everything in the file and extract just what you want.
>
>
>
> If this is correct, then the IDLSave_File object is what you're after.
>
>
>
> Consider a save file created with :
>
> SAVE, var1, var2, file='idl.sav'
>
>
>
> Then you can get *just* var1 with
>
> sObj = OBJ_NEW('IDL_Savefile', 'idl.sav')
>
> sObj->RESTORE, 'var1'
>
> OBJ_DESTROY, sObj ;when you're done with the file
>
>
>
> There are ways to get the variable names if you don't know them a priori. Check out:
>
> http://www.exelisvis.com/docs/IDL_Savefile.html
Thanks very much, that sounds like exactly what I want! I'll give it a go now.
|
|
|
|
| Re: Reading large files with restore [message #86022 is a reply to message #85948] |
Tue, 24 September 2013 10:56 |
Paul Van Delst[1]
Messages: 1157
Registered: April 2002
|
Senior Member |
|
|
Sweet. I did not know about these savefile objects and methods (I
thought they were still pre-object-y).
I'm still using my own handrolled "dumpfile" procedures from the 90's
for quick-and-dirty output -- I guess I can delete 'em now.
Thanks for the info/reminder.
cheers,
paulv
p.s. To the OP, please post the results of your trying PhillipB's tips
below if you get a chance. I'm interested if they work the way you need.
On 09/20/13 12:46, Phillip Bitzer wrote:
> Let me see if I understand:
>
> 1) You have a save file with a *lot* of data.
> 2) You'd like to restore and work with some of the data.
> 3) It takes a long time to restore everything in the file and extract just what you want.
>
> If this is correct, then the IDLSave_File object is what you're after.
>
> Consider a save file created with :
> SAVE, var1, var2, file='idl.sav'
>
> Then you can get *just* var1 with
> sObj = OBJ_NEW('IDL_Savefile', 'idl.sav')
> sObj->RESTORE, 'var1'
> OBJ_DESTROY, sObj ;when you're done with the file
>
> There are ways to get the variable names if you don't know them a priori. Check out:
> http://www.exelisvis.com/docs/IDL_Savefile.html
>
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
