comp.lang.idl-pvwave archive - RDF feed
type: channel
description: Messages from Usenet group comp.lang.idl-pvwave, compiled by Paulo Penteado
items: Re: Object Madness or Restoring Nightmares

Re: Object Madness or Restoring Nightmares
type: item
description: David Fanning writes: > I've proved that it is not the save/restore cycle that is > doing this, because if the study contains just non-objects, > say images, then there is no memory leakage. Only when the > study contains objects do I leak. Oh, my goodness, I love programming! You don't even have to solve the problems. They solve themselves if you can just bring yourself to face abject humiliation and write about them. After writing my last missive, I got a cup of tea (a technique I learned from an Englishman) and sat down to try again. The problem had disappeared!! Sigh...Done correctly, IDL programming really is a spiritual practice. Cheers, David -- David Fanning, Ph.D. Fanning Software Consulting Coyote's Guide to IDL Programming: http://www.dfanning.com/
creator: David Fanning
date: 2004-03-03T00:45:28-00:00

Re: Object Madness or Restoring Nightmares
type: item
description: "David Fanning" wrote ... > David Fanning writes: > >> I've proved that it is not the save/restore cycle that is >> doing this, because if the study contains just non-objects, >> say images, then there is no memory leakage. Only when the >> study contains objects do I leak. > > Oh, my goodness, I love programming! You don't even have > to solve the problems. They solve themselves if you can > just bring yourself to face abject humiliation and > write about them. > > After writing my last missive, I got a cup of tea > (a technique I learned from an Englishman) and sat > down to try again. The problem had disappeared!! Could you maybe speculate as to what was/is happening? I was writing a response to the effect that I have seen this before too when saving and instance of IDLgrModel which contains IDLgrGraphic atoms. When I destroyed the model, (some of? all of? can't remember) the atoms were left strewn about. My "fix" was to extract the atoms and then save them. When restoring I add them to a fresh instance of IDLgrModel. I am curious as to why this is... -Rick
creator: Rick Towler
date: 2004-03-03T01:36:50-00:00

Re: Object Madness or Restoring Nightmares
type: item
description: Rick Towler writes: > Could you maybe speculate as to what was/is happening? > > I was writing a response to the effect that I have seen this before too when > saving and instance of IDLgrModel which contains IDLgrGraphic atoms. When I > destroyed the model, (some of? all of? can't remember) the atoms were left > strewn about. > > My "fix" was to extract the atoms and then save them. When restoring I add > them to a fresh instance of IDLgrModel. > > I am curious as to why this is... Well, this gets curiouser and curiouser. I guess the problem is back. I don't know why I thought it had gone away. Wishful thinking, probably. Or maybe a time warp of some kind. Anyway... it's back. Now here is the thing. This is a large application. *Everything* is an object including the "application" object. So, the object that I am saving is a small object. It contains three other container objects inside it, and one of those containers contains three image objects. So, all together, maybe a dozen objects and pointers. Now, recall that I save it like this from within a SaveSession method: theStudy = self.currentStudy Save, theStudy, Filename='somename.sav' I restore the object like this in a RestoreSession method. Obj_Destroy, self.currentStudy Restore, Filename='somename.sav' self.currentStudy = theStudy I can get to the same place in the program either by running a new study, in which I read some data files, etc. Or by restoring a study. The same objects are created. The interface looks identical. I can destroy the main object at this point. If I read the data to get here, I am completely clean. If I restored the small study object to get here, I have--are you ready for this-- 908 pointers and 784 objects left on the heap!!!!! If I look at these objects and pointers I notice that my entire application is left on the heap, even though I have just destroyed it! In fact, the very first object on the heap is my main program object. How can this be? Here is a clue that may be too grueso…
creator: David Fanning
date: 2004-03-03T02:52:40-00:00

Re: Object Madness or Restoring Nightmares
type: item
description: David Fanning writes: > Well, this gets curiouser and curiouser. OK, another little data point. Just before I restore the save file where I put my simple object, I check to see how many objects I have: objects = Obj_Valid() I have 306 of them. That sounds right. (Big application.) Now I restore my simple object and check again: Restore, Filename='somename.sav' nowObjects = Obj_Valid() I have 1509 of them! Whoa! What in the world is going on here!? Cheers, David -- David Fanning, Ph.D. Fanning Software Consulting Coyote's Guide to IDL Programming: http://www.dfanning.com/
creator: David Fanning
date: 2004-03-03T03:23:15-00:00

Re: Object Madness or Restoring Nightmares
type: item
description: David Fanning writes: > OK, another little data point. Just before I restore > the save file where I put my simple object, I check > to see how many objects I have: > > objects = Obj_Valid() > > I have 306 of them. That sounds right. (Big application.) > Now I restore my simple object and check again: > > Restore, Filename='somename.sav' > nowObjects = Obj_Valid() > > I have 1509 of them! Whoa! What in the world is going on > here!? OK, are you ready for this? Are you sitting down? What you have heard about how easy it is to save and restore objects may be about to change. My application has one large data set, which is stored in an ImageCube object. The data is about 3Mbyte in size. The StudyObject, which I have been trying all day to save, contains a reference to the ImageCube data, but not the data itself. By itself, it is small, maybe 100K or so. OK, so I save the StudyObject. It obviously has a reference to the ImageCube object, so IDL (helpfully, I think) saves the ImageCube object, too. But my application is an object hierarchy. Everything is more or less connected to everything else. It's a family tree, for God's sake. We are *all* related. IDL realizes this, thinks it is being helpful, and saves every object in sight! And although I can't prove it, I think it saves two backup copies as well, because the entire save file tops out at a hefty 10 MBytes. OK, so then I go to restore the small study object. IDL restores *all* the objects it saved (even though most of them are truly orphan object (widgets and what-not that I used last Friday and which are of absolutely no use to me now). And it also restores its two backup copies, because no one can accuse IDL of not being thorough! So I have 310 objects before the restore, and 1513 after the restore. There is a difference of 1203 objects. But I can recover these! If I use the RESTORED_OBJECTS keyword on the RESTORE command, I can get the references to the 1203 "extra" objects. (I am adrift in a *sea* of objects!)…
creator: David Fanning
date: 2004-03-03T04:50:02-00:00

Re: Object Madness or Restoring Nightmares
type: item
description: David Fanning <david@dfanning.com> writes: > IDL realizes this, thinks it is being helpful, and saves every > object in sight! And although I can't prove it, I think it > saves two backup copies as well, because the entire save file > tops out at a hefty 10 MBytes. Craigbot says: I think IDL has a cycle counting bug. If your objects are doubly- (or morely-) linked, then I'm guessing that IDL is trying to resolve the cycles, but fails. I bet if you try a simpler data structure, one without cycles, it will save fine. But robots don't guess or bet, so I must have a logic error. Craig-bot -- ------------------------------------------------------------ -------------- Craig Bot Markwardt, Ph.D. EMAIL: craigmnet@REMOVEcow.physics.wisc.edu Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response ------------------------------------------------------------ --------------
creator: Craig Markwardt
date: 2004-03-03T08:22:37-00:00

Re: Object Madness or Restoring Nightmares
type: item
description: >>>> > "David" == David Fanning <david@dfanning.com> writes: > Folks, Don't you just hate it when you think you understand > something, only to realize (usually at a critical time) > that you don't? > Maybe I've been doing too much programming and not playing > enough tennis lately, but I feel t-i-r-e-d. Good thing the > ol' physical is tomorrow. Maybe I ask for some of those > tiny blue programming pills. :-) Take the red pill, David.
creator: mmiller3
date: 2004-03-03T16:09:45-00:00

Re: Object Madness or Restoring Nightmares
type: item
description: Me echoes Craig. I never had this sort of thing happen with simple linear hierarchies. Then again, my programming style is so clean and flawless (as opposed to David's convoluted self-linked mess), what else do you expect :-) Regarding Mac trash objects: check out this URL: http://www.bushin30seconds.org/view/10_large.shtml Uh, the ever so useful Trash. Pavel David Fanning wrote: > > David Fanning writes: > >> OK, another little data point. Just before I restore >> the save file where I put my simple object, I check >> to see how many objects I have: >> >> objects = Obj_Valid() >> >> I have 306 of them. That sounds right. (Big application.) >> Now I restore my simple object and check again: >> >> Restore, Filename='somename.sav' >> nowObjects = Obj_Valid() >> >> I have 1509 of them! Whoa! What in the world is going on >> here!? > > OK, are you ready for this? Are you sitting down? > What you have heard about how easy it is to save > and restore objects may be about to change. > > My application has one large data set, which is stored > in an ImageCube object. The data is about 3Mbyte in size. > The StudyObject, which I have been trying all day to save, > contains a reference to the ImageCube data, but not the > data itself. By itself, it is small, maybe 100K or so. > > OK, so I save the StudyObject. It obviously has a reference > to the ImageCube object, so IDL (helpfully, I think) saves > the ImageCube object, too. But my application is an object > hierarchy. Everything is more or less connected to everything > else. It's a family tree, for God's sake. We are *all* related. > > IDL realizes this, thinks it is being helpful, and saves every > object in sight! And although I can't prove it, I think it > saves two backup copies as well, because the entire save file > tops out at a hefty 10 MBytes. > > OK, so then I go to restore the small study object. IDL > restores *all* the objects it saved (even though most of > them are truly orphan object (widgets and what-not t…
creator: Pavel Romashkin
date: 2004-03-03T16:40:27-00:00

Re: Object Madness or Restoring Nightmares
type: item
description: On Wed, 03 Mar 2004 02:22:37 -0600, Craig Markwardt wrote: > > David Fanning <david@dfanning.com> writes: > >> IDL realizes this, thinks it is being helpful, and saves every >> object in sight! And although I can't prove it, I think it >> saves two backup copies as well, because the entire save file >> tops out at a hefty 10 MBytes. > > Craigbot says: I think IDL has a cycle counting bug. If your objects > are doubly- (or morely-) linked, then I'm guessing that IDL is trying > to resolve the cycles, but fails. I bet if you try a simpler data > structure, one without cycles, it will save fine. > > But robots don't guess or bet, so I must have a logic error. > > Craig-bot I think it's actually simpler than that. I suspect that if you follow the train of objects containing pointers to objects with pointers etc., you'll find that some object somewhere beneath your "theStudy" object actually has a pointer or object reference to the top-level application object in it. This is very common (just keeping track of that object to consult it later on). If this is the case, here's what happens: IDL very dutifully follows all of these downward-linking object/pointer chains, collecting and saving everything it finds on the way. This is the correct thing to do, since, as far as it knows, to have a valid "theStudy" object on disk requires all of its various holdings. Now, if at some point down the chain, IDL runs into an object which is just a convenience reference to the top level application object, it will dutifully jump right to the top of the heap and start saving the whole thing. It does know better than to save heap variables twice and how to avoid circular references; you can test this with: IDL> b=obj_new('IDL_CONTAINER') IDL> b->Add,b IDL> save,b,FILENAME='~/b.sav' < new session > IDL> help,/HEAP & restore,'~/b.sav',RESTORED_OBJECTS=r & help,/HEAP Heap Variables: # Pointer: 0 # Object : 0 Heap Variables: # Pointer: 1 # Object : 1 <ObjHeapVar1> STRUCT = -> IDL_CONTAINER Arr…
creator: JD Smith
date: 2004-03-03T17:24:10-00:00

Re: Object Madness or Restoring Nightmares
type: item
description: JD Smith writes: > I think it's actually simpler than that. I suspect that if you follow > the train of objects containing pointers to objects with pointers > etc., you'll find that some object somewhere beneath your "theStudy" > object actually has a pointer or object reference to the top-level > application object in it. OK, I like this theory. Here is my problem. (Those of you whose eyes are already glazing over are excused. You can read the Executive Summary to follow in a couple of days.) My study object contains three object references: one to the imageCube object that exists at the main application level and which is stored as a field in the top-level object, and two to IDL container objects. One container is empty, so don't worry about it. The other container contains ImageSlice objects, which are abstractions simply to draw a particular slice of the imageCube object. As such, they also contain references to the original imageCube object. The only other object an ImageSlice contains is a WindowIndex number reference to a drawWidgetObject, that is a child of the "self" or main object. This may be your connection back to the main object. OK, before saving the studyObject I null out the ImageCube reference, and I go through both containers, get all the ImageSlice objects and null out their imageCube references and their WindowIndex references. The imageSlice objects now don't point to anything, and the studyObject only has references to ImageSlice objects that should be nulled out. So now, I get the original data from the ImageCube object everyone has been pointing to, and I save it *along* with the the studyObject in a save file. (I need the data, that is the most important part of a "study".) Are you ready? I am chagrined to see that my save file (which you remember should have been about 3 MB and was already a bloated 10 MB) is now nearly 13 MB!. And, I *still* have 1203 unnecessary objects stored in my save file. I take it my save file is now all the object…
creator: David Fanning
date: 2004-03-03T20:26:36-00:00

Re: Object Madness or Restoring Nightmares
type: item
description: JD Smith wrote: > > [Long and thorough explanation of object saving & restoring issues] > > Good luck, > > JD <irony> Aww come on, JD, you haven't left anything for anyone else to say! </irony> PS. My spelling checker wanted to change "JD" to "CJD". Not entirely appropriate IMHO. -- Mark Hadfield "Ka puwaha te tai nei, Hoea tatou" m.hadfield@niwa.co.nz National Institute for Water and Atmospheric Research (NIWA)
creator: Mark Hadfield
date: 2004-03-03T20:29:53-00:00

Re: Object Madness or Restoring Nightmares
type: item
description: On Thu, 04 Mar 2004 09:29:53 +1300, Mark Hadfield wrote: > JD Smith wrote: >> >> [Long and thorough explanation of object saving & restoring issues] >> >> Good luck, >> >> JD > > <irony> > Aww come on, JD, you haven't left anything for anyone else to say! > </irony> > > PS. My spelling checker wanted to change "JD" to "CJD". Not entirely > appropriate IMHO. Yes, but I didn't mention object transmogrification. Talk amongst yourselves. JD
creator: JD Smith
date: 2004-03-03T21:54:29-00:00

Re: Object Madness or Restoring Nightmares
type: item
description: On Wed, 03 Mar 2004 13:26:36 -0700, David Fanning wrote: > JD Smith writes: > >> I think it's actually simpler than that. I suspect that if you follow >> the train of objects containing pointers to objects with pointers >> etc., you'll find that some object somewhere beneath your "theStudy" >> object actually has a pointer or object reference to the top-level >> application object in it. > > OK, I like this theory. Here is my problem. (Those of you whose > eyes are already glazing over are excused. You can read the > Executive Summary to follow in a couple of days.) > > My study object contains three object references: one to the imageCube > object that exists at the main application level and which is stored > as a field in the top-level object, and two to IDL container objects. > One container is empty, so don't worry about it. The other > container contains ImageSlice objects, which are abstractions simply > to draw a particular slice of the imageCube object. As such, they > also contain references to the original imageCube object. The only > other object an ImageSlice contains is a WindowIndex number reference > to a drawWidgetObject, that is a child of the "self" or main object. > This may be your connection back to the main object. > > OK, before saving the studyObject I null out the ImageCube reference, > and I go through both containers, get all the ImageSlice objects and > null out their imageCube references and their WindowIndex references. > The imageSlice objects now don't point to anything, and the studyObject > only has references to ImageSlice objects that should be nulled out. > > So now, I get the original data from the ImageCube object everyone has > been pointing to, and I save it *along* with the the studyObject in > a save file. (I need the data, that is the most important part of a > "study".) > > Are you ready? I am chagrined to see that my save file (which you > remember should have been about 3 MB and was already a bloated > 10 MB) is now ne…
creator: JD Smith
date: 2004-03-03T22:15:02-00:00

Re: Object Madness or Restoring Nightmares
type: item
description: JD Smith writes: > I wouldn't say you're worse off, just the same (since you added > something else to the save as well). I think you'll eventually find > it. Possibly, but since I'm making about 37 cents an hour on this project now (and it may be closer to a nickel by the time I'm finished with it), I really can't afford more time to look. My personal opinion is there is a bug somewhere. But in the meantime, I've turned the damn study into a structure, saved and restored the structure, and all is well with the world again. Did I mention I love objects. :-( > Part of the reason I communicate among objects using messages is > to avoid problems like this: instead of having to hold a reference to > an object to query it for properties on occasion, you can just > subscribe to messages which give you the relevant properties and the > right time (i.e. property push not pull). Since the messages are > dealt with as they come, they are ephemeral, and don't stick around to > cause problems like this. Well, I use messages too, but those are even more magical than objects and I find it *really* hard to keep track of who, when, and where when I use them. Might as well go back to prayer for all the difficulty I have understanding what's going on. Cheers, David -- David Fanning, Ph.D. Fanning Software Consulting Coyote's Guide to IDL Programming: http://www.dfanning.com/
creator: David Fanning
date: 2004-03-03T23:11:27-00:00

Re: Object Madness or Restoring Nightmares
type: item
description: JD Smith writes: > I wouldn't say you're worse off, just the same (since you added > something else to the save as well). I think you'll eventually find > it. Oh, dear, you may be right. :-( There was one little coordinate object stuck *way* down there somewhere that probably had a connection back to the surface. In fact, two, now that I think about it. Fishing those out is going to be trouble. Oh, my goodness...I can see my solution unravelling again. I *need* objects. I'm committed to objects! This is so unbelievably ugly. When's that flight to the arctic? I may not come back. :-( Cheers, David -- David Fanning, Ph.D. Fanning Software Consulting Coyote's Guide to IDL Programming: http://www.dfanning.com/
creator: David Fanning
date: 2004-03-03T23:53:25-00:00

Re: Object Madness or Restoring Nightmares
type: item
description: Michael A. Miller writes: > Take the red pill, David. Whoa! I don't know if you are as tired of this thread as I am, but after that pill I just don't friggin care! :-) Here is the latest for JD to cogitate over. One of what I used to think of as the "advantages" of my Catalyst Library is that it is an object hierarchy. If you "draw" the top-level object, all the objects below get "drawn" automatically. This means widgets appear, images get drawn in windows, coordinate systems get set up, etc. Neat. Similarly, if you "destroy" the top-level object, all the objects below in the hierarchy get destroyed. No memory leaks, no great effort involved. Very, very neat. But now this "feature" of my library has become a thorn in my side. (Don't worry, I'm not going to push the Mel Gibson imagery much beyond this.) If I save just one object in my save file from this "web" of objects, *all* my objects are saved. I guess that makes sense, they are all connected. But it is enormously inconvenient for me and means I can't use one of the best features of objects: the ability to store the current state of a process or operation. Any ideas for getting out of this mess? (I see over a year of effort and very little income beginning to consolidate as a black cloud just over my right shoulder. Oh wait. That's my wife glaring at me with what look like divorce papers in her hip pocket.) Cheers, David P.S. Let's just say if you had some kind of an upper-type of pill, I would be interested. -- David Fanning, Ph.D. Fanning Software Consulting Coyote's Guide to IDL Programming: http://www.dfanning.com/
creator: David Fanning
date: 2004-03-04T02:44:19-00:00

Re: Object Madness or Restoring Nightmares
type: item
description: David Fanning writes: > One of what I used to think of as the "advantages" of my > Catalyst Library is that it is an object hierarchy. If > you "draw" the top-level object, all the objects below > get "drawn" automatically. This means widgets appear, > images get drawn in windows, coordinate systems get set > up, etc. Neat. > > Similarly, if you "destroy" the top-level object, all the > objects below in the hierarchy get destroyed. No memory > leaks, no great effort involved. Very, very neat. Ah, here is the thing about this hierarchy that you should know. This is an object *containment* hierarchy. The top-level object is a container that holds all the other objects. Every object (except the top object) is both contained in a container and can (potentially) contain other objects. (All objects in my system inherit IDL_CONTAINER.) If you pick any object whatsoever out of this web, you can (apparently easily to judge from how fast IDL does it) traverse the entire object hierarchy. I can see that this is the reason IDL *must* save everything when I save even a single object that belongs in the hierarchy. What I can't see at the moment is a way out of this mess. Cheers, David -- David Fanning, Ph.D. Fanning Software Consulting Coyote's Guide to IDL Programming: http://www.dfanning.com/
creator: David Fanning
date: 2004-03-04T05:06:13-00:00

Re: Object Madness or Restoring Nightmares
type: item
description: David Fanning wrote: > David Fanning writes: ...> > Ah, here is the thing about this hierarchy that you > should know. This is an object *containment* hierarchy. > The top-level object is a container that holds all the > other objects. Every object (except the top object) is > both contained in a container and can (potentially) > contain other objects. (All objects in my system > inherit IDL_CONTAINER.) > > If you pick any object whatsoever out of this web, > you can (apparently easily to judge from how fast > IDL does it) traverse the entire object hierarchy. > I can see that this is the reason IDL *must* save > everything when I save even a single object that > belongs in the hierarchy. > > What I can't see at the moment is a way out of > this mess. > > Cheers, > > David Hi David, Forgive me if I'm asking stupid questions... (OK the if is superfluous!) Clearly each object contains pointers to all of its children so if you save the parent all the objects contained in it are saved. But I don't see why a child (still taking about the containment hierarchy, not the inheritance tree) needs to point to its parent? Where is that pointer coming from and what is it doing? I gather that each object needs to point to the class definition of the top level container since that's also the class definition of the root of the inheritance tree, but I wouldn't have thought that saving the definition of the class means that you have to save every instance of the class. That would certainly seem like a broken implementation for the SAVE functionality. Regards, Tom McGlynn
creator: tam
date: 2004-03-04T14:58:52-00:00

Re: Object Madness or Restoring Nightmares
type: item
description: On Wed, 03 Mar 2004 22:06:13 -0700, David Fanning wrote: > David Fanning writes: > >> One of what I used to think of as the "advantages" of my >> Catalyst Library is that it is an object hierarchy. If >> you "draw" the top-level object, all the objects below >> get "drawn" automatically. This means widgets appear, >> images get drawn in windows, coordinate systems get set >> up, etc. Neat. >> >> Similarly, if you "destroy" the top-level object, all the >> objects below in the hierarchy get destroyed. No memory >> leaks, no great effort involved. Very, very neat. > > Ah, here is the thing about this hierarchy that you > should know. This is an object *containment* hierarchy. > The top-level object is a container that holds all the > other objects. Every object (except the top object) is > both contained in a container and can (potentially) > contain other objects. (All objects in my system > inherit IDL_CONTAINER.) > > If you pick any object whatsoever out of this web, > you can (apparently easily to judge from how fast > IDL does it) traverse the entire object hierarchy. > I can see that this is the reason IDL *must* save > everything when I save even a single object that > belongs in the hierarchy. > > What I can't see at the moment is a way out of > this mess. Why not implement a set of methods in your top-level which leverages the inherent connectedness to detach unnecessary objects before saving? The only technical problem is where to stick the detached objects while you save (you can't stick them somewhere else in the object: you'll be back to the same problem). This can be accomplished by dynamically building a list of objects and their detached components and propagating it all the way up to a variable at the top stack level at the time of the call. Something like this: pro topClass::Detach, detachlist, RECORD=rec if obj_valid(self.parent) then begin ;; Add to or create this object's "detached" record if n_elements(rec) gt 0 && ptr_valid(rec.Detached) then $ …
creator: JD Smith
date: 2004-03-04T17:57:18-00:00