comp.lang.idl-pvwave archive - RDF feed
type: channel
description: Messages from Usenet group comp.lang.idl-pvwave, compiled by Paulo Penteado
items: Re: Object overhead

Re: Object overhead
type: item
description: "Pavel A. Romashkin" <pavel.romashkin@noaa.gov> writes: > I noticed that the size of an object array, where each object only has > one empty poiter field, is 4 times larger than the size of an empty > pointer array of the same length. PTRARR(1000), when saved, takes about > 44 Kb. OBJARR(1000) of {junk, data:PTR_NEW()} takes 176 Kb. Why is that? > Also, it takes twice the time (0.79 vs 0.44 s) to save an object array > than the pointer array. It can not have to do with the volume of saved > data because it takes 0.05 s to save an 80 Kb FLTARR(20000). Hi Pavel-- Since I've mucked around with the format of save files, and in fact made a library to manipulate them, I think I can try to answer your questions. Saving a FLTARR() is fast because IDL can simply write out a block of floating point numbers in one operation. [ after converting to standard endian-ness, of course ] Saving a PTRARR() is slower, because IDL must scan through the entire array looking for non-null pointers. You actually gave an example that was too trivial. If you tried populating your PTRARR(1000) with values, you will find that the save file becomes much larger and take much longer to create. That's because IDL must save each heap variable separately. This is invisible to you, but it happens. Finally, an array of objects is more complicated still. Objects are really structures. Since structures are more or less infinitely flexible, much more complicated programming is used to save them, and there is extra metadata information related to structures, that must also be saved. Vector support for structures and pointers has never been as strong as for simple data types, and this shows in how they are save as well. Hope this helps! Craig -- ------------------------------------------------------------ -------------- Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response ----------------------------------------------------…
creator: Craig Markwardt
date: 2001-09-22T14:50:46-00:00

Re: Object overhead
type: item
description: Thanks, Craig! This kind of helps. Kind of, because you confirm that objects will then always have a limitation: if you have repetitive data that would benefit from objectifying, think twice if you are going to have a lot of them, or you will slow down to a crawl. However, for small numbers, the convenience of objects can't be beat. Thanks again, Pavel P.S. When I say "a lot of them", I think it depends on what you are running on. For my G4, I notice a difference when the number of data objects reaches 5k. And I hoped to be able to convert over 100k... I think I can come up with a way to use objects for processing but store and save data as simple arrays. Craig Markwardt wrote: > > Hi Pavel-- > > Since I've mucked around with the format of save files, and in fact > made a library to manipulate them, I think I can try to answer your > questions. > > Saving a FLTARR() is fast because IDL can simply write out a block of > floating point numbers in one operation. [ after converting to > standard endian-ness, of course ] > > Saving a PTRARR() is slower, because IDL must scan through the entire > array looking for non-null pointers. You actually gave an example > that was too trivial. If you tried populating your PTRARR(1000) with > values, you will find that the save file becomes much larger and take > much longer to create. That's because IDL must save each heap > variable separately. This is invisible to you, but it happens. > > Finally, an array of objects is more complicated still. Objects are > really structures. Since structures are more or less infinitely > flexible, much more complicated programming is used to save them, and > there is extra metadata information related to structures, that must > also be saved. > > Vector support for structures and pointers has never been as strong as > for simple data types, and this shows in how they are save as well. > > Hope this helps! > Craig > > -- > ------------------------------------------------------------ ------------…
creator: Pavel A. Romashkin
date: 2001-09-24T15:32:18-00:00

Re: Object overhead
type: item
description: "Pavel A. Romashkin" wrote: > > I noticed that the size of an object array, where each object only has > one empty poiter field, is 4 times larger than the size of an empty > pointer array of the same length. PTRARR(1000), when saved, takes about > 44 Kb. OBJARR(1000) of {junk, data:PTR_NEW()} takes 176 Kb. Why is that? > Also, it takes twice the time (0.79 vs 0.44 s) to save an object array > than the pointer array. It can not have to do with the volume of saved > data because it takes 0.05 s to save an 80 Kb FLTARR(20000). > I thought of converting a data processing program to objects but found > that storage space required almost doubles (well, partially due to some > redundancy in object fields) and save/restore time increased tenfold. > For storing many thousands of data objects, this matters. > Thank you, > Pavel Why don't you compare: ptrarr(1000) to {junk_struct,data:ptrarr(1000)} to obj_new('junk') with pro junk__define st={JUNK,data:ptrarr(1000)} } I'd suspect they'll be close, especially the last two. The basic issue is that arrays *of* structures, and arrays *in* structures must necessarily receive quite separate treatment, because structures are themselves such complex beasts. Also, as Craig pointed out, this is an unfair test in itself, since the overhead of pointer arrays will be seen only when you actually attach them to some data, and will then likely dominate performance and size. As an example of this for the size issue, I tried: IDL> a=ptrarr(1000) & for i=0,999 do *a[i]=findgen(3*i+1) saving "a" yields 6107296 bytes on disk. IDL> a=replicate({JUNK,data:ptr_new()},1000) IDL> for i=0,999 do a[i].data=ptr_new(findgen(3*i+1)) which is 6107340 bytes on disk. Almost exactly the same! Then I tried: function KNUJ::Init, data self.data=ptr_new(data) return,1 end pro KNUJ__define struct={KNUJ,data:ptr_new()} end IDL> a=objarr(1000) & for i=0,999 do a[i]=obj_new('KNUJ',findgen(3*i+1)) which is 6239328 on disk. I.e., around a 2% savings for the first two me…
creator: John-David T. Smith
date: 2001-09-24T17:26:05-00:00