comp.lang.idl-pvwave archive: archive » Re: slow processing of my k-nearest neighour code

Home » Public Forums » archive » Re: slow processing of my k-nearest neighour code

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: slow processing of my k-nearest neighour code [message #49760]

Tue, 15 August 2006 08:14

James Kuyper
Messages: 425
Registered: March 2000

Senior Member

humphreymurray@gmail.com wrote:
> Wow, that's a great idea to remove most of the code from the loops.
> The only problem is that the code doesn't run on data of any usable
> size. I get a "Unable to allocate memory: to make array" error on this
> line of code:
>
> training_duplicates = REBIN(TRANSPOSE(training_data), $
> num_attributes, num_training_elements, num_testing_elements)
>
> This code works fine with really small data, but when I'm trying to
> classify a 256x256 pixel image, it is trying to create an array of
> dimensions: [15, 400, 65536]. According to my math, this would be
> about 390 million elements, and assuming that each element takes 1 byte
> of memory, it would use 390mb of ram. My machine at uni only has 512mb
> of ram, so I will try this code at home tonight, where I have 1gb of
> ram.
>
> Would the way to fix this problem be to split the number of training
> pixels up, and process them in small groups? For example, analyse a
> row of pixels at a time?

Yes, one of the key limitations of this technique is that it uses extra
memory to achieve faster processing speeds (at least, it should process
faster: I'd recommend doing some performance testing with your actual
data, to make sure). Breaking up the full test data set into smaller
sub-sets is exactly the right way to deal with this problem. I would
have mentioned that if I'd realized that you were working with arrays
that were big enough for that to be a problem.

Report message to a moderator

[Message index]

		Re: slow processing of my k-nearest neighour code By: James Kuyper on Tue, 15 August 2006 08:14
		Re: slow processing of my k-nearest neighour code By: humphreymurray on Mon, 14 August 2006 19:17
		Re: slow processing of my k-nearest neighour code By: humphreymurray on Mon, 14 August 2006 19:07
		Re: slow processing of my k-nearest neighour code By: JD Smith on Mon, 14 August 2006 14:39
		Re: slow processing of my k-nearest neighour code By: James Kuyper on Mon, 14 August 2006 13:50
		Re: slow processing of my k-nearest neighour code By: btt on Mon, 14 August 2006 09:01
		Re: slow processing of my k-nearest neighour code By: news.verizon.net on Mon, 14 August 2006 08:30
		Re: slow processing of my k-nearest neighour code By: Karl Schultz on Mon, 14 August 2006 08:08
		Re: slow processing of my k-nearest neighour code By: btt on Mon, 14 August 2006 07:11
		Re: slow processing of my k-nearest neighour code By: humphreymurray on Mon, 14 August 2006 06:50

Previous Topic:	Help with GRID_TPS
Next Topic:	Re: SHMMAP and structures

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Thu Oct 09 07:49:56 PDT 2025

Total time taken to generate the page: 1.55756 seconds