| histogram and UINT [message #70128] |
Wed, 17 March 2010 10:33 |
Jeremy Bailin
Messages: 618
Registered: April 2008
|
Senior Member |
|
|
I've discovered an interesting "feature" in histogram... I'm sure it's
the expected behaviour, but in this case it had a consequence that
bit me.
I have some unsigned integer input data that I want to histogram. I'm
looking at things right around the saturation level, so most of the
data is exactly 65535, but I also want a bin past that. The following
works:
h = histogram(data, min=65500ul, max=65535ul)
But the following doesn't:
h = histogram(data, min=65500ul, max=65536ul)
% HISTOGRAM: Illegal binsize or max/min.
What seems to be happening is that the min/max values are cast into
the same data type as data, in this case giving a nonsensical max of
0u. To do what I want, I need to cast data into a ULONG in this
histogram call... or use max=65535u and append an appropriate number
of 0s onto h.
-Jeremy.
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
