| BYTSCL bug!? [message #65459] |
Tue, 03 March 2009 09:05 |
spidersapiens
Messages: 6
Registered: June 2006
|
Junior Member |
|
|
I've recently been converting some IDL code to Java and found this
problem: BYTSCL returns different results based on the data type you
pass in. For example, the two lines below return different results for
9065.
print, bytscl([4124, 9065, 18946], top = 2)
print, bytscl(FLOAT([4124, 9065, 18946]), top = 2)
That's fine. Later on I found the explanation in IDL's documentation
"For floating-point input, each value is scaled using the formula (Top
+ 0.9999)*(x - Min)/(Max - Min). For integer input, each value is
scaled using the formula ((Top + 1)*(x - Min) - 1)/(Max - Min). ".
No problem, I can use their formulas. But the bad thing somehow
they're not using the formula for integer to do the actual
calculation. Here's the evidence
print, bytscl([4124, 9065, 18946], top = 2) gives me 0 on 9065.
But if you use their formula for compute it, ((2 + 1) * (9065 - 4124)
- 1) / (18946 - 4124) gives me 1.
This is frustrating me a lot because I need to duplicate the exactly
same numeric results in Java. But with this buggy BYTSCL, it's almost
impossible to duplicate its behavior. Anybody has any idea what
formula they're using internally?
Thanks!
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
