Bitwise and Logical Operations

QUESTION: Can IDL do bitwise operations? Can it do logical operations? Is its behaviour logical?

ANSWER: Yes. Yes. Not entirely.

[This article was prepared by brave volunteer, Mark Hadfield. Mark is a long-time IDL user and newsgroup contributor from New Zealand. If you would like to write an article, please contact the Article Chairman.]

Many programming languages (eg. Fortran) provide a logical data type and a set of logical operators. For example, Fortran logical variables can take values ..TRUE. and .FALSE. and support the logical operators .AND., .OR. and .NOT., according to the rules of Boolean arithmetic. So, for example

(.TRUE. .AND. ..FALSE.) evaluates to .FALSE.

(.TRUE. .OR. ..FALSE.) evaluates to .TRUE.

(.NOT. .TRUE.) evaluates to .FALSE.

IDL does not have a logical data type as such, but it allows you to use other data types to represent logical values. Most IDL users are aware that integers are often used in this way. (By they way, I'm using the word "integer" here to cover all the different integer types supported by IDL: signed and unsigned, 8, 16, 32 and 64 bit.) Here is the official rule:

Odd integers are true and even integers are false.

But many users are unaware (well, I was) that other data types are also considered to be true or false.

For floating point numbers, zero is false and all other values are true.
Complex numbers follow the same rule as floating point numbers, with the imaginary part ignored.
For strings, the empty string is false and all others are true.

Now I can't see why anyone would use a non-integer data type to represent a logical value, but—just in case you come across code where this has been done—you might want to remember that the rule for floating point numbers is different from the rule for integers. For example:

   IDL> if 2 then print, 'true' else print, 'false'
   false
   IDL> if 2.0 then print, 'true' else print, 'false'
   true

OK, so what about the logical operators? IDL has logical operators AND, OR and NOT, which act on logical values just as you would expect. Well, not quite, because they don't accept string or complex operands and their behaviour is a little weird when you start mixing integer and floating point operands. For example:

   IDL> print, 2 or 0.
         2.00000

What's this? By the rules, 2 is false and 0. is false, but the result is 2., which is true. I think what happens is that the 2 (false) gets silently converted to 2. (true) before the "or" is taken. Yuk! For me the lesson is to use only integers to represent logical values.

For completeness, I should mention the XOR (exclusive or) operator, which you can read about in the manual. For reasons that escape me, it doesn't accept floating point operands.

The fact that there is more than one integer value that can represent "true", and more than one that can represent "false", can get a little confusing. IDL functions conventionally return 1 for true and 0 for false. So do comparison operators. (The data type can be a byte, short or long integer at the whim of the programmer.) If we feed 0s and 1s in any combination to the AND, OR and XOR operators then the result is always a 0 or a 1. But the NOT operator is more creative:

   IDL> print, not 0, not 1
         -1      -2

Now these results are correct in a logical sense, because -1 (odd) is true and -2 (even) is false. But what if we have a couple of logical expressions, and we want to test if they return results that are logically equivalent, ie. they are both false or both true? Fortran has an .EQV. operator that does just this, but IDL does not. You can't safely use the integer comparison operator, EQ, because two integers that are logically equivalent may not be equal. In this case you can get the result you want with the NOT and XOR operators—I'll leave it as an exercise to work out how.

But how on earth does (not 0) evaluate to -1? The answer is that AND, OR, NOT and XOR and bitwise operators (for integers, though not for floating point values, but let's ignore that for now). In other words, they operate separately on each of the bits that make up the integer. For example, the binary representation of a 16-bit zero is 0000000000000000. The NOT operator flips each binary digit, so (not 0) in binary is 1111111111111111, which corresponds to the signed integer -1. In these terms the rule that odd integers are true and even integers are false means that the logical value of an integer depends only on its rightmost (least significant) bit. All the other bits are just along for the ride.

I won't go into bitwise operations much except to note that you do all sorts of clever things with them. For example the !D system variable stores 18 separate logical values in a long integer in its FLAG tag. Bit 4 specifies whether the device supports color, and the following code tests it.

   IDL> if (!d.flags and 2^4) ne 0 then print, 'Colorful' else print, 'Colorless'
   Colorful

But really, this sort of thing should be left to the C programmers.

OK, so IDL supports logical values (implemented as integers) and logical operators (implemented as bitwise operators). This is self-consistent, though a little peculiar. Are there any pitfalls? You've guessed it, yes there are. There is a very important IDL function that is often used to process logical expressions, but doesn't respect the "even is false, odd is true" rule. That function is WHERE. Here is an example that illustrates the problem. Suppose we have a floating array in which some of the values may be NaN. We want to replace each of these NaNs with a large, finite number. (Why do we want to do this? Oh, I don't know, perhaps we want to write the data to a file format that doesn't allow NaNs. I just wanted an example in which it is natural to use the NOT operator.) The recommended way to test for the presence of a NaN (or other non-finite value) is the FINITE function. So

   IDL> array = fltarr(5)  &  array [2] = !values.f_nan
   IDL> print, array 
        0.000000     0.000000          NaN     0.000000     0.000000
   IDL> for i=0,n_elements(array )-1 do if not finite(array [i]) then array [i] = 1.E36
   IDL> print, array 
        0.000000     0.000000 1.00000e+036     0.000000     0.000000

But hold it! Isn't using a FOR loop bad style, not to mention slow for large arrays? OK, so the conventional way to do this sort of thing in IDL uses the WHERE function. There are a couple of examples in the documentation for WHERE (but you may notice that neither of them uses the NOT operator). Our example becomes:

   IDL> array  = fltarr(5)  &  array [2] = !values.f_nan
   IDL> print, array 
        0.000000     0.000000          NaN     0.000000     0.000000
   IDL> nan = where(not finite(array ), count)
   IDL> if count gt 0 then array [nan] = 1.E36
   IDL> print, array 
    1.00000e+036 1.00000e+036 1.00000e+036 1.00000e+036 1.00000e+036

That's odd. Our test has flagged all the elements of array as non-finite. The reason is that the WHERE function returns the indices of all the non-zero elements in its input, not all the logically true elements. The WHERE function in our example is acting exactly as the documentation says it should, and so is the NOT operator, but I, for one, find the result surprising. A similar issue arises with the KEYWORD_SET function, which looks as if it should deal with logical input data but returns 1 for all non-zero values.

Ideally IDL would have a logical data type, with only two values, true and false. It doesn't, and retrofitting one would lead to all sorts of backwards-compatibility problems. Another possibility would be to use -1 for true, as is done in Basic, since (not 0) is -1 and (not -1) is 0. But the convention that functions and comparison operators return 1 for true is well established, so this would also lead to backwards-compatibility problems.

The problem can be avoided as follows. Consistently use 0 for false and 1 for true. Replace expressions involving NOT with logically equivalent expressions that always result in 0 or 1. Here are some alternatives to "not finite(array)":

   finite(array) eq 0
   1 - finite(array)
   (not finite(array)) and 1

Most IDL programmers seem to use the first form. To my mind it obscures the intention of the code somewhat, but I guess you get used to it. The second form is the one I've tended to use. The third form uses the fact that ANDing an integer with 1 sets all but the rightmost bit to 0. I've been toying with the idea of bundling the third form into a function called something like LNOT (logical not):

   function LNOT, x
      return, (not x) and 1
   end

Since NOT is a unary expression, it can be replaced with a function without harming readability too much; "not finite(array)" becomes

   lnot(finite(array))

Anyway, rewriting the test expression in any of these forms fixes the problem, for example

   IDL> array = fltarr(5)  &  array[2] = !values.f_nan
   IDL> print, array
        0.000000     0.000000          NaN     0.000000     0.000000
   IDL> nan = where(finite(array) eq 0, count)
   IDL> if count gt 0 then array[nan] = 1.E36
   IDL> print, array
        0.000000     0.000000 1.00000e+036     0.000000     0.000000

ACKNOWLEDGEMENTS: Thanks to David Fanning for inviting me to prepare this article (and sort out the issue for myself). The article draws on several discussions on comp.lang.idl-pvwave, with contributions from Craig Markwardt, J. D. Smith, William Thompson, and others.

Note: In IDL 6.0, RSI introduced the LOGICAL_PREDICATE compiler option. Setting this option with the COMPILE_OPT statement will cause any predicate expression to be FALSE if zero and TRUE if anything else. A predicate expression is an expression that is evaluated as "true" or "false" as part of a statement that controls programs execution (IF, WHILE, REPEAT, etc.).