comp.lang.idl-pvwave archive: archive » division to move a decimal point

Home » Public Forums » archive » division to move a decimal point

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: division to move a decimal point [message #74198 is a reply to message #74068]

Sun, 02 January 2011 18:09

M. Katz
Messages: 69
Registered: May 2005

Member

Try double-precision, indicated by a 'd' rather than just a '.'

IDL> print, 1948005/100d
19480.050

When dividing this long integer by a double-precision floating-point
value, the calculation is performed in double-precision,
and you get a few more significant digits than with single-precision
floating-point, which was causing your rounding.

M.

Report message to a moderator

[Message index]

		division to move a decimal point By: Giuseppe Papa on Fri, 31 December 2010 09:13
		Re: division to move a decimal point By: M. Katz on Sun, 02 January 2011 18:09

Previous Topic:	screen capture of xplot3d
Next Topic:	Re: How to extract pixel values from a GeoTIFF using an Esri Shapefile

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sat Oct 11 00:04:07 PDT 2025

Total time taken to generate the page: 0.51712 seconds