Floating point calculation

Hi all,
I have a field ATFLV of type FLTP length 16 and decimals 16. This field has a value 1.000000000000000E+03. I want to multiply this value with 0.022 and take that value into another field let us say ATFLV1 of same type as ATFLV.
And also, I want the value in ATFLV1 to be taken into another field ATWRT of type Character and length 30.
PLease help. Waiting for replies. Thanks

Rich,
In your code what I am doing is:
data: ATFLV type cawn-ATFLV value '1.000000000000000E+03',
atflv1 type cawn-atflv,
atwrt type cawn-atwrt.
atflv1 = atflv * '.022'.
Now the value in atflv1 is '2.200000000000000E+01',
Now I want this value in the field ATWRT as '22'.
Please help.

Similar Messages

128-bit floating point calculations

I'm looking to buy a used SPARC. Which model is the oldest that provides
C or fortran 128-bit floating point calculations? Does every 64-bit CPU
have 128-bit quad real numbers?
Ron

The AMD 64-bit processor does not support 128-bit
floating numbers. I need a Sparc processor that
does.And yet your question has just what to do with Java -- why did you create a userid and post this question in the Java forums?
A response "Does every person have ten fingers?"
shows me that you don't know much about writting
computer programs that have more than 16 significant
decimal digits.I wouldn't infer that - but now that you've proven to be a whiny little snot, I'm sure everyone is just going to want to help you here.

Float Point Calculations

Hi,
When I do the following in abap.
-> Snip <-
data: e_faktor type f,
      a like drseg-menge value '1.100',
      b like drseg-menge value '2.200',
      c like drseg-menge value '3.300'.
e_faktor = ( a + b - c ).
write e_faktor.
-> Snip <-
the result is 4,4408920985006262E-16.
I would expect it to be 0.
Changing a = 2.2
         b = 2.2
         c = 4.4
Gives me 0 as expected
Am I missing something ?

Hi Adriaan,
?? just copied your code snippet in SAP ECC 6.0 SAP_BASIS     700 SAP_ABA     700 system:
FORM calc .
DATA:
    e_faktor TYPE f,
    a LIKE drseg-menge VALUE '1.100',
    b LIKE drseg-menge VALUE '2.200',
    c LIKE drseg-menge VALUE '3.300',
    d LIKE drseg-menge.
d        = ( a + b - c ).
e_faktor = ( a + b - c ).
SET COUNTRY 'DE'.
WRITE:/ e_faktor, d.
SET COUNTRY 'US'.
WRITE:/ e_faktor, d.
ENDFORM.                    " CALC
this results in
0,0000000000000000E+00            0,000
0.0000000000000000E+00            0.000
How did you proceed, what system are you working on?
Regards,
Clemens

Inconsistent Floating Point Math and NaNs on Windows Laptops?

All -
I am seeing some very strange inconsistent floating point calculations on Windows Laptops, and I am wondering if anyone has any ideas. Read on, as (to me!) it's very interesting....
I have attached a segment of our code, along with some sample output. Looking at the code and the output, it seems like it's totally impossible.
With supposedly non-NaN and non-infinite double values, I am seeing unrepeatable and inconsistent math - the below example only illustrates one such case where we're seeing this behavior.
If you look at the code below, you will see that I do things like:
double rhoYo = ... // some math
double rho = ... // exact same mathStrangely enough, I might get rhoYo = 1.51231231 etc and rho = NaN.
If I reverse those lines (vertically), then again, rhoYo comes out good and rho comes out NaN; however, this is unpredictable and inconsistent. If I project a source point, get a destination point with NaNs as a result, and project the source again, the second destination point may be just fine. Matter of fact, i can put a loop in the code such as:
      double rho = Double.NaN;
      for( int i = 0; i < 10; i++ )
        rho = my_earthRad * my_F / Math.pow(Math.tan(Math.PI/4.0 + latRad/2.0), my_n);
        if( ! Double.isNaN( rho ) )
          break;
        System.out.println("NaN'ed rho");
      }and rho will eventually become non-NaN (random # of iterations)!!
How's that possible? Our code might be tromping on memory somewhere, but this sure seems crazy to me, especially considering that
we're only using local variables. Anyone know of floating point errors on Windows Laptops?
With the exact same codebase, this behavior ONLY happens on Windows Laptops, including brand new Dells, old Dells, IBM, Intel and AMD chips (I've tried several ;-). It does NOT happen on Mac or Linux, including the Linux side of a Linux/Windows dual-boot (where it does happen with the Windows side). Even more strangely, it does NOT happen with Windows desktops. I have tried several 1.5.x JVMs, webstart vs no webstart, etc, to no avail. Always and only on Windows Laptops.
Please help.... ;-) and thanks in advance.
Sample code:
public class Projection
protected Point2D.Double _project(Point2D.Double srcPt, Point2D.Double dstPt) {
    final double my_degToRad = Math.PI / 180.0;
    final double my_originLon = -95.0;
    final double my_originLonRad = my_originLon * my_degToRad;
    final double my_originLat = 25.0;
    final double my_originLatRad = my_originLat * my_degToRad;;
    final double my_stdLat1 = 25.0;
    final double my_stdLat1Rad = my_stdLat1 * my_degToRad;
    final double my_earthRad = 6371.2;
    final double my_n = Math.sin( my_stdLat1Rad );
    final double my_F = Math.cos( my_stdLat1Rad ) * Math.pow( Math.tan( Math.PI / 4.0 + my_stdLat1Rad / 2.0 ), my_n ) / my_n;
    final double my_rhoZero = my_earthRad * my_F / Math.pow( Math.tan( Math.PI / 4.0 + my_originLatRad / 2.0 ), my_n );
    if ( Double.isNaN( my_n ) || Double.isNaN( my_F ) || Double.isNaN( my_rhoZero )) {
      return new Point2D.Double(Double.NaN, Double.NaN);
    if( Double.isNaN( srcPt.x ) || Double.isNaN( srcPt.y ) )
        System.out.println("======= _project received a srcPt with NaNs. Returning NaN point.");
        Point2D.Double nanPoint = new Point2D.Double();
        nanPoint.x = nanPoint.y = Double.NaN;
        return nanPoint;
    if( Double.isInfinite( srcPt.x ) || Double.isInfinite( srcPt.y ) )
        System.out.println("======= _project received a srcPt with isInfinite. Returning NaN point.");
        Point2D.Double nanPoint = new Point2D.Double();
        nanPoint.x = nanPoint.y = Double.NaN;
        return nanPoint;
    // Inputs are lon, lat degrees.
    final double lonRad = srcPt.x * my_degToRad;
    final double latRad = srcPt.y * my_degToRad;
    final double theta = my_n * (lonRad - my_originLonRad);
    // One Std lat -- tangential cone.
    final double rhoYo = my_earthRad * my_F / Math.pow(Math.tan(Math.PI/4.0 + latRad/2.0), my_n);
    final double rho   = my_earthRad * my_F / Math.pow(Math.tan(Math.PI/4.0 + latRad/2.0), my_n);
    // Computes kilometers in lambert space.
    dstPt.x = rho * Math.sin(theta);
    dstPt.y = my_rhoZero - (rho * Math.cos(theta));
    // WANK - Here's the problem! These values shouldnt be NaN!
    if( Double.isNaN( dstPt.x ) || Double.isNaN( dstPt.y ) )
        System.out.println("======= A _projected dstPt has NaNs. Dumping...vvvvvvvvvvvvvvvvvvvvvvvvvvvv");
        if( Double.isNaN( dstPt.x ) )
            System.out.println("======= dstPt.x is NaN");
        if( Double.isNaN( dstPt.y ) )
            System.out.println("======= dstPt.y is NaN");
        System.out.println("======= my_stdLat1 = " + my_stdLat1 );
        System.out.println("======= my_n = " + my_n );
        System.out.println("======= my_originLonRad = " + my_originLonRad );
        System.out.println("======= my_F = " + my_F );
        System.out.println("======= my_earthRad = " + my_earthRad );
        System.out.println("======= lonRad = " + lonRad );
        System.out.println("======= latRad = " + latRad );
        System.out.println("======= theta = " + theta );
        System.out.println("======= Math.tan(Math.PI/4.0 + latRad/2.0) = " + Math.tan(Math.PI/4.0 + latRad/2.0) );
        System.out.println("======= Math.pow(Math.tan(Math.PI/4.0 + latRad/2.0), my_n) = " + Math.pow(Math.tan(Math.PI/4.0 + latRad/2.0), my_n) );
        System.out.println("======= rho = " + rho );
        System.out.println("======= rhoYo = " + rhoYo );
        System.out.println("======= Math.sin(theta) = " + Math.sin(theta) );
        System.out.println("======= dstPt.x = " + dstPt.x );
        System.out.println("======= Math.cos(theta) = " + Math.cos(theta) );
        System.out.println("======= my_rhoZero = " + my_rhoZero );
        System.out.println("======= (rhoYo * Math.cos(theta)) = " + (rho * Math.cos(theta)) );
        System.out.println("======= my_rhoZero - (rhoYo * Math.cos(theta)) = " + (my_rhoZero - (rho * Math.cos(theta)) ));
        System.out.println("======= dstPt.y = " + dstPt.y );
        System.out.println("======= A _projected dstPt had NaNs. Done dumping. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^");
    return dstPt;
}And here's the sample output:
======= A _projected dstPt has NaNs. Dumping...vvvvvvvvvvvvvvvvvvvvvvvvvvvv
======= dstPt.x is NaN
======= dstPt.y is NaN
======= my_stdLat1 = 25.0
======= my_n = 0.42261826174069944
======= my_originLonRad = -1.6580627893946132
======= my_F = 2.5946660025799146
======= my_earthRad = 6371.2
======= lonRad = -2.7564670759053924
======= latRad = 0.3730758324037379
======= theta = -0.4642057102537187
======= Math.tan(Math.PI/4.0 + latRad/2.0) = 1.4652768116539785
======= Math.pow(Math.tan(Math.PI/4.0 + latRad/2.0), my_n) = 1.175224090766834
======= rho = NaN
======= rhoYo = 14066.369269924155
======= Math.sin(theta) = -0.44771270676160557
======= dstPt.x = NaN
======= Math.cos(theta) = 0.8941774612481554
======= my_rhoZero = 13663.082491950498
======= (rhoYo * Math.cos(theta)) = NaN
======= my_rhoZero - (rhoYo * Math.cos(theta)) = NaN
======= dstPt.y = NaN
======= A _projected dstPt had NaNs. Done dumping. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

HI JSchell (and others?) -
I have created a simple example attached below, that when run repeatedly, does indeed generate spurious NaNs. I have made it as simple as possible. In the code, I use my own lon/lat binary data file, though I am sure any will do. Let me know if anyone wants that file.
So the deal is that (with my data at least) this program should never generate NaN results. And if one runs it 25432 (eg: random #) times, it wont, but then the 25433th time, it will create NaNs, etc. ie: inconsistent NaN math results.
As I said before, I have run this on old and new Dell laptops under Windows XP, using java 1.5_02, 1.5_04 and 1.5_08. The latest run was on a brand new Dell with a Intel Core Duo T2600 processor, running XP. If this is a result of the Pentium bug, one would think that would be fixed by now. I have NOT yet tested on AMD, though I will do that this afternoon.
Remember, this ONLY happens with Windows Laptops.
Any ideas anyone? Thanks in advance ;-)
Here's the code that produces spurious NaNs:
import java.awt.geom.Point2D;
import java.io.DataInputStream;
import java.io.EOFException;
import java.io.File;
import java.io.FileInputStream;
public class FloatingPointTest2 implements Runnable
private static final int NUM_ITERATIONS = 100000;
private double _degToRad = Math.PI / 180.0;
private double _originLon = -95.0;
private double _originLat = 25.0;
private double _originLonRad = _originLon * _degToRad;
private double _originLatRad = _originLat * _degToRad;;
private double _stdLat1 = 25.0;
private double _stdLat1Rad = _stdLat1 * _degToRad;
private double _earthRad = 6371.2;
private double _n = _n = Math.sin( _stdLat1Rad );
private double _F = Math.cos( _stdLat1Rad ) * Math.pow( Math.tan( Math.PI / 4.0 + _stdLat1Rad / 2.0 ), _n ) / _n;
private double _rhoZero = _earthRad * _F / Math.pow( Math.tan( Math.PI / 4.0 + _originLatRad / 2.0 ), _n );
private Point2D.Double _project( Point2D.Double srcPt, Point2D.Double dstPt )
    if( Double.isNaN( srcPt.x ) || Double.isNaN( srcPt.y ) )
      System.out.println( "FloatingPointTest2: received a NaN srcPt. Skipping." );
      return new Point2D.Double( Double.NaN, Double.NaN );
    // Inputs are lon, lat degrees.
    final double lonRad = srcPt.x * _degToRad;
    final double latRad = srcPt.y * _degToRad;
    final double theta = _n * ( lonRad - _originLonRad );
    double rho = _earthRad * _F / Math.pow( Math.tan( Math.PI / 4.0 + latRad / 2.0 ), _n );
    dstPt.x = rho * Math.sin( theta );
    dstPt.y = _rhoZero - ( rho * Math.cos( theta ) );
    return dstPt;
public void doTest()
    DataInputStream instream = null;
    int thisRunNaNCount = 0;
    Point2D.Double tempPt = new Point2D.Double();
    Point2D.Double dstPt = new Point2D.Double();
    try
      instream = new DataInputStream( new FileInputStream( System.getProperty(
        "user.home" ) + File.separatorChar + "lonLatBinaryData.bin" ) );
      try
        while( true )
          double lon = instream.readDouble();
          double lat = instream.readDouble();
          if( Double.isNaN( lon ) || Double.isNaN( lat ) )
            continue;
          tempPt.x = lon;
          tempPt.y = lat;
          dstPt = _project( tempPt, dstPt );
          if( Double.isNaN( dstPt.x ) || Double.isNaN( dstPt.y ) )
            thisRunNaNCount++;
      catch( EOFException e )
//        System.out.println( "End of file" );
      if( thisRunNaNCount > 0 )
        System.out.println( "thisRunNaNCount = " + thisRunNaNCount );
      instream.close();
    catch( Exception e )
      e.printStackTrace();
      System.exit( 1 );
public void run()
    doTest();
public static void main( String args[] )
    System.out.println( "Executing FloatingPointTest2." );
    for( int i = 0; i < NUM_ITERATIONS; i++ )
      FloatingPointTest2 test = new FloatingPointTest2();
      test.doTest();
}

Error in mapping for floating Number calculation

Hi All,
   I have a small doubt in floating number calculation in Mapping.
Actually i am geting a floating point number and calculating the SUM and generating the output. The input is of 2 decimal places(Ex: 26.02 and 26.03 ), but when it is adding all the values it is generating a three digit decimal number (Ex: 52.050003)
I dont know from where it is geting one extra number "2" in the output.
Please find the code for the same and let me know if i need to do something else to get ride of this.
   //write your code here
float sum=0;
if(a != null && a.length > 0.00)
   for ( int j =0; j<a.length;j++)
    sum = sum + Float.parseFloat(a[j]);
   result.addValue(String.valueOf(sum));
else
result.addValue("0");
Thanks in Advance,
JAY

Jay,
Please use the below code and let us know, if it helps.
BigDecimal sum= new BigDecimal("0");
BigDecimal bd;
if(a != null && a.length > 0.00)
for ( int j =0; j<a.length;j++)
bd=new BigDecimal(a[j]);
sum=sum.add(bd);
result.addValue(""+sum+"");
else
result.addValue("0");
in import section - java.math.*;
raj.
Edited by: Raj on Feb 18, 2008 11:11 AM

128-bit floating point numbers on new AMD quad-core Barcelona?

There's quite a lot of buzz over at Slashdot about the new AMD quad core chips, announced yesterday:
http://hardware.slashdot.org/article.pl?sid=07/02/10/0554208
Much of the excitement is over the "new vector math unit referred to as SSE128", which is integrated into each [?!?] core; Tom Yager, of Infoworld, talks about it here:
Quad-core Opteron? Nope. Barcelona is the completely redesigned x86, and it’s brilliant
Now here's my question - does anyone know what the inputs and the outputs of this coprocessor look like? Can it perform arithmetic [or, God forbid, trigonometric] operations [in hardware] on 128-bit quad precision floats? And, if so, will LabVIEW be adding support for it? [Compare here versus here.]
I found a little bit of marketing-speak blather at AMD about "SSE 128" in this old PDF Powerpoint-ish presentation, from June of 2006:
http://www.amd.com/us-en/assets/content_type/DownloadableAssets/PhilHesterAMDAnalystDayV2.pdf
WARNING: PDF DOCUMENT
Page 13: "Dual 128-bit SSE dataflow, Dual 128-bit loads per cycle"
Page 14: "128-bit SSE and 128-bit Loads, 128b FADD, 128 bit FMUL, 128b SSE, 128b SSE"
etc etc etc
While it's largely just gibberish to me, "FADD" looks like what might be a "floating point adder", and "FMUL" could be a "floating point multiplier", and God forbid that the two "SSE" units might be capable of computing some 128-bit cosines. But I don't know whether that old paper is even applicable to the chip that was released yesterday, and I'm just guessing as to what these things might mean anyway.
Other than that, though, AMD's main website is strangely quiet about the Barcelona announcement. [Memo to AMD marketing - if you've just released the greatest thing since sliced bread, then you need to publicize the fact that you've just released the greatest thing since sliced bread...]

I posted a query over at the AMD forums, and here's what I was told.
I had hoped that e.g. "128b FADD" would be able to do something like the following:
/* "quad" is a hypothetical 128-bit quad precision */
/* floating point number, similar to "long double" */
/* in recent versions of C++:                       */
quad x, y, z;
x = 1.000000000000000000000000000001;
y = 1.000000000000000000000000000001;
/* the hope was that "128b FADD" could perform the */
/* following 128-bit addition in hardware:          */
z = x + y;
However, the answer I'm getting is that "128b FADD" is just a set of two 64-bit adders running in parallel, which are capable of adding two vectors of 64-bit doubles more or less simultaneously:
double x[2], y[2], z[2];
x[0] = 1.000000000000000000000000000001;
y[0] = 1.000000000000000000000000000001;
x[1] = 2.000000000000000000000000000222;
y[1] = 2.000000000000000000000000000222;
/* Apparently the coordinates of the two "vectors" x & y       */
/* can be sent to "128b FADD" in parallel, and the following   */
/* two summations can be computed more or less simultaneously: */
z[0] = x[0] + y[0];
z[1] = x[1] + y[1];
Thus e.g. "128b FADD", working in concert with "128b FMUL", will be able to [more or less] halve the amount of time it takes to compute a dot product of vectors whose coordinates are 64-bit doubles.
So this "128-bit" circuitry is great if you're doing lots of linear algebra with 64-bit doubles, but it doesn't appear to offer anything in the way of greater precision for people who are interested in precision-sensitive calculations.
By the way, if you're at all interested in questions of precision sensitivity & round-off error, I'd highly recommend Prof Kahan's page at Cal-Berzerkeley:
http://www.cs.berkeley.edu/~wkahan/
PDF DOCUMENT: How JAVA's Floating-Point Hurts Everyone Everywhere
http://www.cs.berkeley.edu/~wkahan/JAVAhurt.pdf
PDF DOCUMENT: Matlab's Loss is Nobody's Gain
http://www.cs.berkeley.edu/~wkahan/MxMulEps.pdf

Floating point Number & Packed Number

Hai can anyone tell me what is the difference in using floating point & packed Number .
when it will b used ?

<b>Packed numbers</b> - type P
Type P data allows digits after the decimal point. The number of decimal places is generic, and is determined in the program. The value range of type P data depends on its size and the number of digits after the decimal point. The valid size can be any value from 1 to 16 bytes. Two decimal digits are packed into one byte, while the last byte contains one digit and the sign. Up to 14 digits are allowed after the decimal point. The initial value is zero. When working with type P data, it is a good idea to set the program attribute Fixed point arithmetic.Otherwise, type P numbers are treated as integers.
You can use type P data for such values as distances, weights, amounts of money, and so on.
<b>Floating point numbers</b> - type F
The value range of type F numbers is 1x10*-307 to 1x10*308 for positive and negative numbers, including 0 (zero). The accuracy range is approximately 15 decimals, depending on the floating point arithmetic of the hardware platform. Since type F data is internally converted to a binary system, rounding errors can occur. Although the ABAP processor tries to minimize these effects, you should not use type F data if high accuracy is required. Instead, use type P data.
You use type F fields when you need to cope with very large value ranges and rounding errors are not critical.
Using I and F fields for calculations is quicker than using P fields. Arithmetic operations using I and F fields are very similar to the actual machine code operations, while P fields require more support from the software. Nevertheless, you have to use type P data to meet accuracy or value range requirements.
reward if useful

BigDecimal vs floating points...

Hi all,
I know its probably been asked amillion times before but I need to finally fully understand and get my head around the two.
Firstly here are some bits I've been told by different people and read in different places (alot of people seem to think differently which is what confuses me):
- I've read that if you are wanting precision for currency for example that floating point shouldnt be used because of its accuracy down to the fact it cant represent every decimal number.
- The some people have told me that it doesnt matter and theres not much point ,ost the time in BigDecimal all you need to do is correct the floating point with formatting.
- I've asked about this before but people just seem to give me a short answer to it but without actually explaining why or where they get it from, you cant just assume an answer based on nothing...
I'm building some engineering software that has a general accuracy of 3 decmial places (millimeters from meters) and my first thought is that if currency at 2 decimal places requires BigDecimal then I surely require it (I cant afford to be missing off mm for every calculation, theres alot!) but the problem is this has resulted in me building pretty much the whole application with BigDecimal which you can probably imagine brings up thoughts about performance and memory uptake, I do calculations with BigDecimal, store data in BigDecimal and infact the only thing I do in double is the graphical display as the accuracy isnt so important.
My last question is if this is an ok way to build an accurate application it makes me start to wonder why is floating points used more than BigDecimals, surely most numbers are required to be accurate in applications especially of an enterprise scale?
Thanks,
Ken

MarksmanKen wrote:
So your a big user of BigDecimal as well then? Thats good to know someone else thinks in similar ways, I was starting to feel like abit of an idiot for using them so extensively lolNot at all. The idiots are the people who use primitives rather than BigDecimal "because they're faster" even though they've never actually experienced any performance problems. Of course, there are lots of cases where the speed of a primitive is preferable, but on the whole those guys know perfectly well who they are and what they're doing.
My program is very calculation heavy and I've not had any real performance issues yet but I was wondering if the performance gain would be significant enough while keeping the accuracy.Testing will show you the way. Don't let any "we tested this calculation a million times using primitives and the same one using BigDecimal, and it showed a remarkable 3 seconds quicker using primitives" sidetrack you, either. All that matters is that your actual production code is performant enough for your application. Generally speaking, anything involving currency will probably be better using BigDecimal, or, really, a Money class which happens to use BigDecimal under the covers. Quite why enterprise-targeted languages don't have some sort of native Money or Currency class out-of-the-box remains a mystery, to be honest.

So... if there's no floating point math, how do all those calculators work?

I see there's no floating point math support in J2ME.
So how do those calculators work? There's lots of 'em for download.
I have an app that's almost done; all I need to do is finish the calculations, but I need to read Strings from TextArea's and compute values based on them.
What am I missing?

You can use third party libraries
(MathFP http://www.jscience.net)
or
implement them with classical algorithms.
Carlos Sanchez
[Intesys]

Question in floating point operation

Hi,
I have question in java floating point operation.
public class test
     public static void main(String args[])
          double d1 = 243.35 ;
          double d2 = 2.3 ;
          System.out.println(d1 * d2) ;
          System.out.println((float)d1 * (float)d2) ;
The result is,
java version "1.4.1_02"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.1_02-b06)
Java HotSpot(TM) Client VM (build 1.4.1_02-b06, mixed mode)
5.597049999999999E8
5.5970502E8
Though the multiplication does not result irrational number like 1/3, the result of the first statement is not accurate enough. In our project, this multiplication involves with money and we cannot ignore this.
Can anyone suggest why this is happening? Do I need to convert all the numbers to float to avoid this...Or Is it a bug?
~ Sathiya Dhanapal.

The underlying problem is that not all numbers can be represented exactly in a floating point representation. But if you perform all calculations using doubles and then round to two fractional digits at the end you should get a "correct" result UNLESS you have used ill-conditioned formulas introducing other kinds of arithmetic errors.
There's another way around this when it comes to counting money and that's to use integers (long or int). You convert every number to the lowest monetary unit (like a cent or whatever). Every money-amount can now be represented exactly but you still have to be careful because the rounding problem is still there (What do you do with the last cent when you split 100 cents in 3).
In your example the "more correct" you've got from using floats instead of doubles is only an illusion. The result has been implictly rounded becasuse fewer bits have been used. If you round the double result to the same precision as the float result, they're the same.
The important lesson in all this is TO KNOW WHEN TO ROUND.

Convert Floating Point Decimal to Hex

In my application I make some calculations using floating point format DBL,and need to write these values to a file in IEEE 754 Floating Point Hex format. Is there any way to do this using LabVIEW?

Mike,
Good news. LabVIEW has a function that does exactly what you want. It is well hidden though...
In the Advanced/Data manipulation palette there is a function called Flatten to String. If you feed this funtion with your DBL precision digital value you get the IEEE-754 hexadecimal floating point representation (64 bit) at the data string terminal (as a text string).
I attached a simple example that shows how it works.
Hope this helps. /Mikael Garcia
Attachments:
ieee754converter.vi ‏10 KB

32 bit floating point ... SLOW...

Hi,
I ran a little test because i found Motion took too much time exporting with 32 bit floating point.
I made a single layer, animated text in Motion with my Quad ( 2.5 G RAM)
When i exported 32 bit floating point QT Animation in Motion, it was very very slow and the CPU were running at 10 to 15%.
When i export 8 bit floating point, it is much faster but CPU run at about 20%.
BUT
In FCP, when i render 8 bit Motion project and .mov (from 8 bit), or 32 bit Motion projet and .mov (from 32 bit), they all render pretty fast...
8 bit Motion prj 45% CPU
.mov from Motion 8 bit 30% CPU
32 bit Motion prj 45% CPU
.mov from Motion 32 bit 60% CPU
Why that much difference ?
I dont understand why the CPU are running higher with a .mov (QT Animation) that has been created in 32 bit floating point ?
I thought that once it has been created (self contained) it did not matter...
thanks

32 bit floating point refers to how it will be rendered. Has nothing to do with the format itself. You're OK ... just edit.
32-bit floating point allows audio calculations, such as fader levels and effects processing, to be
performed at very high resolution with a minimum of error, which preserves the quality of your digital audio.
Jerry

Port of Giac [Longfloat] Library to HP Prime allowing [Variable Precision] Floating Point Arithmetic

HP Prime CAS is based on Giac, but [ misses ] some of its Special Purpose Libraries like the Giac [ Longfloat ] Library, which if [ Ported ] would allow HP Prime to be the First ( handheld ) Calculator to provide [ Variable Precision ] Floating Point Arithmetic routines ( fully integrated at its CAS Kernel level ). HP Prime already have internal calls to [ Longfloat ] library, but resulting in [ Error Messages ], like when selecting more than 14 Digits in [ evalf ] Numerical evaluation, as for example: evalf( 1/7, 14 ) producing 0.142857142857 and evalf( 1/7, 15 ) resulting in "Longfloat library not available Error: Bad Argument Value" The same happens when one tries to Extend the [ Digits ] variable to a value greater than 13, like Digits := 50 which returns Digits := 13 as output ( from any specified value higher than 13 ). The porting of [ Longfloat ] library to HP Prime, would open many New opportunities in [ handheld ] Numerical Computation, usually available only on Top Level Computer Algebra Systems, like Maple, Mathematica or Maxima, and also on Giac/XCas. Its worth mentioning that Any [ Smartphone ] with Xcas/Giac App installed, can fully explore [ Variable Precision ] Floating Point Arithmetic, on current ARM based architectures, which means that a Port of [ Longfloat ] Library from Giac to HP Prime, although requiring some considerable amount of labor, is Not an impossible task. The Benefits of such Longfloat [ Porting ] to a handheld Calculator like HP Prime, would put it several levels Up on the list of Top current Calculator Features, miles and miles away from competitors like TI Nspire CX CAS and Casio ClassPad II fx-CP 400 ... Even HP 49/50g have third party developed routines with limited Variable Precision floating point support, while such feature is Not fully integrated to their native CAS Kernel. For those who do not see "plenty" reason for a [ Longfloat ] Porting to HP Prime its needless to say that the PRIMARY reason for ANY [ CALCULATOR ] is to CALCULATE ! and besides Symbolic Computation ( already implemented on all contemporaries top calculator models ), Arbitrary / [ Variable Precision ] Floating Point Arithmetic is simply The TOP of the TOP ( of the IceCream ) in [ Numerical ] Computation ! ( and beside Computer Algebra Manipulation routines, one of the Main reasons for the initial development of the major packages like Maple, Mathematica or Maxima ).

Thanks for the Link to [ HPMuseum.org ] Page with Valuable Details about the Internal Floating Point implementations both on Home and CAS environments of HP Prime. Its interesting to point to the fact that HP 49/50g has a [ Longfloat ] Version 3.93 package implementation ( with the Same Name but Distinct Code from the Giac Library ) available at [ http://www.hpcalc.org/details.php?id=5363 ] Also its worth mentioning [ Wikipedia ] pages on Arbitrary Precision Arithmetic like [ https://en.wikipedia.org/wiki/Arbitrary-precision_arithmetic ], [ https://en.wikipedia.org/wiki/List_of_arbitrary-precision_arithmetic_software ] and [ https://en.wikipedia.org/wiki/List_of_computer_algebra_systems ] and the Xcas/Giac project at [ https://en.wikipedia.org/wiki/Xcas#Giac ] and Official Site at [ http://www-fourier.ujf-grenoble.fr/~parisse/giac.html ] It would be a Dream come True when a Fully Integrated Variable Precision Floting Point Arithmetic package where definetively incorporated to HP Prime CAS Kernel, like the Giac [ Longfloat ] Library, allowing the Prime to be the First calculator with such Resource trully incorporated at its [ Kernel ] level ( and not like an optional third party module as the HP 49/50g one, which lacks complete integration with their respective Kernel, since HP 49/50g does not have native support for Longfloats ).

Saving floating point images

I currently use averaging to get a noise mitigated image in single precision. I do all calculations with the data in array format but I would like to save the averaged images in floating point for posterity's sake. I have noticed that I can display the images of datatype Grayscale (SGL) no problem but have found no way to save the image without rounding the data into integer format. I think i'm out of luck but I wanted to see if anyone knew of another way.

MikeBoso a écrit :
Whoops, I forgot to mention a key detail in that the images would need to be viewed by users without LabVIEW.
When wonky stuff is observed we use ImageJ for image manipulation.
That was indeed a key detail. Have you had a look at the TIFF format. I know that TIFF can handle FP images, but I don't know if there are the corresponding readers (Photoshop ?). Of course, you'll have to develop your own file saver, since IMAQ vision TIFF file save is restricted to integers.
Chilly Charly (aka CC)
E-List Master - Kudos glutton - Press the yellow button on the left...

SQL Loader and Floating Point Numbers

Hi
I have a problem loading floating point numbers using SQL Loader. If the number has more than 8 significant digits SQL Loader rounds the number i.e. 1100000.69 becomes 1100000.7. The CTL file looks as follows
LOAD DATA
INFILE '../data/test.csv' "str X'0A'"
BADFILE '../bad/test.bad'
APPEND
INTO TABLE test
FIELDS TERMINATED BY "," OPTIONALLY ENCLOSED BY '"'
Amount CHAR
and the data file as follows
"100.15 "
"100100.57 "
"1100000.69 "
"-2000000.33"
"-100000.43 "
the table defined as follows
CREATE TABLE test
Amount number(15,4)
) TABLESPACE NNUT050M1;
after loading a select returns the following
100.15
100100.57
1100000.7
-2000000
-100000.4
Thanks in advance
Russell

Actually if you format the field to display as (say) 999,999,999.99, you will see the correct numbers loaded via SQL Loader.
null

Floating point calculation

Similar Messages

Maybe you are looking for