[Cfp-interest] aA formatting with small precision modifier

[Fred J. Tydeman]

On Tue, 21 Aug 2012 11:58:19 -0700 Jim Thomas wrote:
>>> 
>>> What is the result of:
>>>  printf("%.1Ha\n", DEC32_MAX);
>>> Is it INFINITY or 1e97?  I believe it is INFINITY. 
>>> As if round source value.
>> 
>> Yes, (0, 9999999, 90) -> inf
>> 
>> What if rounding down? what does "rounds the input, in the type, according to the current rounding direction for decimal floating-point operations, to the number of digits specified by the precision 
>modifier" really mean?
>> 
>> (0, 1234567, 0) -> (0, 1000000, 0) or (0, 1, 6)? Check implementations. Former has same quantum exponent as input.
>> 
>> For the max case the latter isn't an option because it's at the limit of the exponent range:
>>  (0, 9999999, 90) -> (0, 9000000, 90) -> 9.000000e96
>> 
>> Designed to reflect the quantum exponent?
>
>In the Aug teleconference we agreed to check the 754-2008 standard for guidance on this issue and if none is given to use the first (quantum preserving) approach, where the intermediate rounding in the example 
>above gives
>
>(0, 1234567, 0) -> (0, 1000000, 0)
>
>I didn't find any help in 754-2008 on this issue. Did anyone else?
>
>-Jim

IEEE-754-2008, 4.3, 4th paragraph has: 

Operations converting from an operand in a floating-point format to a result in
integer format or to an external character sequence (see 5.8 and 5.12) shall use 
the rounding-direction attribute associated with the radix of the operand.

5.12.2, 1st paragraph has:

An implementation shall provide operations that convert from all supported 
floating-point formats to external decimal character sequences (see 5.4.2). 
For finite numbers, these operations can be thought of as parameterized by 
the source format, the number of significant digits in the result (if 
specified), and whether the quantum is preserved (for decimal formats). 
Note that specifying the number of significant digits and specifying quantum 
preservation are mutually incompatible. The means of specifying the number of
significant digits and of specifying quantum preservation are language-defined 
and are typically embodied in the conversionSpecification of 5.4.2.

5.4.2 last paragraph has:

decimalCharacterSequence convertToDecimalCharacter(source, conversionSpecification)
See 5.12 for details. The conversionSpecification specifies the precision and 
formatting of the decimalCharacterSequence result.


So, it seems the 5.4.2 and 5.12 pass the buck to the other and
neither specifies the details.

---
Fred J. Tydeman        Tydeman Consulting
tydeman at tybor.com      Testing, numerics, programming
+1 (775) 287-5904      Vice-chair of PL22.11 (ANSI "C")
Sample C99+FPCE tests: http://www.tybor.com
Savers sleep well, investors eat well, spenders work forever.