[Cfp-interest 1734] Re: footnote about sufficient %a formatting precision

[Jim Thomas]

Here’s a suggested replacement for footnote 296 to address the issues mentioned in the earlier message (below). 
https://wiki.edg.com/pub/CFP/WebHome/footnote_296_rev.pdf <https://wiki.edg.com/pub/CFP/WebHome/footnote_296_rev.pdf>
The use of P for formatting precision and p for type precision is potentially confusing. But both are used in the current C2X draft in the description of A,a formatting in 7.21.6.1, and they seem easy enough to tell apart.

I assume the sufficiency inequality in the current version of the footnote is based on the property:

  b1^(p1-1) > b2^p2 implies base-b1 numbers of precision p1 distinguish base-b2 numbers of precision p2.

The improved inequality in the footnote revision comes from …

C can print floating point numbers in hexadecimal form: 

[-]h.h…hp+/-d 

where each h represents a hex digit, d is a decimal integer power of 2, and the h to the left of the radix point is nonzero (but otherwise unspecified). Where P is the formatting precision, i.e., the number hex digits to the right of the radix point, the hex form can represent at least all binary numbers with precision 4P+1.

Using the property above,

2^(4P+1-1) > 10^n

or

16^P > 10^n

is sufficient for the hex output to distinguish decimal numbers of type precision n.

Thanks to Jerome Coonen for consultation.

- Jim Thomas

> On Jul 23, 2020, at 3:55 PM, Jim Thomas <jaswthomas at sbcglobal.net> wrote:
> 
> This is about footnote 296) in the C2X draft (N 2478). 
> 
> The anchor for the footnote is talking about the formatting precision which is denoted in the subclauses by P, to distinguish it from a type precision p. So it would seem the footnote should be talking about P instead of p. 
> 
> Then, could the sufficiency inequality be loosened to 16^(P-1) >= b^n? P is the number of digits to the right of the decimal point character, not the total number of digits, though the digit to the left of decimal point character might just be 1.
> 
> - Jim Thomas
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.oakapple.net/pipermail/cfp-interest/attachments/20200731/26206e82/attachment.htm>