UCBTEST announcement with details
David G. Hough at validgh
dgh
Mon Mar 13 16:31:44 PST 1995
The following was posted to comp.arch.arithmetic on 9 March 1995.
The first release of UCBTEST is available now from netlib, as a gzip'ed
tar file ucbtest.tar.gz, by
ftp from netlib.att.com in /netlib/fp/ucbtest.tar.gz,
or by WWW from
http://netlib.att.com/netlib/fp/index.html.
The .gz file is about 1 MB and requires
gunzip to decompress. The uncompressed directory is about 7 MB.
Some versions of Mosaic have trouble with .gz files; the one I used left
the .gz file in /var/tmp as a file with a tmpname.
This is what you get in return for "send index from fp" mail to
netlibaresearch.att.com:
file fp/ucbtest.tar.gz
for testing certain difficult cases of IEEE 754 floating-point arithmetic
lang C
size 1 megabyte
# Retrieve by ftp or http, not email.
The README follows:
A B S T R A C T
UCBTEST is a suite of programs for testing certain difficult cases of IEEE 754
floating-point arithmetic. While most of the source code is NOT public
domain, the usage restrictions are modest; please read and observe the
copyright notices in the source code modules that have them. Please report
any porting problems or logic errors to fdlibm-commentsasunpro.sun.com. This
readme may be formatted with commands like "tbl readme | troff -ms".
UCBTEST is a suite of programs for testing certain difficult cases of
IEEE 754 floating-point arithmetic. Some of the difficult test cases are
obtained from number-theoretic algorithms developed by Turing Award winner
Prof. W. Kahan, Department of Electrical Engineering and Computer Science,
University of California, Berkeley, as part of ongoing research into test
methods for computer arithmetic.
After Prof. Kahan presented a Floating-Point Indoctrination lecture
series at Sun Microsystems in 1988, some lecture attendees and former graduate
students of Prof. Kahan wrote programs to test the correctness of alleged IEEE
754 implementations of multiplication, division, and square root. The suite
has subsequently expanded to include the following:
ucbmultest - by number-theoretic means, generates multiplication test
cases that are exactly halfway and nearly halfway between representable
numbers. The test algorithm is in ucbtest/mul.c.
ucbdivtest - by number-theoretic means, generates division test cases
nearly halfway between representable numbers. Exactly halfway cases don't
arise in division. The test algorithm is in ucbtest/div.c.
ucbsqrtest - by number-theoretic means, generates sqrt test cases nearly
halfway between representable numbers. Exactly halfway cases don't arise in
sqrt. The test algorithm is in ucbtest/sqr.c.
ucbpitest - attempts to determine a rational approximation A/B close to a
system's approximate value of J used in trigonometric argument reduction, and
tests whether the system's approximation is used consistently. ucbctest/pi.c
is a complete translation into C of ucbftest/pi.F, derived from the pirats
program.
ucbpartest - attempts to determine general properties of computer arith-
metic by ostensibly portable means. ucbftest/par.F and ucbctest/par.c are
complete translations from the original Basic program of Kahan. After rework-
ing by various collaborators, ucbpartest's current pass/fail criterion
reflects one particular set of prejudices about x**0 and double rounding dur-
ing underflow; others may prefer different choices.
ucbfindpitest - determines an approximation to a system's approximate
value of J used in trigonometric argument reduction, suitable for use in
ucbeef/local.h.
ucbeefscan - a postprocessor that summarizes ucbeeftest's detailed out-
put.
ucbeeftest - programs for testing sin, cos, atan, exp, expm1, log, and
log1p. Random arguments test the functions for comparison to independently
computed values accurate to better than test function precision. In UCBTEST,
the program is compiled and run separately for each function. The testing
interface is thus somewhat different from the original form of the program
described in its README. Note the usage restrictions set forth in that README
and source code.
ucblibtest - hex test vectors for library functions in single, double,
and quad precision. Rounding precision control is not yet implemented, so
some x86 double test cases misleadingly appear to fail by one ulp, due to dou-
ble rounding. x86 and m68k double extended are not supported. Considerable
additional work remains to be done on this program. The source codes provided
in UCBTEST may be helpful now, but are subject to ongoing improvement.
Pentium flaw
An instance of the infamous Pentium FDIV flaw can be found in less than a
minute by compiling ucbdivtest in single precision with -DNTESTS=1000000 -DSP
etc. That finding is unexpected since the program was not designed to seek
out such gross errors inherent in a particular division method, but rather
looks for least-significant-bit errors independent of division method. Run-
ning in double or extended precision, ucbdivtest does not detect the Pentium
bug in a reasonable amount of time.
Porting
All the UCBTEST programs are intended for users with a good understanding
of IEEE 754 arithmetic and C. Some sample *.sh scripts in ucbREADME/ compile
and run all the programs, for three floating-point types: IEEE 754 single,
double, and double-extended (including quad). The test programs are designed
to terminate with non-zero exit status in case they find an error, unless com-
piled with -DNO FAIL, which causes zero exit status even in error cases. At
the end, the scripts summarize run-time failures indicated by UCBFAIL in the
output or by the existence of incomplete *.output.tmp files. The criterion
for ucblibtest is that all test vectors succeed; the criterion for ucbeeftest
is that no monotonicity or symmetry failures occur and that the worst case
errors are less than one ulp. These summaries don't catch compile and link
errors - those errors terminate Makefiles and scripts unless make -k is used.
ucbmultest, ucbdivtest, ucbsqrtest, and ucblibtest were developed on
SPARCstations running SunOS 4 or SunOS 5, and on x86 PC's running SunOS 5.
ucbdivtest and ucbsqrtest may be ported to other RISC/Unix systems with little
difficulty. Systems with extended precision, non-Unix systems, and ucbmul-
test require more effort to provide the equivalents of the Sun functions
ieee flags etc.
On SPARCstations, "int" and "long" and Fortran "INTEGER" are 32 bits
wide. On DOS systems, "int" is usually 16 and "long" 32. On some supercom-
puters, "int" and "long" might be 64. In those cases, these definitions in
libctest and ucbtest may require modification:
libctest.h: #define INTEGERTYPE int
libctest.h: #define INTEGER INTEGERTYPE *
ucbtest.h: #define FINT int /* C equivalent of Fortran integer */
ucbtest.h: #define INT32 int /* 32 bit int */
ucbtest.h: #define UINT32 unsigned int /* 32 bit unsigned */
Some Fortran compilers append underscores to subprogram identifiers - in
order to avoid name space conflicts when C and Fortran programs are mixed -
and some do not. Since UCBTEST depends on that underscore differentiation,
aix.sh for instance specifies "-DUNDERSCORE= " in $cppopts, and
libftest/libftest.h and ucbftest/ucbftest.h provide definitions like
#if defined( hpux) || defined( AIX)
#define UNDERSCORE
#else
#define UNDERSCORE
#endif
HPUX and AIX require the explicit underscore because they don't provide it by
default.
Misapplication
Misapplication often causes these programs to fail, reporting hardware or
software bugs where none exist. If misapplied - especially after porting to a
new environment for the first time - they may report arithmetic "errors" when
nothing is amiss. Nor do "error-free" results prove correct arithmetic,
although they may "lend artistic verisimilitude to an otherwise bald and
unconvincing" proof. Common misapplications include:
precision
On extended-precision systems such as x86 and 68k, the extended-precision
precision control must be set to the precision of the type of floating-
point variable being tested. On Sun systems,
ieee flags("set","precision",...) is useful.
prototypes
Traditional C and ANSI-C differ in their treatment of float function
arguments and function results.
traditional C expression evaluation
Traditional C may evaluate expressions involving floats by promoting them
first to double, causing two rounding errors where UCBTEST expects one.
Compiling with "-fsingle" is helpful with /bin/cc on SunOS 4.
optimizers
Optimizing compilers that allocate variables to extended-precision regis-
ters and fail to round them to storage precision, even in response to
source-code assignments, may cause problems. Certain variables have
been declared "volatile" in hope of avoiding those problems. Particu-
larly on PC's, compilers seem to have trouble correctly optimizing code
and libraries to work with rounding precision set to single instead of
extended. Perhaps more "volatile" declarations are required. On GCC,
-ffloat-store sometimes helps.
sqrt functions
C compilers usually don't convert calls to sqrtf, sqrt, or sqrtl to
inline machine sqrt instructions by default. Thus testing those instruc-
tions, rather than the subroutine library, may require extra steps such
as coding in assembly language. Sun compilers invoked with "-xlibmil"
provide the necessary inlining to test SPARC or x86 sqrt instructions.
Source code
Some compile-time flags used to configure the tests include:
-DNTESTS number of test cases to try
-DDEBUG turn on debugging output
-DSUN IEEE ieee flags, ieee retrospective available
-DSUN MATH sunmath.h and libsunmath.a are available
-DSUN LIBS Fortran-callable versions available for
Sun libm and libsunmath functions
-DNO FUNCF no ANSI-C reserved functions sqrtf, expf, etc.
-DNO FAIL do not abort on certain errors
-DSP test IEEE single precision
-DDP test IEEE double precision
-DQP test IEEE extended precision
Several of the programs are set up so that the C infrastructure may be
used to test the arithmetic and libraries of any language. Sample wrappers
are provided for C and Fortran. Outer main programs are in ucbctest/ and
ucbftest/; they use inner main programs and utility functions from ucbtest/ to
invoke the test functions in libctest/ or libftest/.
Makefiles are provided in the ucbctest/, ucbftest/, libctest/, and
libftest/ directories. Disable .SILENT: targets for more debugging output;
enable them for less. The Makefiles avoid infinite loops in ucblibtest, but
aux.sh gets into infinite loops in ucbpartest and ucbmultest in single preci-
sion. In order to continue UCBTEST, such looping processes need to be ter-
minated manually with a kill command.
Sample *.sh scripts for invoking the Makefiles are in ucbREADME/. The
scripts define -NTESTS=1000000; reduce that to 1000 to simplify debugging;
increase it to one billion for a more comprehensive test.
ucb{mul,div,sqr,eef}test have been adjusted so their run times are approxi-
mately comparable for comparable NTESTS. ucb{pi,par,findpi,lib}test run times
do not depend on NTESTS.
The scripts define which precisions and which languages are to be tested,
how each language is to be compiled, and whether (via make -k)
script/compile/link errors terminate execution. To create a script for a new
platform, start by copying one of the existing scripts for a similar system,
or start with generic.sh and modify it according to the suggestions in its
comments until it works acceptably.
Generally speaking, the source code uses -DSP, -DDP, and -DQP to distin-
guish single, double, and extended precision. -DQP refers flexibly to
double-extended on x86 and quadruple on SPARC on all the tests except ucblib-
test, where the inputs and output test vectors are specified in hex, so -DQP
means SPARC quadruple precision, and double-extended isn't supported. The
programs start out by setting extended rounding precision on x86 to match that
of the nominal precision (SP, DP, QP), but because some of the libm functions
misbehave when invoked that way, ucb{pi,findpi,eef}test prevent that by res-
toring the default precision.
The source code file ucbtest/fpce.S is intended to provide a convenient
place to integrate assembly-language implementations of the functions to set
IEEE rounding direction and precision. sunos5.gcc.sh depends on those to pro-
vide a substitute for ieee flags in Sun's unbundled libsunmath.a. Unfor-
tunately in Solaris 2, there's no convenient way to set const lib version,
defined in <math.h>, to -1 in order to get the same effect as Sun's -xlibmieee
option; this causes some ucblibtest vectors to fail with gcc. That can be
remedied by reconfiguring GCC to link with values-Xi.o instead of values-X?.o
(see config/svr4.h) where values-Xi.c is:
#include <math.h>
const enum version lib version = libm ieee;
Directory information
ucbctest/ and ucbftest/ contain the main programs for each test, in C and
Fortran respectively.
libctest/ and libftest/ contain the test modules which invoke each func-
tion to be tested, in C and Fortran respectively. Each function is in a
separate module so that compilation failure on one will not affect other
tests. On systems which lack some common library functions, it is necessary
to improvise; see for instance libctest/testexpm1*.c.
ucbtest/ contains the C logic for the programs that can be invoked from
main programs in C or Fortran.
ucbeef/ contains various source files invoked by ucbeeftest and ucbfind-
pitest. See the separate README.
ucblib/ contains the input files for ucblibtest.
ucbREADME/ contains this readme and also readme.about.testing that
discusses the limitations of various testing approaches. The various *.sh
sample test scripts are here, along with logs of typical executions.
The sc301.sh script for Sun's SC3.0.1 compilers on SunOS 5.3 passes the
largest number of successful tests of any of these sample scripts, not
surprisingly because predecessors of UCBTEST have been part of Sun's compiler
test regime for years.
ucbDOS contains Warren Ferguson's adapted source codes and compiled exe-
cutables for MSDOS. See the separate readme.
Results Obtained During Development
The following results are recorded in *.log files obtained with
corresponding *.sh scripts. In many cases the hardware, operating systems,
compilers, and libraries may not be current products, and the compilation op-
tions may not be optimal. The sample *.sh scripts are only starting points and
may require modification when hardware, operating systems, compilers, or li-
braries change. Many of the anomalies in the logs have not been investigated;
some may represent problems with the hardware, operating systems, compilers,
or libraries; problems with the UCBTEST port may underlie other anomalies,
corresponding to the various types of "misapplication" discussed above. Con-
tributions of improved scripts and log files are welcome.
The table indicates the fraction of "UCBPASS tests" / "total tests" pos-
sible with the tested languages and precisions, and when available, the total
time in minutes, either real or user+sys (u+s), for the script to create the
log file. In most cases other jobs were also running on the test systems, so
the times are very approximate. Some of the log files were produced by early
versions of the scripts that don't report as much information as the current
versions.
RISC systems with 113-bit extended or none
.sh hardware c UCBPASS
.log op sys fortran minutes
sc301 SS10/41 3.0.1 cc -O -xcg92 ... 77/84
sc301 SunOS 5.3 3.0.1 f77 -O -xcg92 ... 210 u+s
sunos4 SPARCclassic /bin/cc -O ... 56/56
sunos4 SunOS 4.1.3U1 1.0 f77 -O ... 170 u+s
apogee30 SPARC 2000, 60MHz 3.0 apcc -O -cg92 ... 38/56
apogee30 SunOS 5.4 3.0 apf77 -O -cg92 49 u+s
hpux HP 9000/712 /bin/c89 -DNTESTS=10000 ... 34/70
hpux HP-UX 9.05 /usr/bin/f77 ... 60 real
irix SGI ? /usr/bin/cc -O -lm 24/28
irix IRIX 5.2 - 60 real
mips.ultrix.gcc DEC mips gcc -O -lm 21/28
mips.ultrix.gcc Ultrix 4.1 - ?
alpha.osf.gcc DEC alpha gcc -O ... 11/14
alpha.osf.gcc OSF 3.0 - ?
CISC systems with 64-bit extended
.sh hardware c UCBPASS
.log op sys fortran minutes
pc201 Pentium 90 fixed 2.0.1 cc -O -xpentium ... 54/70
pc201.fix SunOS 5.4 2.0.1 f77 -O -pentium ... 89 u+s
pc201 Pentium 90 flawed 2.0.1 cc -O -xpentium ... 52/70
pc201.flaw SunOS 5.4 2.0.1 f77 -O -pentium ... 88 u+s
sunos3 Sun-3/260 /bin/cc -O -f68881 ... 46/56
sunos3 SunOS 3.5.2 /usr/bin/f77 -f68881 ... 3200 u+s
linux 486 66DX2 /usr/bin/cc -O ... 44/56
linux slackware 1.1.59 /usr/bin/f77 -O 170 u+s
sunos3.fpa Sun-3/260+FPA /bin/cc -O -ffpa ... 41/56
sunos3.fpa SunOS 3.5.2 /usr/bin/f77 -O -ffpa ... 1300 u+s
aux Apple Quadra 800 /bin/cc -O -lm 40/56
aux A/UX 3.0.2 /usr/bin/f77 -O 440 real
sunos5.gcc Pentium 90 fixed 2.6.3 gcc -O ... 24/28
sunos5.gcc SunOS 5.4 - 43 real
sunos5.fdlibm.gcc Pentium 90 fixed 2.6.3 gcc -O ... 24/28
sunos5.fdlibm.gcc SunOS 5.4 - 32 u+s
m68.hp.netbsd1.0a HP m68k gcc -O ... 24/28
m68.hp.netbsd1.0a BSD 1.0a - ?
Source Code Authors; Translators, Porters, Maintainers
ucbdivtest - K.-C. Ng; M. Alemi, S. Ma, D. Priest.
ucbeeftest - Z. A. Liu inspired in part by W. J. Cody and W. Waite's
ELEFUNT.
ucbfindpitest - Z. A. Liu.
ucblibtest - K.-C. Ng.
ucbmultest - B. Toy; M. Alemi, S. Ma, M. Mueller, K.-C. Ng, D. Priest.
ucbpartest - W. Kahan inspired in part by W. J. Cody's MACHAR; D. Feen-
berg, D. Gay, R. Karpkinski, T. Quarles, T. Sumner, G. Taylor, B. Wichmann.
ucbpitest - W. Kahan; D. Priest.
ucbsqrtest - K.-C. Ng; M. Alemi, S. Ma, D. Priest.
Z. A. Liu generously permitted distribution, as part of UCBTEST, of a
version of his Berkeley Elementary Function tests, according to the conditions
set forth in the comments of those source codes.
Sun Microsystems generously permitted distribution, as part of UCBTEST,
of versions of certain source codes and test vectors developed at Sun, accord-
ing to the conditions set forth in the header comments of those source codes
and test vectors.
Porting and Quality Acknowledgements
David Gay improved portability to non-Solaris workstations running vari-
ous proprietary compilers. Stephen Moshier improved portability to non-Solaris
workstations running GCC. Warren Ferguson improved portability to MSDOS and
contributed the ucbDOS directory. Intel lent systems equipped with flawed and
unflawed Pentiums for testing purposes.
Problems
Please report bugs and portability enhancements for other systems to
fdlibm-commentsasunpro.sun.com.
Futures
Enhancements planned to be incorporated some day:
* Ng's libm test vectors - ucblibtest - are still in the process of conver-
sion to the UCBTEST format, and the program's tests for rounding modes
and exceptions are not yet working portably.
* Kahan's tests for SRT-PLA-style dividers.
* Coonen's IEEE Test Vectors program and data.
* Hough's tests for correctly-rounded base conversion.
More information about the Numeric-interest
mailing list