compiler comparison report available

[David Hough]

The report summarized below may be of interest to people developing
software on SPARCstations, and to people considering using GCC or F2C for
any purpose.   On request I will send source code to print it with
	tbl | troff -ms

Summary

     Many SPARCstation customers wonder if they can get better C and Fortran
compilation speed or execution speed or correctness or robustness by obtaining
compilers from other sources than SunPro, but they often lack the resources to
thoroughly study the matter.   In many cases, however, the answer appears to
be "no".  I compared some available C and Fortran compilers on a
SPARCstation-2 running SunOS 4.1.2.   The results are:

1)   GCC 2.1 and F2C are robust "free" alternatives to SunPro SC1.0 compilers.

2)   Lucid-C 2.1 and KAP are much less robust than the SC1.0 compilers they
     intend to replace or enhance, often generating incorrectly optimized
     code.

3)   SunOS 4.1.2 bundled /bin/cc is not as robust at -O4 as SC1.0 cc; many
     Fortran programs translated with F2C require compilation at -O2.

4)   I found no better alternative to SC 1.0 compilers that provides con-
     sistently superior execution-time performance, especially on floating-
     point programs, except SC2.0.1 compilers.

5)   I found no better alternative to SC1.0 cc for compile and execution time
     performance with -g, the normal debugging mode for C programs.

     SC1.0 compilers are SunPro's current release; SC2.0.1 compilers are the
next release under development.

Definition of Execution-Time Performance

     The performance comparison between two compilers is based on relative
user+sys performance of programs that ran correctly and for at least ten
seconds with both compilers.  On a particular test program, 100% is defined to
be the best correct performance obtained so far on a SPARCstation-2 with any
compiler, any options, and any operating system.

     Thus a rating of 50% corresponds to 2X slower than the best.  All runs
were ranked in order, and first and third quartile percentages recorded.  In
the following tables, for instance, "47-70" means that one fourth of the tests
achieved less than 47% of best known correct performance, one fourth achieved
more than 70% of best known correct performance, and half achieved between 47%
and 70%.

     Only execution times of ten seconds or more, from realistic applications,
were tabulated in the execution time comparisons.  Data was collected in
multi-user mode, and so may vary somewhat from run to run; hence differences
of a few percentage points aren't significant.

     specint92 and specfp92 are also included.   The usefulness of these sin-
gle measures is reduced by lack of a corresponding measure of dispersion.  And
they are based on user+sys time under the SunPro test harness rather than real
time under official SPEC Makefiles, and so may vary from official SPEC
results.

     The SunPro test harness is a collection of Makefiles and sh and awk
scripts, combined with modified versions of test programs from a variety of
sources, that is used to test Unix system-level correctness and measure Unix
system-level performance of CPU-bound programs. As such it tests hardware,
operating system, compilers, and libraries.  For this report, in order to com-
pare compilers and their libraries, the hardware and operating system were
kept constant: a SPARCstation-2 with 32 MB RAM running SunOS 4.1.2.

Definition of Compile-Time Performance

     Like execution times, compilation times are evaluated as a percentage of
the best user+sys compile time ever recorded on a SPARCstation-2, from any
compiler, with any options.  Compile times are given for all programs in a
particular language, whether realistic applications or synthesized test pro-
grams.  Because compilation involves substantial disk and net I/O, and sub-
stantial overhead for the SunPro test harness shell scripts and makefiles,
times are more variable and less reproducible than execution times, which are
mostly CPU-bound.