home .. forth .. misc mail list archive ..

Re: Benchmarks


On Fri, 4 Aug 1995, Mike Losh wrote:

> Excuse any ignorance on my part about the SPEC benchmarks, but is a port of the 
> SPECint integer benchmark possible for F21?  Or is a C compiler MANDATORY for 
> the benchmark to be considered a true "SPEC" benchmark?  It seems to me that one
> could port the source into hand-optimized F21 machine code which performs the 
> same or equivalent computations.  If SPECint requires wide (64 bit?)  
> computations, F21 may not be extremely efficient, having to string 20 bit ops 
> together, but so what?  At least then we could say "F21 provides a SPECint of XX
> out of SRAM, YY out of DRAM, at cost $$."  The Great Masses could start 
> thinking, "Hmm, I can get 1/3 of the SPECInt of an Alpha workstation for 1/10th 
> the cost..." (or whatever the numbers are going to be).  Hopefully, SPECint 
> running on a cluster of 8 (or whatever) of F21s will be really impressive for 
> the price.
> 
> SPEC benchmarks may not be interesting to most MISC readers (who envision 20 bit
> apps), but I think most workstation buyers/users expect to see them.  If F21 and
> later chips are going to be a commercial success in the scientific and 
> engineering market, we must convince many of these people to take a look.  If we
> can get a foot in the door with SPECint, it may be easier to convice them of the
> advantages of MISC for their applications.  

I do not see why would people need to look at SPECint and SPECfp to 
measure the performance of a computer. As John Gustafson and Quinn Snell 
put it:

----
The SPEC benchmark is popular among workstation vendors. It is not an 
independent measure; a consortium of vendors determine what is in SPEC 
and how to report it. SPEC does not scale , and runs on a narrow range of 
computers at any given time. It has had to be revised once, as the first 
version proved too small for workstations after a few years of 
technological progress. SPEC claims to be the geometric ratio of the 
time reduction of various kernels and applications to the time required 
by a VAX-11/780. Unfortunately, the VAX-11/780 currently gets a SPECmark 
of about 3, indicating it is three times as fast as itself! SPEC survives 
largely because of the lack of credible alternatives.
----

They go ahead and develop a benchmark -- HINT -- that is scalable from humans
with pen and pencil, through micros, through PCs, through workstations,
through SMPs, to vector machines, MMPs and beyond. 

I do think, that HINT is a much more relevant benchmark than MIPS, FLOPS,
SPECS and WinMarks. And I think, that MISC machines will compare quite 
favorably with the rest of the flock of machines the scientific community 
is using.



--
Penio Penev <Penev@venezia.Rockefeller.edu> 1-212-327-7423