SPECjbb2000 Frequently Asked Questions
 |
Frequently Asked Questions (FAQ)
About the SPECjbb2000 Benchmark
|
General Information
- What is SPECjbb2000?
- Who will use SPECjbb2000 and how will they use it?
- Why is the benchmark important for vendors?
- Why did SPEC decide to develop SPECjbb2000?
- Does this benchmark measure end-to-end Java performance?
- Does SPECjbb2000 represent real applications?
- What specific aspects of performance does SPECjbb2000 measure?
- What factors affecting Java performance are not measured
by SPECjbb2000?
- Why doesn't SPECjbb2000 cover AWT/graphics performance?
- Is this benchmark suitable for measuring client performance?
- Does SPECjbb2000 use Enterprise Java Beans (EJBs), servlets
or Java Server Pages (JSPs)?
- Why doesn't the benchmark measure network and database
performance?
- Does this benchmark replace any other benchmark?
- Can I compare my SPECjbb2000 results with my TPC-C results?
- Can SPECjbb2000 results be compared to results from other
SPEC benchmarks?
- Is this related to any of IBM's BOB benchmarks?
- Can I use this application for non-benchmarking purposes?
- Do you provide source code for the benchmark?
- Does floating point precision matter for this benchmark?
- Are SPECjbb2000 results repeatable?
- Where can I find out more about the benchmark?
- What metrics does SPECjbb2000 use to report performance?
-
Setting up SPECjbb2000
- What are the requirements for running SPECjbb2000?
- What other software do I need? Web server, database, proxy
server?
- Can SPECjbb2000 run under Java 2?
- Can I use this benchmark for embedded processor platforms?
- Can I run SPECjbb2000 on a uniprocessor?
- Can I run SPECjbb2000 on a shared memory processor (SMP)
configuration and will it scale?
- How long does it take to run the SPECjbb2000 benchmarks?
- How much JVM heap memory is required?
- How much system memory is needed to run the benchmarks?
- Why can't I recompile this benchmark?
-
Running SPECjbb2000
- What do you mean by warehouses, points and threads?
- Is there a warmup period before each measured point?
- Where can official SPECjbb2000 results be obtained?
- Can I run SPECjbb2000 on a cluster configuration and compare
results with SMP configurations? If not, why?
- Can SPECjbb2000 help users compare Java and C++ performance?
- Does garbage collection affect SPECjbb2000 results?
- I see a lot of paging and disk I/O but I had a compliant
run. Can I publish my results?
- Why is it necessary to look at the SPECjbb2000 graph to
get a good idea of performance?
- My graph looks strange. Are there two graph formats?
-
Ordering SPECjbb2000
- What do you get when you order SPECjbb2000?
- What is the price of a SPECjbb2000 license and when will
it be available?
- How do I contact SPEC for more information?
General Information
Q1:
What is SPECjbb2000?
A1: SPECjbb2000
is a Java program emulating a 3-tier system with emphasis on the middle tier.
Random input selection represents the 1st tier user interface. SPECjbb2000
fully implements the middle tier business logic. The 3rd tier database is replaced
by binary trees.
SPECjbb2000 is inspired by the TPC-C benchmark and loosely follows the
TPC-C specification for its schema, input generation, and transaction profile.
SPECjbb2000 replaces database tables with Java classes and replaces data
records with Java objects. The objects are held by either binary trees (also
Java objects) or other data objects.
SPECjbb2000 runs in a single JVM in which threads represent terminals in
a warehouse. Each thread independently generates random input (tier 1 emulation)
before calling transaction-specific business logic. The business logic operates
on the data held in the binary trees (tier 3 emulation). The benchmark does
no disk I/O or network I/O.
For additional details, refer to the SPECjbb2000 whitepaper.
Q2:
Who will use SPECjbb2000 and how will they use it?
A2: Buyers
of middle-tier servers can compare different platforms both in terms of differences
in hardware and software (e.g., JVM, JIT, and OS).
Q3:
Why is the benchmark important for vendors?
A3: SPECjbb2000
represents a middleware application written in Java. Hardware vendors can use
the benchmark's results to analyze their platforms' scalability when
running Java applications. Software vendors can evaluate the efficiency of
their JVMs, JITs, garbage collectors and thread implementations.
Q4:
Why did SPEC decide to develop SPECjbb2000?
A4: SPEC
has an existing Java client benchmark, SPECjvm98, and wanted to offer a Java
benchmark from the server end. There are a few Java server benchmarks available,
but SPECjbb2000 is the only one that provides SPEC's methodology for ensuring
consistent, repeatable results.
Q5:
Does this benchmark measure end-to-end Java performance?
A5: No.
Please see the answers that follow for areas not covered by SPECjbb2000.
Q6:
Does SPECjbb2000 represent real applications?
A6: Yes.
Many companies have developed Java application frameworks for deploying large
commercial systems. SPECjbb2000 models the business logic and object caches
that make up the middle tier of these frameworks. The emulation of the other
tiers isolates this tier and simplifies the benchmark by not requiring user
emulation or a database. The implication is that combining a JVM with a high
SPECjbb2000 throughput and a database tuned for online transaction processing
will provide a business with a fast and robust multi-tier environment.
In developing this benchmark, SPEC worked toward a relatively flat method
profile to avoid overly good optimization candidates. SPEC also compared
the method profiles to other Java server applications within the member companies
to make sure that SPECjbb2000 behaves like a server application.
Q7: What specific
aspects of performance does SPECjbb2000 measure?
A7: SPECjbb2000
measures the implementation of Java Virtual Machine (JVM), Just-in-time compiler
(JIT), garbage collection, threads and some aspects of the operating system.
It also measures the performance of CPUs, caches, memory hierarchy and the
scalability of Shared Memory Processors (SMPs) platforms on the specified commercial
workload.
Q8:
What factors affecting Java performance are not measured by SPECjbb2000?
A8: AWT,
network, I/O and graphics performance are not measured by the benchmark.
Q9:
Why doesn't SPECjbb2000 cover AWT/graphics performance?
A9: SPECjbb2000
doesn't cover AWT/graphics because this is not considered server work.
Q10:
Is this benchmark suitable for measuring client performance?
A10: No.
Q11:
Does SPECjbb2000 use Enterprise Java Beans (EJBs), servlets or Java Server
Pages (JSPs)?
A11: No.
Q12:
Why doesn't the benchmark measure network and database performance?
A12: A
two- or three-tier configuration would be necessary to measure network and
database performance. This would be a complex and expensive configuration to
test. Many other benchmarks exist to measure database and network performance.
Q13:
Does this benchmark replace any other benchmark?
A13: No.
Q14:
Can I compare my SPECjbb2000 results with my TPC-C results?
A14: No, absolutely
not. SPECjbb2000 is memory-resident, uses totally different data-set
sizes and workload mixes, performs no I/O to disks, and has no think times.
It has a different set of run and reporting rules, a different measure of
throughput, and a different metric. Such comparison would be a SERIOUS
violation of SPEC's run and reporting rules and of TPC's "fair
use policy." Violations are subject to penalties.
Q15:
Can SPECjbb2000 results be compared to results from other SPEC benchmarks?
A15: No.
There is no logical way to translate results from one benchmark to another.
Q16:
Is this related to any of IBM's BOB benchmarks?
A16: SPECjbb2000
is descended from IBM's pBOB benchmark. However, SPEC has extensively modified
both the design and the code during the development of SPECjbb2000. Therefore,
SPECjbb2000 is definitely not comparable to IBM's pBOB or any other BOB
benchmark you may have gotten from IBM.
P.S. Many thanks to IBM for providing the original code.
Q17:
Can I use this application for non-benchmarking purposes?
A17: No.
Q18:
Do you provide source code for the benchmark?
A18: Yes,
but you are required to run with the jar files provided with the benchmark.
Recompilation is forbidden in the run rules and will invalidate your results.
Q19:
Does floating point precision matter for this benchmark?
A19: No.
The SPECjbb2000 workload contains primarily integer computation.
Q20:
Are SPECjbb2000 results repeatable?
A20: Yes.
Measurement on a variety of platforms shows the SPECjbb2000 metric has a 95%
confidence interval of +/- 2% of the score.
Q21:
Where can I find out more about the benchmark?
Q22:
What metrics does SPECjbb2000 use to report performance?
A22: SPECjbb2000
ops/second is a composite throughput measurement representing the averaged
throughput over a range of points. It is described in detail in the document
"SPECjbb2000 Run and Reporting Rules."
Setting up SPECjbb2000
Q23: What are the requirements for
running SPECjbb2000?
A23: The
user needs a machine with a minimum of 256 MB physical memory, and a JVM environment
supporting the Java or Java 2 APIs. JVMs generally need 32 MB or more of disk
space for the installed software. The benchmark itself requires 10 MB of disk
space. For competitive numbers, more than 196 to 300 MB of physical memory
is almost certainly needed, depending on your JVM.
Q24:
What other software do I need? Web server, database, proxy server?
A24: None
of these. The benchmark does not use a web server, database or any other additional
software besides the JVM.
Q25:
Can SPECjbb2000 run under Java 2?
A25: Yes.
Q26:
Can I use this benchmark for embedded processor platforms?
A26: No.
Q27:
Can I run SPECjbb2000 on a uniprocessor?
A27: Yes,
but you will need to run all points from one to eight warehouses to get a reportable
result.
Q28:
Can I run SPECjbb2000 on a shared memory processor (SMP) configuration and
will it scale?
A28: Yes.
In theory, the peak throughput point in the SPECjbb2000 benchmark will occur
when the number of warehouses (or threads) equals the number of CPUs in an
SMP system. This may not always be the case depending your JVM, operating system
and hardware. During the development of SPECjbb2000, scaling up to 12 CPUs
was measured on more than one hardware platform and JVM.
Q29:
How long does it take to run the SPECjbb2000 benchmarks?
A29: A
one-to-eight warehouse SPECjbb2000 run takes close to half an hour. Specifically,
it takes just about three minutes per point measured, where a point is a number
of warehouses. It measures throughput in this fixed amount of time, so faster
machines do more work in the allotted time.
Q30:
How much JVM heap memory is required?
A30: Of
the JVMs tested, the heap sizes ranged between 196 MB and 300 MB to run up
to 8 warehouses. Typically, however, performance increases as you add more
heap.
Q31:
How much system memory is needed to run the benchmarks?
A31: Your
system memory should always exceed the maximum heap size for the JVM. Most
systems will need about 64 MB memory in addition to the java heap in order
to run SPECjbb2000 without swapping.
Q32:
Why can't I recompile this benchmark?
A32: The
intent of this benchmark is to measure the performance of the Java platform,
given a known set of application bytecodes. Therefore, you may not recompile
the source code if you want to produce a valid result.
If, however, you wish to experiment with the code, be aware that to successfully
compile the spec.reporter package you will need a Java 2 developer
kit that contains the com.sun.image.codec.jpeg package.
Running SPECjbb2000
Q33: What do you mean by warehouses,
points and threads?
A33: A
warehouse is a unit of stored data. It contains roughly 22 MB of data stored
in many objects in many Btrees.
A thread represents a terminal user within a warehouse. There is a one-to-one
mapping between warehouses and threads plus a few threads for SPECjbb2000
main and various JVM functions. As the number of warehouses increases during
the full benchmark run, so does the number of threads.
By "point", we mean a two-minute measurement at a given number
of warehouses. A full benchmark run consists of a sequence of measurement
points with an increasing number of warehouses (and thus an increasing number
of threads).
Q34:
Is there a warmup period before each measured point?
A34: For
each point, the benchmark creates data for that many warehouses, then starts
the threads making requests as fast as possible. The threads run for 30 seconds
(the warmup) before starting the two-minute measurement. This warmup may get
records into memory, but is probably not as influential as warmup time on an
actual database benchmark.
Q35:
Where can official SPECjbb2000 results be obtained?
Q36: Can I run
SPECjbb2000 on a cluster configuration and compare results with SMP configurations?
If not, why?
A36: No,
SPECjbb2000 was not designed to be an appropriate benchmark for clustered environments.
Q37:
Can SPECjbb2000 help users compare Java and C++ performance?
A37: No.
SPECjbb2000 is a JVM server benchmark, not a language benchmark.
Q38:
Does garbage collection affect SPECjbb2000 results?
A38: Yes.
Garbage collection definitely affects throughput, especially as the number
of warehouses increases. Increasing heap may lessen the effects of garbage
collection. Even with a 2G heap, garbage collection will probably still occur
at some point during a run. Your JVM may offer you flags to specify garbage
collector algorithm or parameters. The
-verbosegc switch will tell
you more about the characteristics of your garbage collector during this benchmark.
You may find that forcing garbage collection between points influences
performance. You may set the property input.forcegc to "true" or "false" to
control whether the test will force a garbage collection between each point.
Q39:
I see a lot of paging and disk I/O but I had a compliant run. Can I publish
my results?
A39: Yes,
but it would be unwise. This is a memory-resident server benchmark and you
should have adequate memory for your OS, JVM, the benchmark and heap. If you
observe a lot of disk activity, your results will be non-competitive and should
not be published.
Q40:
Why is it necessary to look at the SPECjbb2000 graph to get a good idea of
performance?
A40: SPECjbb2000
measures the efficiency of the JVM, JIT compiler and garbage collector. As
the amount of parallelism increases with more warehouses (and therefore more
threads), the balance between these components changes, particularly when you
get to more threads than CPUs. The shape of the curve shows you whether the
system handles overload gracefully.
Q41:
My graph looks strange. Are there two graph formats?
A41: The
reporter tools
can make two types of html: html with an html graph, and html with a jpg graph.
The jpg graph creation requires Java 2 and, under UNIX, the DISPLAY environment
variable set to a working X display (this may require setting
xhost + on
the display). The html graph is provided for those without Java 2 or without
the display device.
If, initially, the result appears in the html with an html graph format,
you can make the other format later (after adding Java 2 or an X host display)
by rerunning the reporter. See the User's Guide for instructions.
Compliant results sent for publication to SPEC will appear on the SPECjbb2000 website in the html with
a jpg graph format.
Ordering SPECjbb2000
Q42: What
do you get when you order SPECjbb2000?
A42: You
get a CD-ROM of the SPEC-licensed product.
Q43:
What is the price of a SPECjbb2000 license and when will it be available?
Q44:
How do I contact SPEC for more information?
A44:
Standard Performance Evaluation Corporation (SPEC)
6585 Merchant Place, Suite 100
Warrenton, VA 20187, USA
Phone: 1 540 349 7878 FAX: 1 540 349 5992
E-Mail:
info@spec.org
This FAQs document was prepared by Kaivalya Dixit, Anne Troop, Rajiv Arora,
Robert Reynolds, Steve Munroe, Walter Bays, Ruth Morgenstein and others. Last
Modification 06/2/2000 - Anne
