CPU98 Search Program

SPEC holds to the principle that better benchmarks can be developed from actual applications. With this in mind, SPEC is seeking to encourage those outside of SPEC to assist SPEC in locating applications that could be used in the next SPEC CPU-intensive benchmark suite. The following is the text of SPEC's Search Program.

Note that the SPEC CPU98 Search Program concluded on December 1, 1997 and no new submissions are being accepted. Programs submitted by December 1, 1997 are still being worked through the CPU98 Search process.

The SPEC CPU98 Benchmark Search Program

Rules, Categories, & Criteria

What is the SPEC CPU98 Search Program?

Sponsored by the Standard Performance Evaluation Corporation (SPEC, a non-profit consortium), the CPU98 Benchmark Search Program is designed to encourage and recognize achievements of the academic and industrial community in providing and developing application code and datasets that could be used as components of the next revision of the SPEC CPU benchmark suites.

For each step of the SPEC process that a submission passes, SPEC will compensate the program submittor (in recognition of the submittor's effort and skill). A submission that passes all of the steps and is included in the next SPEC CPU benchmark suite will, overall, receive $5000 and a license for the released benchmark suite.

What is SPEC looking for?

Overall, the SPEC CPU benchmarks have provided metrics for comparing systems on a known workload, with emphasis on the system's processor, memory hierarchy and compiler. SPEC is expecting the next CPU suite to continue this trend.

SPEC has held to the principle that better benchmarks are based on applications. These applications can come from any area of work. For example, the current SPEC CPU95 suite includes applications from the following areas:

• AI game theory
• processor simulators
• compilers
• interpreters
• data compression
• databases
• weather prediction
• fluid dynamics
• physics
• chemistry
• image processing

And SPEC is not limiting consideration to just these areas. A proposed application may come from any area or field of work. The submission must contain a description of the utility of the program. One criterion SPEC will use for all categories is how close the entry is to the state of the art of the given field.

In terms of other criteria, SPEC also considers the following important for the next CPU benchmark suite:

• that the program can be made compute bound
• that the program can be made portable across difference hardware architectures and operating systems

The SPEC CPU98 Search Program Official Rules

No purchase is necessary to participate. In all cases, the burden of proof is on the submittors. SPEC will make an honest effort to evaluate relative characteristics of different programs solving different problems running on different machines, but they will depend primarily on the submitted material. All evaluation and awards are within the sole discretion of SPEC.

Who May Submit Benchmarks?

Open to residents, of legal majority age or older, of Argentina, Australia, Austria, Belgium, Brazil, Bulgaria, Canada (except Quebec), Chile, Colombia, Croatia, Czech Republic, Denmark, Egypt, Finland, Germany, Greece, Hong Kong, Hungary, Ireland, Israel, Japan, Korea, Malaysia, Mexico, the Netherlands, New Zealand, Norway, the Republic of the Philippines, Poland, Portugal, Romania, Russia, Singapore, South Africa, Spain, Sweden, Switzerland, Taiwan, Turkey, United Kingdom, and the United States (except Vermont) with the exception of any employees or contractors of any of the SPEC OSG member companies together with those with whom such persons are domiciled, and government employees associated with procurement (US, international, state and local), unless otherwise authorized under the laws of the jurisdiction concerned.

Entries may be submitted on an individual entrant basis (defined as a natural person), or on a group or team basis, in which case the registration and submission will identify a leader of such group or team.

Corporate entities are not eligible.

Award program is sponsored by Standard Performance Evaluation Corporation, 10754 Ambassador Drive, Suite 201, Manassas, Virginia, 20109, U.S.A.

Participation Constitutes Agreement to Official Rules

Participation in the Search Program constitutes Submittor's full and unconditional agreement to and acceptance of these Official Rules.

Timing

Submissions for the first step in the search program will be accepted by SPEC beginning March 3, 1997 and ending December 1, 1997 at midnight United States Pacific Standard Time.

Status updates through the rest of the process will be provided to the submittors at least every three months until the release of the next SPEC CPU suite.

How To Enter

Step 1:

Submit a CPU Search Program Proposal to SPEC.

Note that the Search program officially ended on December 1, 1997 and that SPEC is no longer accepted submissions. More information can be obtained by contacting SPEC at:

• SPEC
  Search Program
  10754 Ambassador Drive, Suite 201
  Manassas, Virginia 20109

While the program was ongoing, SPEC needed the following information:

• the submittor's name and contact information
• the name of the program
• a description of what the program does
• a statement declaring that the submittal has the ability to give SPEC the right to distribute the program as part of a SPEC benchmark suite

For all submissions submitted during the program, the following basic process was used:

Upon receipt of the proposal, SPEC will submit it to the SPEC CPU Subcommittee. At the second CPU Subcommittee meeting following submission of the proposal (a meeting is currently held every two weeks), the SPEC CPU Subcommittee will vote on acceptance of the proposal, which is solely within SPEC's discretion. The submittor will then be notified of the results. For the first proposal accepted by SPEC from a single submittor, the submittor will receive a SPEC CPU95 license (or a refund if they already own a license) and directions for preparing their code for SPEC. All submittors of accepted proposals will be assigned a contact person within the SPEC CPU Subcommittee.

The SPEC CPU Subcommittee voting process is defined as:

• all SPEC Open Systems Group members and associate members are members of the SPEC CPU Subcommittee
• each OSG member or associate member has one vote
• a minimum of 7 members must be present
• a vote passes if supported by one greater that 2/3 of those voting (yes) or (no)

Step 2:

If the proposal is accepted by the SPEC CPU subcommittee, the submittor must now prepare their code for submission to SPEC and then submit it to SPEC. The SPEC CPU subcommittee has the following expectations:

• that the submittal has signed a "permission to use" form for the submitted code
• that the submittal has prepared the code in the SPEC CPU95 framework with
  • the necessary source code
  • a proposed small test input (runs 1-30 seconds just to confirm that the binary works) with a verifiable output file(s).
  • a proposed medium training input (runs 10-60 seconds that is a small input that could possibly be used for feedback directed optimization) with a verifiable output file(s).
  • a proposed reference input to be used as the official benchmark workload with a verifiable output file(s).
  • a completed SPEC Description file.
• that the SPEC CPU Subcommittee will be able to observe the following on the SPEC CPU Subcommittee test machine:
  • That the program can be compiled using the SPEC CPU95 framework
  • That the program can be run with the SPEC CPU95 framework.
  • That the program can correctly verify the program's output with the SPEC CPU95 framework.
• On the SPEC CPU95 Subcommittee test machine (a 200MHz Pentium Pro processor system) that:
  • Over 95% of the execution time is spent in the submitted code.
  • Over 95% of the time is compute bound.
  • The program runs at least two minutes.

If, in SPEC's sole judgement, the submission does not meet these criteria, the submittor will be notified as to what expectation was not met. If the submitted code meets these criteria, the submittor is awarded $1000 and moves to the next step.

Step 3:

The SPEC CPU Subcommittee votes in its sole discretion whether to accept the program as an official candidate for the next SPEC CPU benchmark suite. If the program is accepted the submittor receives $1000 and the program enters the official SPEC CPU suite process, where it is evaluated and tested by the SPEC CPU Subcommittee members in his, her or its sole judgement.

Step 4:

If the program makes it through the SPEC CPU suite process and is recommended by the SPEC CPU Subcommittee and the SPEC Open Systems Steering Committee in their sole discretion to the OSG membership, the submittor receives $1000 and the program moves to the next step. Otherwise, the SPEC CPU Subcommittee will notify the submittor as to why the program was removed from consdieration.

Step 5

If the program is accepted by the OSG membership in its sole discretion (requiring a majority of those voting), then the program is included in the suite and the submittor will receive $2000 and a license for the suite when it is released.

Submittors may enter as many times as they wish however, each entry must be preceded by a separate registration. Sponsor will not return any entries to the submittor.

Personal Data

Sponsor will make its best effort to use personal data about submittor only for the administration of the award program in accordance with these Official Rules and for internal marketing purposes. Submittor agrees to such use of the personal data.

Encryption

No entries may include encryption as a feature or part of a submission.

Important Terms

Sponsor is not responsible for lost, late, incomplete, illegible or misdirected e-mail, for failed, partial or garbled computer transmissions, or for technical failures of any kind. Sponsor reserves the right to cancel or modify the awards, if fraud, technical failures or other events or circumstances impair the integrity of the award program as determined by Sponsor in its sole discretion.

Submittor agrees that Sponsor's only obligations are to submit entries from eligible submissions to the CPU Subcommittee according to the procedures and criteria set forth in these Official Rules, and to award the compensation set forth herein to submittors as criteria in SPEC's sole judgement are met, subject to the terms and conditions herein.

Submittors will be held harmless from any liability incurred from the intended or unintended use of the application as a benchmark or application.

Submittor agrees that Sponsor's obligations are fair and adequate consideration for any entry submitted, and that Submittor is not entitled to and shall not seek any further compensation.

By participating in the Search program, Submittor waives all claim to intellectual property rights in the entry, including patent rights and copyright, and waives all moral rights, except where prohibited. To the extent that such waiver is ineffective or unenforceable, Submittor hereby grants Sponsor a non-exclusive license to copy, modify, display, and sublicense the entry without geographical limitations or further compensation to submittor of any kind, for a period of five (5) years from the date of entry. Submittor agrees that e-mail shall satisfy any writing requirement which may apply to intellectual property licenses. Upon request of Sponsor, Submittor agrees to obtain written consent from the owner of the copyright in the applet, if that person is not Submittor, and to execute any documents required to effectuate the terms of these Official Rules.

As a condition of submission to this award program, Submittor agrees that: (1) any and all disputes, claims, and causes of action arising out of or connected with this award program, or any prizes awarded, shall be resolved individually, without resort to any form of class action, and exclusively by arbitration under the International Arbitration Rules of the American Arbitration Association in San Francisco, California; (2) any and all claims, judgments and awards shall be limited to actual out-of-pocket costs incurred, including costs associated with entering this Contest but in no event atterneys' fees; and (3) under no circumstances will submittor be permitted to obtain awards for and submittor hereby waives all rights to claim punitive, incidental and consequential damages and any other damages, other than for actual out-of-pocket expenses, and any and all rights to have damages multiplied or otherwise increased.

All issues and questions concerning the construction, validity, interpretation and enforceability of these Official Rules, or the rights and obligations of Submittor and Sponsor in connection with the Award Program, shall be governed by, and construed in accordance with, the laws of the State of California, U.S.A., without giving effect to any choice of law or conflict of law rules or provisions (whether of the State of California, U.S.A., or any other jurisdiction), which would cause the application of the laws of any jurisdiction other than the State of California, U.S.A.

Taxes, customs duties, fees, freight charges and other related charges, on prizes are the sole responsibility of the winners. Winners may waive the right to receive compensation. Sponsor will reimburse winners up to U.S. $1,000 for payments made for customs duties, fees, freight charges (but not taxes, including taxes on income, goods and services, and value added taxes), upon presentation by submittal of original, official receipts for such payments.

Submittal Notification

Submittors will be notified by registered mail and may be required to complete an affidavit of eligibility and publicity release (except where prohibited by law), which must be returned within 20 days from receipt by submittal . Failure to sign and return the affidavit or release, or to comply with any term or condition in these Official Rules, may result in disqualification and forfeiture of compensation.

This Program is subject to all applicable laws. Void where prohibited.

Go To: [Home] [SPEC] [OSG] | [CPU98] [Notes] [Submit] | [CPU95]