What Is SPEC?
SPEC is a non-profit organization that establishes, maintains and endorses standardized benchmarks and tools to evaluate performance and energy efficiency for the newest generation of computing systems. Its membership comprises more than 120 leading computer hardware and software vendors, educational institutions, research organizations, and government agencies worldwide. The founders of this organization believe that the user community will benefit greatly from an objective series of applications-oriented tests, which can serve as common reference points and be considered during the evaluation process. While no one benchmark can fully characterize overall system performance, the results of a variety of realistic benchmarks can give valuable insight into expected real performance.
Legally, SPEC is a non-profit corporation registered in California.
SPEC basically performs two functions:
-
SPEC develops suites of benchmarks intended to measure computer performance. These suites are packaged with source code and tools and are extensively tested for portability before release. They are available to the public for a fee covering development and administrative costs. By license agreement, SPEC members and customers agree to run and report results as specified in each benchmark suite’s documentation.
-
SPEC publishes news and benchmark results at https://www.spec.org/ This provides a centralized source of information for SPEC benchmark results. Both SPEC member and non-SPEC member companies may publish on the SPEC website, though there is a fee for non-members. (Note that results may be published elsewhere, as long as the format specified in the SPEC Run Rules and Reporting Rules is followed.)
For more in depth inquiry, we have more information online about SPEC, including its background, the philosophy, and organizational structure.
How to Contact SPEC?
For SPEC contact information, see the SPEC Contact Info Page
What Groups Make Up SPEC?
SPEC is comprised of six groups:
-
Embedded Group (EG): Develops industry-standard benchmarks for the hardware and software used in autonomous driving, mobile imaging, the Internet of Things, mobile devices, and many other applications.
-
Graphics and Workstation Performance Group (GWPG): Develops consistent, repeatable graphics and workstation performance benchmarks in a way that reflects user experiences with popular applications.
-
High Performance Group (HPG): Develops benchmarks to represent high-performance computing applications for standardized, cross-platform performance evaluation.
-
International Standards Group (ISG): Oversees the establishment of standardized benchmarks primarily developed for the use in government regulations and programs, and collaborates with national and international standard development organizations to enhance global standards.
-
Open Systems Group (OSG): Develops component- and systems-level benchmarks for workstations and servers running open operating system environments.
-
Research Group (RG): Promotes research on benchmarking methodologies and tools used to deliver benchmark suites and performance analysis frameworks for established and newly emerging technologies.
Why Should I Join SPEC?
Membership in SPEC is open to any interested company or entity that is willing to commit to SPEC’s standards and is able to provide for the support of the organization.
Membership is group-based. One can be a member of one or more groups depending upon one’s interests. Each group (EG, GWPG, HPG, ISG, OSG, RG) is run independently; each member is responsible for paying the full dues and other responsibilities for each group they join – there are no “bulk discounts”.
SPEC does make an effort to accommodate the participation of academic institutions and other non-profit organizations with reduced (or waived) dues and other arrangements. SPEC has “associates”, which pay little or no dues, but still get full access to the benchmark development process.
The dues go towards offsetting SPEC’s operating costs. Typically these dues are a few thousand dollars per year with a similar initiation fee to cover starting costs; however, the exact values vary from group to group (and potentially year to year) depending upon the specific costs.
Membership allows for full access to existing benchmarks (this included company-wide licenses for current benchmark products) and full participation in ongoing development of new benchmarks. Additionally, members get unlimited result publication through SPEC (as membership dues already cover much of the cost). The benefits for associates are the same as for members except for having voting rights. The intention for associates is that they can act in an advisory capacity to SPEC, getting first-hand experience in an area that is widely neglected in academia but nonetheless very important in the “real world”, and providing technical input to SPEC’s tasks.
All members and associates of each group are welcome at all meetings for that group. Meetings are usually held four to six times a year, with subcommittee teleconferences held on an as-needed basis (often weekly or bi-weekly during benchmark development). Final decisions are usually made by a “steering committee” elected by the membership of the group, with benchmark standards needing ratification by the groups full membership.
For further information about membership in any of SPEC’s groups, contact SPEC at info@spec.org.
What Are the SPEC Products and Services?
SPEC’s main product is the benchmark suites. The code in the suites are developed by SPEC from code donated by various sources. SPEC works on portability and creates tools and meaningful workloads for the codes chosen as benchmarks. Therefore, the SPEC benchmarks are not the same as public domain programs that might exist under a similar name or the same name, and their execution times will, in general, be different.
The SPEC benchmark sources are generally available; though, they are licensed software products. Each SPEC group is charging separately for its benchmark suites. The income from the benchmark source sales is intended to support the administrative costs of the corporation. SPEC distributes its benchmarks as downloads via Cleverbridge.
What does “retired” mean in relation to SPEC benchmarks?
SPEC will often retire a benchmark when there is a more recent replacement or sometimes when the original benchmark has become too out-of-date to continue to support. "Retired" means that SPEC no longer provides technical support or accepts result submissions for posting on the SPEC web site. Recognizing that there may be historical interest in these older benchmarks, in many cases it may still be possible to obtain copies of such a benchmark by contacting the SPEC office, but the buyer should be aware that additional support will not be provided. Notice of planned retirements will be posted on the SPEC web site, and such news is usually reported months in advance of the retirement date.
Are there trial or evaluation versions of the SPEC benchmarks available?
We do not offer trial versions of the SPEC benchmarks. If you are dissatisfied with the product the original software purchase price can be refunded within 30 days of purchase. To request a refund, please contact SPEC at info@spec.org
What kind of license is included with the software?
SPEC benchmark suites are provided with a site license which
allows the licensee to install and run the software on as many
systems within their organization as they wish. You do not need
to purchase multiple licenses.
A sample general site license agreement
and the sample SERT site license agreement
are available for review prior to purchase.
Who qualifies for the non-profit and educational pricing?
Educational discounts are available only when the software is purchased by and licensed to an educational institution; the end user must be a faculty member or a full-time employee of the institution, not a student.
Non-profit pricing is available to recognized tax-exempt organizations and government agencies. Individuals do not qualify for the discounted pricing.
How Do I Order from SPEC?
SPEC products can be obtained by using our online order form. For other methods or additional questions about ordering, contact the SPEC staff.
Which System/CPU/Component Would Be Better for Me?
Unfortunately, SPEC does not make recommendations on computer purchases. SPEC will provide what data it can (all available on the Web site https://www.spec.org/) but can not offer opinions on it.
Without knowing a lot about your exact workload, it would be difficult to provide any useful specific advice; and providing general advice can get us all in legal trouble.
What About SPEC 1170?
SPEC 1170 does not have anything to do with SPEC, the Standard Performance Evaluation Corporation. It’s a different spec.
SPEC 1170 is an older name for what is now called UNIX 98, which is an X/Open brand for compliance with a "Single UNIX Specification" – basically a standardized UNIX. More information about UNIX 98 can be obtained from: http://www.unix-systems.org/
Is SPEC requiring anything with regard to publicly reported security mitigations?
From time to time there can be reports of security mitigation concerns. SPEC’s current approach with regard to security mitigations can be seen here: Security Q&A.
Specific questions can be addressed to: info@spec.org.
Security Mitigations and SPEC benchmarks Q1: What is SPEC doing with regard to reported computer security vulnerabilities and their mitigation? As noted on www.spec.org, SPEC is a non-profit corporation formed to establish, maintain and endorse standardized benchmarks and tools to evaluate performance and energy efficiency. SPEC also reviews results as to their compliance with run rules, and, after review, publishes (on www.spec.org) rule compliant results from SPEC benchmark licensees. Note that SPEC does not perform any testing of its own nor does SPEC provide editorial comment/judgement on rule compliant results; thus in the context of security vulnerabilities and their mitigation, SPEC expects: