IEEE Technical Committee on Scalable Computing

- Technical Area -

Performance Evaluation: Benchmarking

Introduction

System performance may be viewed quite differently by a computational physicist and an Internet Service Provider. And while one ISP views performance in terms of transactions per second, another may focus on system availability. Each user must decide which of the many performance metrics are most important to their operation and choose a benchmarking suite that measures them. Benchmark results can uncover bottlenecks in your system and help quantify its performance. Once a baseline performance measurement is established, subsequent benchmarking can be used to validate a system tuning effort.

This Technical Area page includes links to a variety of benchmarking tools. Some of them give very detailed results about a specific computing subsystem and some are designed to present an overall picture of your system's performance. While we can neither endorse nor recommend any of them, each of them may contain tutorial information that can help you better understand what a benchmark can do for you, and what it cannot.

Conferences and Workshops

WCNC 2010 - IEEE Wireless Communications and Networking Conference, April 18-21, 2010, Sydney, Australia.

PerCom 2010 - 8th Annual IEEE International Conference on Pervasive Computing and Communications, March 29 - April 2, 2010, Mannheim, Germany.

PerNEM 2010 - 1st Annual Workshop on Pervasive Networks for Emergency Management, March 29 - April 2, 2010, Mannheim, Germany.

ICDS 2010 - 4th International Conference on Digital Society, February 10-15, 2010, St. Maarten, Netherlands Antilles.

HiPEAC 2010 - 5th International Conference on High Performance and Embedded Architectures and Compilers, January 25-27, 2010, Pisa, Italy.

AusPDC 2010 - 8th Australasian Symposium on Parallel and Distributed Computing, January 18-22, 2010, Brisbane, Australia.

FCST 2009 (URL TBA) - 4th International Conference on Frontier of Computer Science and Technology, December 17-19, 2009, Shanghai, China.

MoMM2009 - 7th @WAS International Conference on Mobile Computing and Multimedia, December 14-16, 2009, Kuala Lumpur, Malaysia.

iiWAS2009 - 11th @WAS International Conference on Information Integration and Web-based Applications & Services, December 14-16, 2009, Kuala Lumpur, Malaysia.

ADCOM 2009 - 17th annual International Conference on Advanced Computing and Communications Conference, December 14-18, 2009, Bangalore, India.

ISAM-2009 - International Workshop on Information Security and Assurance Management, December 14-16, 2009, Kaohsiung, Taiwan.

I-Span'09 - 10th International Symposium on Pervasive Systems, Algorithms and Networks, December 14-16, 2009, Kaohsiung, Taiwan.

ICDCS'09 - 2nd International Workshop on Internet and Distributed Computing Systems, December 10-12, 2009, Jeju Island, Korea.

eScience 2009 - 5th IEEE International Conference on e-Science, December 7-9, 2009, Oxford, United Kingdom.

CloudCom'09 - 1st International Conference on Cloud Computing Technology, December 1-4, 2009, Beijing, China.

ICSOC'09 - 7th International Joint Conference on Service Oriented Computing, November 24-27, 2009, Stockholm, Sweden.

QShine 2009 - 6th International ICST Conference on Heterogeneous Networking for Quality, Reliability, Security and Robustness, November 23-26, 2009, Canary Islands, Spain.

AAA-IDEA 2009 - 3rd International Workshop on Advanced Architectures and Algorithms for Internet DElivery and Applications, November 26, 2009, Canary Islands, Spain.

ComputationWorld 2009 - A federated event focusing on advanced topics concerning the areas of computation, November 15-20, 2009, Athens, Greece.

SC09 - International Conference on High Performance Computing, Networking, Storage and Analysis, November 14-20, 2009, Portland, Oregon, USA.

CBHPC 2009 - Workshop on Component-Based High Performance Computing, November 15-16, 2009, Portland, Oregon, USA.

SSS 2009 - 11th International Symposium on Stabilization, Safety, and Security of Distributed Systems, November 3-6, 2009, Lyon, France.

SBAC-PAD 2009 - 21st International Symposium on Computer Architecture and High Performance Computing, October 28-31, 2009, Sau Paulo, Brazil.

NGNM 2009 - 6th International Workshop on Next Generation Networking Middleware, October 26-30, 2009, Venice, Italy.

DS-RT 2009 - 13th IEEE/ACM International Symposium on Distributed Simulation and Real Time Applications, October 25-28, 2009, Singapore, Singapore.

CloudComp 2009 - 1st International Conference on Cloud Computing, October 19-21, 2009, munich, Germany.

NSS 2009 - 3rd International Conference on Network and System Security, October 19-21, 2009, Gold Coast, Australia.

NPC 2009 - 6th IFIP International Conference on Network and Parallel Computing, October 19-21, 2009, Gold Coast, Australia.

Grid 2009 - IEEE Grid 2009 conference, October 13-15, 2009, Banff, Canada.

E2GC2 2009 - Energy Efficient Grids, Clouds and Clusters Workshop, October 13-15, 2009, Banff, Canada.

ADVCOMP 2009 - 3rd International Conference on Advanced Engineering Computing and Applications in Sciences, October 11-16, 2009, Sliema, Malta.

CIT 2009 - 9th IEEE International Conference on Computer and Information Technology, October 11-14, 2009, Xiamen, China.

InVeNet 2009 - 1st International Workshop on Intelligent Vehicular Networks, October 5, 2009, Macau SAR, China.

ScalCom-09 - IEEE International Confederate on Scalable Computing and Networking, September 25-27, 2009, Dalian, China.

ICPP-2009 - 38th International Conference on Parallel Processing, September 22-25, 2009, Vienna, Austria.

LBS-09 - Workshop on Location-based services and applications, September 22, 2009, Vienna, Austria.

MCSoC-09 - 4th International Symposium on Embedded Multicore Systems-on-Chip, September 22, 2009, Vienna, Austria.

SRMPDS'09 - 5th International Workshop on Scheduling and Resource Management for Parallel and Distributed Systems, September 22, 2009, Vienna, Austria.

MASCOTS 2009 - 17th Annual Meeting of the IEEE/ACM International Symposium on Modelling, Analysis and Simulation of Computer and Telecommunication Systems, September 21-23, 2009, Imperial College London, UK.

PACT 2009 - 18th International Conference on Parallel Architectures and Compilation Techniques, September 12-16, 2009, Raleigh, North Carolina, USA.

MEDEA 2009 - Workshop on MEmory performance: DEaling with Applications, systems and architecture, September 12-16, 2009, Raleigh, North Carolina, USA.

Euro MPI/PVM 2009 - 16th European PVM/MPI Users' Group Meeting, September 7-10, 2009, Espoo, Finland.

Cluster 2009 - IEEE International Conference on Cluster Computing, August 29-September 4, 2009, New Orleans, Louisiana, USA.

PPAC 2009 Workshop on Parallel Programming on Accelerator Clusters - August 31, 2009, New Orleans, Louisiana, USA.

CSE-09 - 12th IEEE International Conference on Computational Science and Engineering, August 29-31, 2009, Vancouver, Canada.

SEAA 2009 - 35th Euromicro Conference on Software Engineering and Advanced Applications, August 27-29, 2009, Patras, Greece.

ABC:MI - 6th Workshop on Agent Based Computing: from Model to Implementation, August 27-29, 2009, Patras, Greece.

ICCGI 2009 - 4th International Multi-Conference on Computing in the Global Information Technology, August 23-29, 2009, Cannes/Cote d'Azur, France.

PacRim'09 - IEEE Pacific Rim Conference on Communications, Computers, and Signal Processing, August 23-26, 2009, Victoria, British Columbia, Canada.

Salford 2009 - Salford Data Mining Conference, August 23-25, 2009, San Diego, California, USA.

ScalCom'09 - 1st IEEE International Conference on Scalable Computing and Communications, August 23-25, 2009, Dalian, China.

ADMA 2009 - 5th International Conference on Advanced Data Mining and Applications, August 17-19, 2009, Beijing, China.

ISPA 2009 - 7th IEEE International Symposium on Parallel and Distributed Processing with Applications, August 10-12, 2009, Chengdu and Jiuzhai Valley, China.

ICCCN 2009 - 18th International Conference on Computer Communications and Networks, August 2-6, 2009, San Francisco, California, USA.

PMECT 2009 - 3rd IEEE International Workshop on Performance Modeling and Evaluation in Computer and Telecommunication Networks, August 2-6, 2009, San Francisco, California, USA.

SN 2009 - 2nd International Workshop on Sensor Networks, August 2-6, 2009, San Francisco, California, USA.

COGCOM 2009 - 2nd International Workshop on Cognitive Networks and Communications, August 2-6, 2009, San Francisco, California, USA.

MCC 2009 - 1st International Workshop on Multimedia Computing and Communications, August 2-6, 2009, San Francisco, California, USA.

GridPeer 2009 - 1st IEEE Workshop on Grid and P2P Systems and Applications, August 2-6, 2009, San Francisco, California, USA.

SecureCPN 2009 - International Workshop on Security, Privacy and Trust of Computer and Cyber-Physical Networks, August 2-6, 2009, San Francisco, California, USA.

NanoCom 2009 - International Workshop on Nano, Molecular, and Quantum Communications, August 2-6, 2009, San Francisco, California, USA.

BIFE 2009 - 2nd IEEE International Conference on Business Intelligence and Financial Engineering, July 24-26, 2009, Beijing, China.

PDPTA'09 - International Conference on Parallel and Distributed Processing Techniques and Applications, July 13-16, 2009, Las Vegas, Nevada, USA.

SDMAS'09 - 3rd International Workshop on Scalable Distributed and Multi/Many-core Applications and Systems, July 13-16, 2009, Las Vegas, Nevada, USA.

CDES'09 - International Conference on Computer Design, July 13-16, 2009, Las Vegas, Nevada, USA.

NAS 2009 - 4th IEEE International Conference on Networking, Architecture, and Storage, July 9-11, 2009, Zhang Jia Jie, Hunan, China.

ATC-09 - 6th International Conference on Autonomic and Trusted Computing, July 7-10, 2009, Brisbane, Australia.

CPI'09 - 1st International Symposium on Cyber-Physical Intelligence, July 7-10, 2009, Brisbane, Australia.

UIC'09 - 6th International Conference on Ubiquitous Intelligence and Computing, July 7-10, 2009, Brisbane, Australia.

Benchmarking Links

HPL: the HPL benchmark is a software package that solves a (random) dense linear system in double precision (64 bits) arithmetic on distributed-memory computers. It can thus be regarded as a portable as well as freely available implementation of the High Performance Computing Linpack Benchmark.

LINPAK: the LINPACK benchmark is a collection of Fortran subroutines that analyze and solve linear equations and linear least-squares problems. The package solves linear systems whose matrices are general, banded, symmetric indefinite, symmetric positive definite, triangular, and tridiagonal square. In addition, the package computes the QR and singular value decompositions of rectangular matrices and applies them to least-squares problems. LINPACK uses column-oriented algorithms to increase efficiency by preserving locality of reference.

LMBENCH: the LMBENCH benchmark is a suite of simple, portable, ANSI/C microbenchmarks for UNIX/POSIX. In general, it measures two key features: latency and bandwidth. LMBENCH is intended to give system developers insight into basic costs of key operations.

NAS: the NAS Parallel Benchmarks (NPB) are a small set of programs designed to help evaluate the performance of parallel supercomputers. The benchmarks, which are derived from computational fluid dynamics (CFD) applications, consist of five kernels and three pseudo-applications. The NPB come in several flavors. NAS solicits performance results for each from all sources.

SPEC: the Standard Performance Evaluation Corporation is a non-profit corporation formed to establish, maintain and endorse a standardized set of relevant benchmarks that can be applied to the newest generation of high-performance computers. SPEC develops benchmark suites and also reviews and publishes submitted results from SPEC member organizations and other benchmark licensees.

SPLASH: Stanford's SPLASH benchmark evaluates performance of shared memory multiprocessors.

STREAM: the STREAM benchmark is the de facto industry standard benchmark for the measurement of computer memory bandwidth. The STREAM benchmark measures "real world" bandwidth sustainable from ordinary user programs -- not the theoretical "peak bandwidth" provided by most vendors.

TPC: the Transaction Processing Council defines transaction processing and database benchmarks and delivers trusted results to the industry.

Performance Metrics

Cost-Effectiveness: An application may have an execution cost requirement that is just as important as execution time. As the number of nodes in use by the application increases, the execution time goes down (good), but the cost goes up (bad) due to the increased inter-process communication overhead. It may be important to determine an optimal number of nodes that minimizes both the execution time and the operating cost.

Cost/Performance Ratio: One modern approach to low-cost computing is to interconnect a whole lot of PCs over a very fast external data network, and either customize the operating system or run specialized middleware that turns them into a multicomputer, or cluster. A parallel application that was intended to run on a shared memory multiprocessor will run on a cluster. The cost/performance ratio tells you how much would it cost to buy a multiprocessor that would give the SAME performance as the cluster, compared to the cost of the cluster. Ratios of 10:1 were common in the early days of cluster computing, meaning the cluster ran an application as well as a commercial processor costing ten times as much. Today, the ratios have dropped to about 3:1, but a project with more time than money has options that it did not have twenty years ago.

Execution Times: Real-time applications may have an execution time limit. Calculating the location of a landing zone for an incoming missile, for example, should complete before it arrives.

Processor Speed: Some applications have to process a task with a complexity that varies dramatically over time. Predicting the weather, for example, has a complexity that depends upon current local conditions. In these cases, the system may have a speed requirement in gflops (billions of floating point operations) per second, for example, rather than a specific execution time limit.

Speedup: this standard measure of performance improvement calculates the ratio of the completion times on some reference system to the system under test. Completion time is often that of a collection of batch jobs or tasks, measured from the time the first starts until the last one completes.

Throughput: Many applications process a stream of tasks whose arrival rate is more critical than the complexity of each task; the time it takes to process one of them is not as important as how many can be processed over a finite period of time. Be sure to include more tasks in your test stream than you have nodes in your cluster to process them to ensure a backlog at some point in the processing time period.

Utilization/Loading: Parallel applications may distribute processes across a network of computer nodes. A balanced system is one in which the loading at each node, or processor utilization, is about equal. Fault tolerance aside, one would like to have the workload distributed evenly across each node in the distributed system's network.

Note: If you would like a related item added to any of the above lists, please send the details to the Technical Area Coordinator:

Contact

Dr. Alex Vrenios
DSRLab, LLC
Distributed Systems
Research Laboratory
Phoenix, Arizona, USA
Voice: +1 602 377-7720
Email: alex@DSRLab.com

This web page is updated regularly. Your questions, suggestions and comments are welcome!

Last updated July 2, 2009