IEEE Technical Committee on Scalable Computing - Technical Area -
Performance Evaluation: Benchmarking
(Contact Information)
Introduction
System performance may be viewed quite differently by a computational physicist and an Internet Service Provider. And while one ISP views performance in terms of transactions per second, another may focus on system availability. Each user must decide which of the many performance metrics are most important to their operation and choose a benchmarking suite that measures them. Benchmark results can uncover bottlenecks in your system and help quantify its performance. Once a baseline performance measurement is established, subsequent benchmarking can be used to validate a system tuning effort.
This Technical Area page includes links to a variety of benchmarking tools. Some of them give very detailed results about a specific computing subsystem and some are designed to present an overall picture of your system's performance. While we can neither endorse nor recommend any of them, each of them may contain tutorial information that can help you better understand what a benchmark can do for you, and what it cannot.
Conferences and Workshops
ICPP-2009 - 38th International Conference on Parallel Processing, September 22-25, 2009, Vienna, Austria.
MASCOTS 2009 - 17th Annual Meeting of the IEEE/ACM International Symposium on Modelling, Analysis and Simulation of Computer and Telecommunication Systems, September 21-23, 2009, Imperial College London, UK.
Cluster 2009 - IEEE International Conference on Cluster Computing, August 29-September 4, 2009, New Orleans, Louisiana, USA.
NAS 2009 - 4th IEEE International Conference on Networking, Architecture, and Storage, July 9-11, 2009, Zhang Jia Jie, Hunan, China.
ATC-09 - 6th International Conference on Autonomic and Trusted Computing, July 7-10, 2009, Brisbane, Australia.
UIC'09 - 6th International Conference on Ubiquitous Intelligence and Computing, July 7-10, 2009, Brisbane, Australia.
ISPDC'2009 - 8th International Symposium on Parallel and Distributed Computing, June 30-July 4, 2009, Lisbon, Portugal.
HPCC-09 - 11th IEEE International Conference on High Performance Computing and Communications, June 25-27, 2009, Seoul, Korea.
ICAC-09 - 6th IEEE International Conference on Autonomic Computing, June 15-19, 2009, Barcelona, Spain.
Performance 2009 - Joint International Conference on Measurement and Modeling of Computer Systems, June 15-19, 2009, Seattle, Washington, USA
AINA-09 - 23rd IEEE International Conference on Advanced Information Networking and Applications, May 26-29, 2009, Bradford, UK.
MAW-09 - IEEE International Symposium on Mining and Web, May 26-29, 2009, Bradford, UK.
QuEST-09 - International Workshop on Quantitative Evaluation of large-scale Systems and Technologies, May 26-29, 2009, Bradford, UK.
IPDPS 2009 - IEEE International Parallel & Distributed Processing Symposium, May 25-29, 2009, Rome, Italy.
HP-PAC 2009 - 5th Workshop on High-Performance, Power-Aware Computing, May 25, 2009, Rome, Italy.
Hot-P2P 2009 - 6th International Workshop on Hot Topics in Peer-to-Peer Systems, May 29, 2009, Rome, Italy.
ICCS 2009 - International Conference on Computational Science, May 25-27, 2009, Baton Rouge, Louisiana, USA.
atICCS 2009 - Workshop on Using Emerging Parallel Architectures for Computational Science, May 25-27, 2009, Baton Rouge, Louisiana, USA.
CCGrid 2009 - 9th IEEE/ACM International Symposium on Cluster Computing and the Grid, May 18-21, 2009, Shanghai, China.
Cloud 2009 - International Workshop on Cloud Computing, May 18-21, 2009, Shanghai, China.
IRADSN'09 - 4th International Symposium on Innovations and Real-Time Applications of Distributed Sensor Networks, May 18-21, 2009, Hangzhou, China.
ICWM2009 - 4th International Workshop on Workflow Management, May 4-8, 2009, Geneva, Switzerland.
SYSTOR 2009 - Israeli Experimental Systems Conference, May 4-6, 2009, Haifa, Israel.
InfoSys 2009 - IARIA InfoSys Conference, April 21-25, 2009, Valencia, Spain.
ICAS 2009 - 5th International Conference on Autonomic and Autonomous Systems, April 21-25, 2009, Valencia, Spain.
ICNS 2009 - 5th International Conference on Networking and Services, April 21-25, 2009, Valencia, Spain.
INTENSIVE 2009 - 1st International Conference on Intensive Applications and Services, April 21-25, 2009, Valencia, Spain.
LMPCNAP 2009 - 1st International Workshop on Learning Methodologies and Platforms used in the Cisco Networking Academy, April 21-25, 2009, Valencia, Spain.
VIPSI-09 - VIP Symposia on Internet related research with elements of M+I+T++, April 2-5, 2009, Belgrade, Serbia.
PerCom 2009 - 7th Annual IEEE International Conference on Pervasive Computing and Communications, March 16-20, 2009, Dallas, TX, USA.
PerSeNS 2009 - 5th IEEE PerCom International Workshop on Sensor Networks and Systems for Pervasive Computing, March 16-20, 2009, Dallas, TX, USA.
ARES 2009 - 4th International Conference on Availability, Reliability and Security, March 16-19, 2009, Fukuoka, Japan.
SecSE 2009 - 3rd International Workshop on Secure Software Engineering, March 16-19, 2009, Fukuoka, Japan.
GlobeNet 2009 - IARA GlobeNet Conference, March 1-6, 2009, Gosier, Guadeloupe, French Caribbean.
DB 2009 - 1st International Conference on Advances in Databases, March 1-6, 2009, Gosier, Guadeloupe, French Caribbean.
ICN 2009 - 8th International Conference on Networks, March 1-6, 2009, Gosier, Guadeloupe, French Caribbean.
ICONS 2009 - 4th International Conference on Systems, March 1-6, 2009, Gosier, Guadeloupe, French Caribbean.
PDP2009 - 17th Euromicro International Conference on Parallel, Distributed, and network-based Processing, February 18-20, 2009, Weimar, Germany.
DigitalWorld 2009 - Digital World 2009, February 1-6, 2009, Cancun, Mexico.
ACHI 2009 - 2nd International Conferences on Advances in Computer-Human Interactions, February 1-6, 2009, Cancun, Mexico.
eKNOW 2009 - International Conference on Information, Process, and Knowledge Management, February 1-6, 2009, Cancun, Mexico.
eL&mL 2009 - International Conference on Mobile, Hybrid, and On-line Learning, February 1-6, 2009, Cancun, Mexico.
eTELEMED 2009 - International Conference on eHealth, Telemedicine, and Social Medicine, February 1-6, 2009, Cancun, Mexico.
GEOWS 2009 - International Conference on Advanced Geographic Information Systems & Web Services, February 1-6, 2009, Cancun, Mexico.
ICDS 2009 - 3rd International Conference on Digital Society, February 1-6, 2009, Cancun, Mexico.
ICQNM 2009 - 3rd International Conference on Quantum, Nano and Micro Technologies, February 1-6, 2009, Cancun, Mexico.
AusGrid 2009 - 7th Australasian Symposium on Grid Computing and e-Research, January 20-23, 2009, Wellington, New Zealand.
MMCN 2009 - SPIE/ACM Multimedia Computing and Networking 2009, January 18-22, 2009, San Jose, USA.
ICCIT'08 - 11th IEEE International Conference on Computer and Information Technology, December 25-27, 2008, Khulna, Bangladesh.
IDCS'08 - International Workshop on Internet and Distributed Computing Systems, December 25-27, 2008, Khulna, Bangladesh.
EUC 2008 - IEEE/IFIP International Conference on Embedded and Ubiquitous Computing, December 17-20, 2008, Shanghai, China.
HiPC 2008 - 15th IEEE International Conference on High Performance Computing, December 17-20, 2008, Bangalore, India.
WUGC 08 - Workshop on Grid and Utility Computing, December 17, 2008, Bangalore, India.
SENOPT'08 - Workshop on Service-Oriented Engineering and Optimizations, December 17, 2008, Bangalore, India.
WoNGeN 2008 - Workshop on Next Generation Wireless Networks, December 17, 2008, Bangalore, India.
(TBD) 2008 - Workshop on Advanced Memory Technologies for HPC Systems, December 17, 2008, Bangalore, India.
(TBD) 2008 - Workshop on High Performance FPGA/Reconfigurable Computing, December 17, 2008, Bangalore, India.
ICDM 2008 - IEEE International Conference on Data Mining, December 15-19, 2008, Pisa, Italy.
HPDM 2008 - 10th International Workshop on High Performance Data Mining, December 15, 2008, Pisa, Italy.
ADCOM 2008 - 16th International Conference on Advanced Computing and Communication, December 14-17, 2008, Chennai, India.
ICDCIT 2008 - 5th International Conference on Distributed Computing and Internet Technologies, December 10-13, 2008, New Delhi, India.
IWSOS 08 - 3rd International Workshop on Self-Organizing Systems, Self-Organizing Networks and Networked Systems, December 10-12, 2008, Vienna, Austria.
ISPA-08 - IEEE International Symposium on Parallel and Distributed Processing with Applications, December 10-12, 2008, Sydney, Australia.
QuaSM-08 - Workshop on Quality Management of Services and Middleware, December 10-12, 2008, Sydney, Australia.
CSS 2008 - IEEE International Workshop on Cyberspace Safety and Security, December 12, 2008, Sydney, Australia.
ICPADS'08 - 14th IEEE International Conference on Parallel and Distributed Systems, December 8-10, 2008, Melbourne, Australia.
CMG'08 - International Computer Measurement Group conference, December 7-12, 2008, Las Vegas, Nevada, USA.
eScience 2008 - 4th IEEE International Conference on e-Science, December 7-12, 2008, Indianapolis, Indiana, USA.
AHEMA 2008 - 1st International Workshop on Advances in High-Performance E-Science Middleware and Applications, December 11, 2008, Indianapolis, Indiana, USA.
ADAS 2008 - 1st International Workshop on Abstractions for Distributed Applications and Systems, December 12, 2008, Indianapolis, Indiana, USA.
Middleware 2008 - ACM/IFIP/USENIX 9th International Middleware Conference, December 1-5, 2008, Leuven, Belgium.
MGC2008 - 6th International Workshop on Middleware for Grid Computing, December 1-5, 2008, Leuven, Belgium.
PDCAT'08 - 9th International Conference on Parallel and Distributed Computing, Applications and Technologies, December 1-4, 2008, Dunedin, New Zealand.
Benchmarking Links
HPL: the HPL benchmark is a software package that solves a (random) dense linear system in double precision (64 bits) arithmetic on distributed-memory computers. It can thus be regarded as a portable as well as freely available implementation of the High Performance Computing Linpack Benchmark.
LINPAK: the LINPACK benchmark is a collection of Fortran subroutines that analyze and solve linear equations and linear least-squares problems. The package solves linear systems whose matrices are general, banded, symmetric indefinite, symmetric positive definite, triangular, and tridiagonal square. In addition, the package computes the QR and singular value decompositions of rectangular matrices and applies them to least-squares problems. LINPACK uses column-oriented algorithms to increase efficiency by preserving locality of reference.
LMBENCH: the LMBENCH benchmark is a suite of simple, portable, ANSI/C microbenchmarks for UNIX/POSIX. In general, it measures two key features: latency and bandwidth. LMBENCH is intended to give system developers insight into basic costs of key operations.
NAS: the NAS Parallel Benchmarks (NPB) are a small set of programs designed to help evaluate the performance of parallel supercomputers. The benchmarks, which are derived from computational fluid dynamics (CFD) applications, consist of five kernels and three pseudo-applications. The NPB come in several flavors. NAS solicits performance results for each from all sources.
SPEC: the Standard Performance Evaluation Corporation is a non-profit corporation formed to establish, maintain and endorse a standardized set of relevant benchmarks that can be applied to the newest generation of high-performance computers. SPEC develops benchmark suites and also reviews and publishes submitted results from SPEC member organizations and other benchmark licensees.
SPLASH: Stanford's SPLASH benchmark evaluates performance of shared memory multiprocessors.
STREAM: the STREAM benchmark is the de facto industry standard benchmark for the measurement of computer memory bandwidth. The STREAM benchmark measures "real world" bandwidth sustainable from ordinary user programs -- not the theoretical "peak bandwidth" provided by most vendors.
TPC: the Transaction Processing Council defines transaction processing and database benchmarks and delivers trusted results to the industry.
Performance Metrics
Cost-Effectiveness: An application may have an execution cost requirement that is just as important as execution time. As the number of nodes in use by the application increases, the execution time goes down (good), but the cost goes up (bad) due to the increased inter-process communication overhead. It may be important to determine an optimal number of nodes that minimizes both the execution time and the operating cost.
Cost/Performance Ratio: One modern approach to low-cost computing is to interconnect a whole lot of PCs over a very fast external data network, and either customize the operating system or run specialized middleware that turns them into a multicomputer, or cluster. A parallel application that was intended to run on a shared memory multiprocessor will run on a cluster. The cost/performance ratio tells you how much would it cost to buy a multiprocessor that would give the SAME performance as the cluster, compared to the cost of the cluster. Ratios of 10:1 were common in the early days of cluster computing, meaning the cluster ran an application as well as a commercial processor costing ten times as much. Today, the ratios have dropped to about 3:1, but a project with more time than money has options that it did not have twenty years ago.
Execution Times: Real-time applications may have an execution time limit. Calculating the location of a landing zone for an incoming missile, for example, should complete before it arrives.
Processor Speed: Some applications have to process a task with a complexity that varies dramatically over time. Predicting the weather, for example, has a complexity that depends upon current local conditions. In these cases, the system may have a speed requirement in gflops (billions of floating point operations) per second, for example, rather than a specific execution time limit.
Speedup: this standard measure of performance improvement calculates the ratio of the completion times on some reference system to the system under test. Completion time is often that of a collection of batch jobs or tasks, measured from the time the first starts until the last one completes.
Throughput: Many applications process a stream of tasks whose arrival rate is more critical than the complexity of each task; the time it takes to process one of them is not as important as how many can be processed over a finite period of time. Be sure to include more tasks in your test stream than you have nodes in your cluster to process them to ensure a backlog at some point in the processing time period.
Utilization/Loading: Parallel applications may distribute processes across a network of computer nodes. A balanced system is one in which the loading at each node, or processor utilization, is about equal. Fault tolerance aside, one would like to have the workload distributed evenly across each node in the distributed system's network.
Note: If you would like a related item added to any of the above lists, please send the details to the Technical Area Coordinator:
Contact
Dr. Alex Vrenios
DSRLab, LLC
Distributed Systems
Research Laboratory
Phoenix, Arizona, USA
Voice: +1 602 377-7720
Email: alex@DSRLab.com
This web page is updated regularly. Your questions, suggestions and comments
are welcome!
Last updated November 25, 2008