IEEE Technical Committee on Scalable Computing - Technical Area -
Performance Evaluation: Benchmarking
(Contact Information)
Introduction
System performance may be viewed quite differently by a computational physicist and an Internet Service Provider. And while one ISP views performance in terms of transactions per second, another may focus on system availability. Each user must decide which of the many performance metrics are most important to their operation and choose a benchmarking suite that measures them. Benchmark results can uncover bottlenecks in your system and help quantify its performance. Once a baseline performance measurement is established, subsequent benchmarking can be used to validate a system tuning effort.
This Technical Area page includes links to a variety of benchmarking tools. Some of them give very detailed results about a specific computing subsystem and some are designed to present an overall picture of your system's performance. While we can neither endorse nor recommend any of them, each of them may contain tutorial information that can help you better understand what a benchmark can do for you, and what it cannot.
Conferences and Workshops
GlobeNet 2009 - IARA GlobeNet Conference, March 1-6, 2009, Gosier, Guadeloupe, French Caribbean.
DB 2009 - 1st International Conference on Advances in Databases, March 1-6, 2009, Gosier, Guadeloupe, French Caribbean.
ICN 2009 - 8th International Conference on Networks, March 1-6, 2009, Gosier, Guadeloupe, French Caribbean.
ICONS 2009 - 4th International Conference on Systems, March 1-6, 2009, Gosier, Guadeloupe, French Caribbean.
PDP2009 - 17th Euromicro International Conference on Parallel, Distributed, and network-based Processing, February 18-20, 2009, Weimar, Germany.
DigitalWorld 2009 - Digital World 2009, February 1-6, 2009, Cancun, Mexico.
ACHI 2009 - 2nd International Conferences on Advances in Computer-Human Interactions, February 1-6, 2009, Cancun, Mexico.
eKNOW 2009 - International Conference on Information, Process, and Knowledge Management, February 1-6, 2009, Cancun, Mexico.
eL&mL 2009 - International Conference on Mobile, Hybrid, and On-line Learning, February 1-6, 2009, Cancun, Mexico.
eTELEMED 2009 - International Conference on eHealth, Telemedicine, and Social Medicine, February 1-6, 2009, Cancun, Mexico.
GEOWS 2009 - International Conference on Advanced Geographic Information Systems & Web Services, February 1-6, 2009, Cancun, Mexico.
ICDS 2009 - 3rd International Conference on Digital Society, February 1-6, 2009, Cancun, Mexico.
ICQNM 2009 - 3rd International Conference on Quantum, Nano and Micro Technologies, February 1-6, 2009, Cancun, Mexico.
MMCN 2009 - SPIE/ACM Multimedia Computing and Networking 2009, January 18-22, 2009, San Jose, USA.
ICCIT'08 - 11th IEEE International Conference on Computer and Information Technology, December 25-27, 2008, Khulna, Bangladesh.
IDCS'08 - International Workshop on Internet and Distributed Computing Systems, December 25-27, 2008, Khulna, Bangladesh.
EUC 2008 - IEEE/IFIP International Conference on Embedded and Ubiquitous Computing, December 17-20, 2008, Shanghai, China.
HiPC 2008 - 15th IEEE International Conference on High Performance Computing, December 17-20, 2008, Bangalore, India.
ADCOM 2008 - 16th International Conference on Advanced Computing and Communication, December 14-17, 2008, Chennai, India.
ICDCIT 2008 - 5th International Conference on Distributed Computing and Internet Technologies, December 10-13, 2008, New Delhi, India.
IWSOS 08 - 3rd International Workshop on Self-Organizing Systems, Self-Organizing Networks and Networked Systems, December 10-12, 2008, Vienna, Austria.
ISPA-08 - IEEE International Symposium on Parallel and Distributed Processing with Applications, December 10-12, 2008, Sydney, Australia.
QuaSM-08 - Workshop on Quality Management of Services and Middleware, December 10-12, 2008, Sydney, Australia.
CSS 2008 - IEEE International Workshop on Cyberspace Safety and Security, December 12, 2008, Sydney, Australia.
ICPADS'08 - 14th IEEE International Conference on Parallel and Distributed Systems, December 8-10, 2008, Melbourne, Australia.
CMG'08 - International Computer Measurement Group conference, December 7-12, 2008, Las Vegas, Nevada, USA.
eScience 2008 - 4th IEEE International Conference on e-Science, December 7-12, 2008, indianapolis, Indiana, USA.
Middleware 2008 - ACM/IFIP/USENIX 9th International Middleware Conference, December 1-5, 2008, Leuven, Belgium.
MGC2008 - 6th International Workshop on Middleware for Grid Computing, December 1-5, 2008, Leuven, Belgium.
PDCAT'08 - 9th International Conference on Parallel and Distributed Computing, Applications and Technologies, December 1-4, 2008, Dunedin, New Zealand.
MoMM2008 - 6th international conference on Advances in Mobile Computing And Multimedia, November 24-26, 2008, Linz, Austria.
iiWAS2008 - 10th International Conference on Information Integration and Web-based Applications & Services, November 24-26, 2008, Linz, Austria.
ICYCS 2008 - 9th International Conference for Young Computer Scientists, November 18-21, 2008, Zhang Jia Jie, Hunan, China.
TrustCom 2008 - 1st International Symposium on Trusted Computing, November 18-20, 2008, Zhang Jia Jie, Hunan, China.
GADA'08 - Grid computing, high-performAnce and Distributed Applications, November 13-14, 2008, Monterrey, Mexico.
EuroSSC 2008 - 3rd European Conference on Smart Sensing and Context, October 29-31, 2008, Zurich, Switzerland.
MSWiM 2008 - 11th ACM International Symposium on Modeling, Analysis and Simulation of Wireless and Mobile Systems, October 27-31, 2008, Vancuver, British Columbia, Canada.
PE-WASUN 2008 - 5th ACM International Workshop on Performance Evaluation of Wireless Ad Hoc, Sensor, and Ubiquitous Networks, October 27-31, 2008, Vancuver, British Columbia, Canada.
I-CENTRIC 2008 - International Conference on Advances in Human-oriented and Personalized Mechanisms, Technologies, and Services, October 26-31, 2008, Sliema, Malta.
IMCSIT 2008 - International Multiconference on Computer Science and Information Technology, October 20-22, 2008, Wisla, Poland.
AAIA'08 - 3rd International Symposium on Advances in Artificial Intelligence and Applications, October 20-22, 2008, Wisla, Poland.
ABC'08 - Workshop on Agent Based Computing V, October 20-22, 2008, Wisla, Poland.
CANA'08 - Computer Aspects of Numerical Algorithms, October 20-22, 2008, Wisla, Poland.
LaSCoG'08 - 4th Workshop on Large Scale Computations on Grids, October 20-22, 2008, Wisla, Poland.
MMAP'08 - 1st International Symposium on Multimedia – Applications and Processing, October 20-22, 2008, Wisla, Poland.
RTS'08 - International Workshop on Real Time Software, October 20-22, 2008, Wisla, Poland.
SIS'08 - 3rd International Workshop on Secure Information Systems, October 20-22, 2008, Wisla, Poland.
WCO'08 - Workshop on Computational Optimization, October 20-22, 2008, Wisla, Poland.
WAHOC'08 - Workshop on Wireless and Unstructured Networking, October 20-22, 2008, Wisla, Poland.
EMSOFT 2008 - International Conference on Embedded Software, October 19-24, 2008, Atlanta, Georgia, USA.
IWSSPS 2008 - 3rd International Workshop on Software Support for Portable Storage, October 23, 2008, Atlanta, Georgia, USA.
NPC 2008 - IFIP International Conference on Network and Parallel Computing, October 18-20, 2008, Shanghai, China.
NSS 2008 - IFIP International Workshop on Network and System Security, October 18-19, 2008, Shanghai, China.
CompArch 2008 - Federated Events on Component-Based Software Engineering and Software Architecture, October 14-17, 2008, Karlsruhe, Germany.
CBSE 2008 - Conference on Component-Based Software Engineering, October 14-17, 2008, Karlsruhe, Germany.
CBHPC 2008 - Component-Based High Performance Computing Workshop, October 14-17, 2008, Karlsruhe, Germany.
QoSA 2008 - 4th International Conference on the Quality of Software-Architectures, October 14-17, 2008, Karlsruhe, Germany.
WiMOB 2008 - 4th IEEE International Conference on Wireless and Mobile Computing, Networking and Communications, October 12-14, 2008, Avignon, France.
BWIA 2008 - 2nd International Workshop on Broadband Wireless Internet Access, October 12, 2008, Avignon, France.
Cluster 2008 - IEEE Cluster 2008, September 29 - October 1, 2008, Tsukuba, Japan.
HiperIO'08 2008 - 2nd International Workshop on High Performance I/O Systems and Data Intensive Computing, October 1, 2008, Tsukuba, Japan.
iWAPT 2008 - 3rd international Workshop on Automatic Performance Tuning, October 1, 2008, Tsukuba, Japan.
ADVCOMP 2008 - 2nd International Conference on Advanced Engineering Computing and Applications in Sciences, September 29 - October 4, 2008, Valencia, Spain.
UBICOMM 2008 - 2nd International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies, September 29 - October 4, 2008, Valencia, Spain.
ENICS 2008 - International Conference on Advances in Electronics and Micro-electronics, September 29 - October 4, 2008, Valencia, Spain.
MASS 2008 - 5th IEEE International Conference on Mobile Ad-hoc and Sensor Systems, September 29 - October 2, 2008, Atlanta, Georgia, USA.
Grid 2008 - 9th IEEE/ACM International Conference on Grid Computing, September 29 - October 1, 2008, Tsukuba, Japan.
SYNASC-2008 - 10th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing, September 26-29, 2008, Timisoara, Romania.
HPCC-08 - 10th IEEE International Conference on High Performance Computing and Communications, September 25-27, 2008, DaLian, China.
Visions 08 - The British Computer Society International Academic Research Conference on Visions of Computer Science, September 22-24, 2008, London, England.
IDC'2008 - 2nd International Symposium on Intelligent Distributed Computing, September 18-20, 2008, Catania, Italy.
ICPP-08 - 37th International Conference on Parallel Processing, September 8-12, 2008, Portland, Oregon, USA.
MASCOTS 2008 - 16th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems, September 8-10, 2008, Baltimore, Maryland, USA.
WISE 2008 - 9th International Conference on Web Information Systems Engineering, September 1-4, 2008, Aukland, New Zealand.
E-BAG 2008 - 1st International Workshop on Web Information Systems Engineering for Electronic Businesses and Governments, September 1-4, 2008, Aukland, New Zealand.
IWWUA 2008 - 2nd International Workshop on Web Usability and Accessibility, September 1-4, 2008, Aukland, New Zealand.
MEM&LCW 2008 - 1st Workshop on Mashups, Enterprise Mashups and Lightweight Composition on the Web, September 1-4, 2008, Aukland, New Zealand.
WISR 2008 - 1st International Workshop on Web Information Systems Reengineering, September 1-4, 2008, Aukland, New Zealand.
SBLP 2008 - 12th Brazilian Symposium on Programming Languages, August 27-29, 2008, Fortaleza, Ceara, Brazil.
ChinaGrid 2008 - 3rd ChinaGrid Annual Conference, August 20-22, 2008, Dunhuang, Gansu, China.
CS 2008 - IASTED International Conference on Circuits and Systems, August 18-20, 2008, Kailua-Kona, Hawaii, USA.
DTVCS 2008 - Design, Testing and Formal Verification Techniques for Integrated Circuits and Systems, August 18-20, 2008, Kailua-Kona, Hawaii, USA.
WHNC 2008 - IEEE Wireless Hive Networks Conference, August 7-8, 2008, Austin, Texas, USA.
ICCCN 2008 - 17th IEEE International Conference on Computer Communications and Networks, August 3-7, 2008, Virgin Islands, USA.
NGASHP 2008 - 1st IEEE International Workshop on Next Generation Autonomous Storage and High Performance Computing, August 3-7, 2008, Virgin Islands, USA.
IC3 2008 - 1st International Conference on Contemporary Computing, August 7-9, 2008, Noida (outskirts of New Delhi), India.
ICESS 2008 - Fifth International Conference on Embedded Software and Systems, July 29-31, 2008, Chengdu, Sichuan, China.
DATICS 08 - Design, Analysis and Tools for Integrated Circuits and Systems, July 22-24, 2008, Crete Island, Greece.
CSE 08 - 11th IEEE International Conference on Computational Science and Engineering, July 16-18, 2008, Sao Paulo, Brazil.
IMPAR-08 - Impact of New Architectures on Parallel Programming, July 16-18, 2008, Sao Paulo, Brazil.
Web2Touch-08 - Workshop on living experience through the Web, July 16-18, 2008, Sao Paulo, Brazil.
NMEP 08 - Workshop on Numerical Mathematics for Engineering Problems, July 16-18, 2008, Sao Paulo, Brazil.
PerGRID-08 - International Symposium on Pervasive Grid, July 16-18, 2008, Sao Paulo, Brazil.
WSUPT-08 - Workshop on Simulation of Urban Planning and Transportation Systems, July 16-18, 2008, Sao Paulo, Brazil.
WSCEE-08 - Workshop on Scientific Computing in Electronics Engineering, July 16-18, 2008, Sao Paulo, Brazil.
WCEMP-08 - Workshop on Advanced Computing for Critical Systems and EMergency Preparedness and Response, July 16-18, 2008, Sao Paulo, Brazil.
PDPTA'08 - International Conference on Parallel and Distributed Processing Techniques and Applications, July 14-17, 2008, Las Vegas, Nevada, USA.
SDMAS'08 - 2nd International Workshop on Scalable Data Management Applications and Systems, July 14-17, 2008, Las Vegas, Nevada, USA.
ISPDC 2008 - 7th International Symposium on Parallel and Distributed Computing, July 1-5, 2008, Krakow, Poland.
ICDT 2008 - 3rd International Conference on Digital Telecommunications, June 29 - July 5, 2008, Bucharest, Romania.
WoWMoM 2008 - 9th IEEE International Symposium on A World of Wireless, Mobile, and Multimedia Networks, June 23-27, 2008, Irvine, California, USA.
ICIMP 2008 - 3rd International Conference on Internet Monitoring and Protection, June 29 - July 5, 2008, Bucharest, Romania.
CTRQ 2008 - International Conference on Communication Theory, Reliability, and Quality of Service, June 29 - July 5, 2008, Bucharest, Romania.
ICDT 2008 - 3rd International Conference on Digital Telecommunications, June 29 - July 5, 2008, Bucharest, Romania.
DSN'08 - 38th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, June 24-27, 2008, Anchorage, Alaska, USA.
CATARS'08 - 1st Workshop on Compiler and Architectural Techniques for Application Reliability and Security, June 26, 2008, Anchorage, Alaska, USA.
Benchmarking Links
HPL: the HPL benchmark is a software package that solves a (random) dense linear system in double precision (64 bits) arithmetic on distributed-memory computers. It can thus be regarded as a portable as well as freely available implementation of the High Performance Computing Linpack Benchmark.
LINPAK: the LINPACK benchmark is a collection of Fortran subroutines that analyze and solve linear equations and linear least-squares problems. The package solves linear systems whose matrices are general, banded, symmetric indefinite, symmetric positive definite, triangular, and tridiagonal square. In addition, the package computes the QR and singular value decompositions of rectangular matrices and applies them to least-squares problems. LINPACK uses column-oriented algorithms to increase efficiency by preserving locality of reference.
LMBENCH: the LMBENCH benchmark is a suite of simple, portable, ANSI/C microbenchmarks for UNIX/POSIX. In general, it measures two key features: latency and bandwidth. LMBENCH is intended to give system developers insight into basic costs of key operations.
NAS: the NAS Parallel Benchmarks (NPB) are a small set of programs designed to help evaluate the performance of parallel supercomputers. The benchmarks, which are derived from computational fluid dynamics (CFD) applications, consist of five kernels and three pseudo-applications. The NPB come in several flavors. NAS solicits performance results for each from all sources.
SPEC: the Standard Performance Evaluation Corporation is a non-profit corporation formed to establish, maintain and endorse a standardized set of relevant benchmarks that can be applied to the newest generation of high-performance computers. SPEC develops benchmark suites and also reviews and publishes submitted results from SPEC member organizations and other benchmark licensees.
SPLASH: Stanford's SPLASH benchmark evaluates performance of shared memory multiprocessors.
STREAM: the STREAM benchmark is the de facto industry standard benchmark for the measurement of computer memory bandwidth. The STREAM benchmark measures "real world" bandwidth sustainable from ordinary user programs -- not the theoretical "peak bandwidth" provided by most vendors.
TPC: the Transaction Processing Council defines transaction processing and database benchmarks and delivers trusted results to the industry.
Performance Metrics
Cost-Effectiveness: An application may have an execution cost requirement that is just as important as execution time. As the number of nodes in use by the application increases, the execution time goes down (good), but the cost goes up (bad) due to the increased inter-process communication overhead. It may be important to determine an optimal number of nodes that minimizes both the execution time and the operating cost.
Cost/Performance Ratio: One modern approach to low-cost computing is to interconnect a whole lot of PCs over a very fast external data network, and either customize the operating system or run specialized middleware that turns them into a multicomputer, or cluster. A parallel application that was intended to run on a shared memory multiprocessor will run on a cluster. The cost/performance ratio tells you how much would it cost to buy a multiprocessor that would give the SAME performance as the cluster, compared to the cost of the cluster. Ratios of 10:1 were common in the early days of cluster computing, meaning the cluster ran an application as well as a commercial processor costing ten times as much. Today, the ratios have dropped to about 3:1, but a project with more time than money has options that it did not have twenty years ago.
Execution Times: Real-time applications may have an execution time limit. Calculating the location of a landing zone for an incoming missile, for example, should complete before it arrives.
Processor Speed: Some applications have to process a task with a complexity that varies dramatically over time. Predicting the weather, for example, has a complexity that depends upon current local conditions. In these cases, the system may have a speed requirement in gflops (billions of floating point operations) per second, for example, rather than a specific execution time limit.
Speedup: this standard measure of performance improvement calculates the ratio of the completion times on some reference system to the system under test. Completion time is often that of a collection of batch jobs or tasks, measured from the time the first starts until the last one completes.
Throughput: Many applications process a stream of tasks whose arrival rate is more critical than the complexity of each task; the time it takes to process one of them is not as important as how many can be processed over a finite period of time. Be sure to include more tasks in your test stream than you have nodes in your cluster to process them to ensure a backlog at some point in the processing time period.
Utilization/Loading: Parallel applications may distribute processes across a network of computer nodes. A balanced system is one in which the loading at each node, or processor utilization, is about equal. Fault tolerance aside, one would like to have the workload distributed evenly across each node in the distributed system's network.
Note: If you would like a related item added to any of the above lists, please send the details to the Technical Area Coordinator:
Contact
Dr. Alex Vrenios
DSRLab, LLC
Distributed Systems
Research Laboratory
Phoenix, Arizona, USA
Voice: +1 602 377-7720
Email: alex@DSRLab.com
This web page is updated regularly. Your questions, suggestions and comments
are welcome!
Last updated June 25, 2008