The Scalasca Trace Tools are a collection of highly scalable, trace-based tools for in-depth analysis of concurrent behavior in HPC applications. A distinctive feature is its scalable automatic trace-analysis component. It identifies potential performance bottlenecks – in particular those concerning communication and synchronization – and offers guidance in exploring their causes.

Core features

The trace-analysis component of the Scalasca Trace Tools currently focusses on performance bottlenecks in MPI, OpenMP, and hybrid MPI & OpenMP parallelization.¹ It provides the following core features:

• Detection of wait states
  By comparing event timestamps across locations, the analyzer determines time spans in communication and synchronization operations where processes or threads sit idle waiting for their peers. In addition, it also collects statistics about the detected wait-state instances.
• Identification of their root causes
  Traversing the trace data in backward direction allows the analyzer to pinpoint delays due to load or communication imbalances as the root causes of such wait states, even along far-reaching cause-effect chains.
• Discovery of activities on the critical path
  Based on the collected wait-state information, the analyzer can compute a profile of all activities on the critical path of the target application. This determines the length of the program execution, and therefore highlights those routines that constitute the best candidates for optimization.
• Proven scalability
  Due to its parallel design, the analyzer is able to handle even extreme-scale traces. It has been successfully applied to traces from applications executing with up to 1.8 million parallel threads.

In addition, the Scalasca Trace Tools provide convenience commands to simplify application measurement, in particular trace collection and analysis, as well as analysis report post-processing.

Analysis workflow

The Scalasca Trace Tools use OTF2 event traces as input. Thus, an execution trace of the target application needs to be collected first. For this task, the Scalasca Trace Tools rely on the Score-P instrumentation and measurement system.

Since event traces can quickly grow extremely large, it is strongly recommended to create a profile measurement first. This allows to assess the measurement overhead, estimate the expected trace size, and adjust the measurement configuration appropriately. The performance overview provided by the profile can also be used to narrow the focus of the analysis, for example, by restricting trace collection to a specific execution phase or a couple of representative solver iterations.

After successful trace collection, the automatic trace-analysis component then scans this trace data for potential performance bottlenecks and their root causes. The result is written as a performance profile augmented with additional metrics that can then be examined and processed further, for example, with the Cube Framework.

Recommended citations

If you find the Scalasca Trace Tools helpful for your research, please reference them in your publications. To cite the Scalasca Trace Tools in general, please use the following two publications:

• Zhukov, I. et al. (2015).
  Scalasca v2: Back to the Future.
  In: Niethammer, C., Gracia, J., Knüpfer, A., Resch, M., Nagel, W. (eds) Tools for High Performance Computing 2014, pp. 1–24. Springer, Cham.
  DOI: 10.1007/978-3-319-16012-2_1
• Geimer, M. et al. (2009).
  A scalable tool architecture for diagnosing wait states in massively parallel applications.
  Parallel Computing 35(7), pp. 275–388.
  DOI: 10.1016/j.parco.2009.02.003

When referring to specific topics, please use one (or more) of the following:

• Parallel wait-state search:
  Geimer, M. et al. (2009).
  A scalable tool architecture for diagnosing wait states in massively parallel applications.
  Parallel Computing 35(7), pp. 275–388.
  DOI: 10.1016/j.parco.2009.02.003
• Root-cause analysis:
  Böhme, D. et al. (2010).
  Identifying the Root Causes of Wait States in Large-Scale Parallel Applications.
  Proc. 39^th International Conference on Parallel Processing (ICPP), pp. 90–100.
  DOI: 10.1109/ICPP.2010.18
• Critical-path analysis:
  Böhme, D. et al. (2012).
  Scalable Critical-Path Based Performance Analysis.
  Proc. IEEE 26^th International Parallel and Distributed Processing Symposium (IPDPS), pp. 1330–1340. IEEE.
  DOI: 10.1109/IPDPS.2012.120
• Trace event timestamp correction:
  Becker, D. et al. (2013).
  Extending the scope of the controlled logical clock.
  Cluster Computing 16, pp. 171–189.
  DOI: 10.1007/s10586-011-0181-8