It was an easy mistake to fall into. The customer had collected data on an application taking less than a second to run on an Intel® Architecture host processor, and then tried the same workload on the attached coprocessor. The issue was one of scale but provides a practical example to talk about appropriate sampling methods.
On the host using a recent, Intel® Xeon® architecture CPU, the sample looked like this:
This seems to be a pretty uniform sampling, albeit a little short and a little thin, but trying the same thing on one of the attached coprocessors revealed a seriously deteriorated view:
Each of these was a General Exploration run, on different micro-architectures, which means that different events, appropriate for each micro-architecture, were collected with each run. In order to be able to collect many events in the course of a single run, VTune Amplifier utilizes event multiplexing, wherein the analyzer periodically changes which events are being collected—not all events are collected all the time, but hopefully the application does the same things long enough that those same activities can be sampled using all the events of interest and so generate a complete picture of the application and its performance.
But that’s exactly the case before us now: the application runs for such a short duration that there’s not enough application to run to collect all the events of interest. Because of the design decisions to maximize core efficiency in the current generation of Intel® Xeon Phi™ coprocessors, each coprocessor thread only has two PMRs (performance monitoring registers) and it takes 7 runs of an application for VTune Amplifier to collect all the events needed for its General Exploration Run (at least for the micro-architecture codename Knight’s Corner). When multiplexing events, the analyzer changes which events are collected every 30 milliseconds or so, which makes it over 200 ms of gap between subsequent collections of any particular pair. Looking at the VTune Amplifier timeline, or just using the “time” command on the coprocessor, we can determine the runtime of our anonymous little test application at around 1 second. Over the course of that one second run, each pair of events may only get collected through 4 sections out of maybe 34 (34 x 30 ms = 1 sec or so there’s only a limited number of slots).
Fortunately, there are a couple of things that can be done to improve data quality. We can either increase our sampling rate, or not multiplex event sampling at all. Both are possible using VTune Amplifier XE.
In the advanced section of the Project Properties Target tab you’ll find the pull-down and the check-box shown above. Sampling rate is adjustable using the Duration time estimate, whose default value is “between 1 and 15 minutes,” or in the command line tool, –target-duration-type=short. Modifying this setting in either tool adjusts a multiplier for the sampling rate, the sample after value or SAV used for regulating event sampling in Intel PMUs.
The check-box, Allow multiple runs, (or –allow-multiple-runs in the command line tool) is a way to turn off multiplexing between multiple sets of events, but requires the ability to rerun the application. This is the way we determined earlier that the coprocessor General Exploration run requires 7 sets of PMU settings to sample all the events over the entire duration of the application, by setting this check-box and counting the number of times the application announced itself in stdout (you can also see the repeated runs in the VTune Amplifier timeline). As long as the application executes the same way each time, there’s some chance that the correlation between events collected on separate runs will be close enough to reality to be useful.
Across these two axes we did a bunch of collection runs and can show how the quality of the data improves as we both vary between the two shortest sampling intervals (across the horizontal) and turn off the MUX (the vertical dimension, between the two tables):
Here is a summary of 4 runs each using the multiplexer to collect General Exploration events, on the left using the default “short” interval and on the right using “very short.” Immediately it’s apparent that the shorter sampling increases the chances of seeing data, but some of the statistics are probably unreliable: estimated latency impact requires four events to compute, at least two separate multiplexer combinations that may not even have been collected in adjacent blocks of execution. So, while reducing the sampling interval at least produced something other than zeroes, we may not want to put much trust in them yet.
Turning off the multiplexer increases data volume by executing multiple runs, collecting event samples spanning the entire run rather than brief intervals across it.
You can see that even choosing the “short” sampling period, more fields in the summary are non-zero than in the MUX-instrumented runs. But even here we can see the effects of sampling period: not so much with clock ticks and instructions retired—this are the high frequency events that don’t suffer as much from the longer sampling period and their values, plus the CPI rate that is derived from them are indistinguishable by eye. But that’s not true of the cache-miss events, for both L1 and L2_DATA. The first thing we notice in the slower sampling rate is what looks like undersampling: the event counts on the right are decidedly larger than those on the left. The improved sampling was enough to bring the latency impact estimates down by a factor of 4, though they still seem too high to be believable.
Of the four variants for collection just surveyed, the multi-run collection with the very short sampling interval looks to be producing the best data. If you need to analyze such a small workload, we recommend you avoid the defaults and select the tightest resolution you can to avoid as much as possible the chance of aliasing in your data collection.
But seriously, the Intel® Xeon Phi™ coprocessor is a pretty big machine and small workloads can end up as just a smear on the boundaries of the architecture. If your plans include tuning performance for your application on the coprocessor, make sure you start with a big enough—and parallel enough—workload to actually have enough to measure when the work is divided between 240 or more hardware threads.