As a quality assurance provider, Enzyme tests a large array of games on a weekly basis. When it comes to compatibility and performance testing, we have stumbled upon many instances where apparent performance hitches were not evident in the data collected during frame rate benchmarks.
The frame rate, or frames per second (FPS), while being a viable and simple method of measurement, is not the be-all and end-all data to consider when assessing performance. Another important metric to consider regarding frames is frame time.
Frame time mostly refers to the time it takes for the software to render each frame. More precisely, the data takes into consideration the duration of the benchmark multiplied by the average FPS of the application or rather, the total amount of frames rendered over the length of the benchmark rather than its duration in seconds. This data can be collected using benchmarking tools, such as the suite of Intel® INDE Graphic Performance Analyzers, and is crucial to take into account when assessing performance.
To illustrate this, consider a one-second frame time. If 60 frames are rendered during the first 500ms, followed by an absence of frames in the last 500ms of that second, the frame rate measure would still show an average of 60 FPS since the total number of frames displayed within that second is indeed 60.
While this case is unlikely, it demonstrates a worst-case scenario: a visible in-game hitch that is ignored by the frame rate data.
The frame time measurement provides more precise data as it keeps track of the time interval between every displayed frame. In the same case described above, a peak of 500ms will be easily and clearly identified when running the benchmark, allowing your team to troubleshoot the probable performance loss causes.
Small hitches can be identified through the frame time and be completely ignored by the frame rate.
From a testing perspective, this type of data is often analyzed when comparing different hardware to identify the source of performances issues. We more often rely on the frame time than the frame rate as its linear nature makes it easier to analyze and is generally a more reliable source of information when it comes to a game's performances.
For instance, we have used frame time to compare performances with different storage devices. The following charts are representative of a situation where a clear difference is visible between an HDD and an SSD, yet can only be seen through frame time.
SSD's frame time is more stable than an HDD's. Frame rates are very similar for both.
Again, the performance issues are not represented in the FPS benchmark, whereas the frame time measurements clearly demonstrate an obvious increase in hitches on a system with an HDD compared to a system with an SSD. This type of comparison can prove to be useful for games that are heavy on texture streaming as we can determine if the type of storage device used causes a bottleneck.
In the end, FPS is a measurement that tends to even out the very short spikes. Frame time will catch everything; it might be a bit more intimidating to analyze since it contains a lot more data, but it is crucial to take it into account if you want to thoroughly troubleshoot and debug your software.
If you have questions on frame time measurement or would like to discuss other performance analysis best practices, don’t hesitate to contact us.
About the author
Enzyme was founded in 2002 by Yan Cyr and Emmanuel Viau, two pioneers in the video game industry. Using their international experience and the expertise of all the Enzymers, we have combined creativity and discipline in order to create Quality Assurance (QA) services and a testing methodology that add value to the clients’ products.
We are a passionate community dedicated to QA for video games, apps, software and websites. Whether you need QA testing, PC/Mobile compatibility testing, project evaluation or focus groups, or you are looking for localization or linguistic resources, or you need methodology or project management consultants, partnering with us will contribute to the achievement of your goals.
Our mission is to put our passionate workforce to use and contribute to the success of your projects.