How to verify that OpenSSL is using multibuffer with your web server

The article "Enabling Multibuffer for AES-CBC Encryption in Apache* and nginx*" describes the procedures for enabling OpenSSL's* multibuffer capability with ciphers that use AES in CBC mode for the block cipher. This article describes how to check if the multibuffer code path is being used so that systems administrators can verify that their web server is operating as expected. Note that this procedure assumes that the web server is running on a Linux*-based system.

What you'll need

1. The Linux "perf" package. It is available for all major Linux distributions and should be easy to install using the distro's package manager.
2. A utility that can make numerous HTTPS requests in succession, and which allows you to choose the SSL cipher for the session. This article specifically recommends the Apache Benchmark tool, ab, which is included in the Apache HTTP Server distribution.

Step 1: Run the web server in single-process debug mode

A very simple way to determine which code path OpenSSL is using (which functions are being called) is to use a trace utility to analyze the process. Servers which use a forking model to spawn multiple worker processes make tracing more difficult since it's not known which of the child processes is actively handling the workload during the test run. There are ways to work around this problem, but they involve gathering more data which lowers the overall signal-to-noise ratio. It's easiest to simply put the server into a single process mode and eliminate the ambiguity.

The Apache web server can be run in single process mode using the -X switch:

apachectl start -X

To put the nginx web server in a single process mode you'll need two directives in the configuration file:

master_process off;
worker_processes 1;

You can optionally specify "daemon off" as well if you don't want nginx to detach from the controlling terminal.

Step 2: Determine the process ID of the web server process

This can be done by using ps and grepping for the server process name, or by looking in the pid file created by the web server.

Step 3: Run perf to trace the process

Use the perf record utility to begin tracing the process using the PID obtained from step 2. You may need to run this as root.

# perf record -p pid

perf works by preiodically sampling the process and examining its call stack. The higher the sample rate the more accurate the results will be, but this occurs at the expense of application performance. The defaults are sufficient for the purposes of this procedure since the goal is only to determine whether or not the multibuffer functions are being called, not to attempt performance analysis.

Note that perf will run in the foreground. Use ^C to quit.

Step 4: Fetch a large file multiple times

With perf record still running, fetch a file from the web server that is large enough to trigger the multibuffer code. Multibuffer only engages when the file being transferred is 64 KB in size or larger. It is important to fetch the file multiple times to ensure that perf record can catch the encryption functions during its sample periods. Be sure to specify a cipher that uses AES in CBC mode for the block encryption, and that it is one that is accepted by your server.

This command runs the Apache Benchmark utility to fetch the target file 100 times in rapid succession, negotiating with the server to use the AES128-SHA cipher defined in OpenSSL. This corresponds to the cipher suite TLS_RSA_WITH_AES_128_CBC_SHA in the RFCs. (A mapping of OpenSSL cipher names to RFC cipher suites can be found at https://testssl.sh/openssl-rfc.mappping.html).

% ab -n 1000 -Z AES128-SHA https://servername/bigfile

Make sure the file requests are successful, and then stop the performance trace by hitting ^C.

Step 5: Generate the trace report

With the trace completed. use the perf report command to generate a report of the process trace:

# perf report

By default, this will send the output to your terminal using a pager such as less or more. If you had to run perf record as root, then you'll need to run perf report as root as well.

Multibuffer Functions

If you see these functions listed in the report, then OpenSSL is using multibuffer for the CBC encryption:

     3.92%    nginx  nginx               [.] sha1_multi_block_avx
     2.19%    nginx  nginx               [.] aesni_multi_cbc_encrypt

Though they only make up a small percentage of the CPU utilization, they should appear near the top of the report (within the first dozen lines).

Stitched Functions

If you see functions similar to these then OpenSSL is using the stitched code path for CBC encryption:

     2.03%    nginx  nginx               [.] sha256_block_data_order_avx2
     0.69%    nginx  nginx               [.] sha1_block_data_order_avx2             

Note that the exact function names will depend on your server's architecture (this example shows the AVX2 code path).

§