In part 10 of the Intel® Software Guard Extensions (Intel® SGX) tutorial series we’ll examine two utilities in the Intel SGX Software Development Kit (SDK): the Intel SGX Debugger, and the Enclave Memory Measurement Tool (EMMT). First, we’ll learn how to use the Intel SGX Debugger to inspect enclaves in mixed-mode applications such as our Tutorial Password Manager. Then, we’ll use the EMMT to analyze the memory usage of our enclave in order to optimize its configuration. In the process of doing that, we’ll also find and fix a program bug that has been in the code since Part 3 of the series!
You can find a list of all the published tutorials in the article Introducing the Intel® Software Guard Extensions Tutorial Series. Source code is provided with this installment of the series.
The Intel® SGX Debugger
Intel SGX enclaves are intended to keep secrets from being exposed to unauthorized users and programs. Even code running in Ring 0 cannot bypass an enclave’s protections because the security perimeter is enforced by the CPU, regardless of the execution environment. These protections ensure that no untrusted code—whether that be another user, a superuser, the BIOS, or even a virtual machine manager—can inspect enclave memory. Attempts to map enclave memory outside the enclave will yield only the encrypted memory pages. The inability to inspect production enclaves is a key component of the Intel SGX security model.
Software developers, however, frequently use a debugger to analyze a program’s execution in order to find and fix runtime errors in their code, and a debugger can’t function if it doesn’t have access to an application’s memory space. This creates a conflict between security and usability for Intel SGX: The security of enclaves must be preserved, but software developers need the ability to run a debugger when building and testing applications.
Intel’s solution to this problem is twofold:
- Only the Intel SGX Debugger can debug enclaves.
- Only debug-mode enclaves can be debugged.
The Intel SGX Debugger is required to debug enclaves because Intel SGX defines a specific interface for debugging: It requires the execution of special Intel SGX supervisor instructions. Debuggers without Intel SGX support will not be able to inspect enclaves. The CPU enforces the requirement that only debug-mode enclaves can be inspected with a debugger. This allows developers to create debuggable enclaves in their development environment, and non-debuggable enclaves for distribution.
An enclave’s state as a debug-mode or release-mode enclave is set at build time, and this state is included in the enclave’s metadata. The enclave and its metadata are cryptographically signed using the developer’s signing key. This prevents malicious users from turning a release-mode enclave into a debug-mode enclave: Attempts to manipulate the enclave’s state will change its signature, and the CPU will not launch an enclave if the runtime signature does not match the one that was generated at build time. As an additional protection, debug-mode and release-mode enclaves derive different encryption keys for the Intel SGX sealing functions. A debug-mode enclave cannot access, and thus expose, secrets that were sealed by a release-mode enclave.
Running the Intel SGX Debugger on Mixed-Mode Applications
When building a native application on Windows*, using the Intel SGX Debugger is as easy as changing the debugger from the Local Windows Debugger to the Intel SGX Debugger, as shown in Figure 1, and setting the working directory to the same location as the enclave object files (in typical solutions, setting the working path to $(OutDir) will suffice, as shown in Figure 2).
Figure 1. Selecting the Intel® SGX Debugger in Microsoft Visual Studio*.
Figure 2. Setting the working directory for the Intel® SGX Debugger.
When running managed applications, however, you are not given a choice of which debugger to run, and the Intel SGX Debugger can’t debug managed code, anyway. That means you cannot use the Start Debugging feature in Microsoft Visual Studio* to directly debug enclaves that are part of a mixed-mode application. Instead, you need to use the Attach to Process command in the Debug menu. The procedure is as follows:
- In Visual Studio, go to the Debug menu and select Attach to Process…
- In the Attach to Process dialog box, where it says Attach to… click the Select… button. See Figure 3. This will bring up the Select Code Type dialog box.
![Attach to process]()
Figure 3. The Attach to Process window. - In the Select Code Type dialog box, click Debug these code types and choose Intel® SGX, then click OK.
![]()
Figure 4. Selecting the code type to debug. - Execute the application from the output directory via the Windows shell or Windows File Explorer. (Donot run it from within Visual Studio, as this will launch it under the Windows debugger.)
- Return to Visual Studio. In the Attach to Process dialog, hit Refresh to update the process list, then select the application, and click Attach.
![Selecting the process to debug]()
Figure 5. Selecting the process to debug.
Note that the Intel SGX Debugger can only inspect native code, whether it’s inside or outside the enclave. You cannot use it to debug managed code.
The Enclave Memory Measurement Tool
All enclaves are instantiated in a region of memory known as the enclave page cache, or EPC. The EPC is a shared resource, which means that all running enclaves on the system must fit inside of it. On Windows, the size of the EPC is fixed in the BIOS and cannot be changed (on Linux* systems, the EPC supports paging, which allows the operating system to expand it as needed, but this has performance implications and should be avoided). Because of this restriction, it is extremely important that enclaves be sized to their actual memory usage, and not be allocated more memory than they will use.
The amount of memory allocated to an enclave is set in the enclave’s configuration file; it defaults to 256 KB of stack space per thread and 1 MB of global heap space. The developer can change these values either by editing the .xml file directly, or via the user interface in Visual Studio. Though these values are entered in bytes, they must be 4 KB aligned, since EPC pages are made up of 4 KB chunks.
Figure 6. The enclave settings for the Tutorial Password Manager.
The purpose of the EMMT is to give Windows developers the ability to measure the real memory use of their enclaves so that they can adjust the allocated values accordingly.
Running the EMMT
The EMMT application is part of the Intel SGX SDK. Execution is straightforward:
sgx_emmt [ –-enclave=enclaves ] program [ arguments ... ]
The --enclaves parameter is optional: If your application has multiple enclaves, you can list one or more of them that you want to target; otherwise, the EMMT measures every enclave in your application.
After the application exits, the EMMT prints a memory usage summary for each measured enclave. Figure 7 shows results for an enclave that used 2 KB of stack space and 4 KB from the heap during execution.
Enclave: "Enclave.signed.dll"
[Peak stack use]: 0x2KB
[Peak heap use]: 0x4KB
Figure 7. Output from the Enclave Memory Measurement Tool.
There are two important caveats that must be kept in mind when using the EMMT:
- The results show the actual memory used during that specific execution of the application and its enclave. These values may not be typical for the enclave, or even a worst-case usage. It is important that the test run where the measurements are made be as representative of real-world usage as possible, and cover all possible functions. It may be necessary to write a dedicated front-end whose sole job is to stress-test the enclave by exercising its API.
- The EMMT does not work on mixed-mode applications. If the main application is written in .NET*, you will not be able to use the EMMT directly on the final application. You will need to write a native front-end application to stress-test the enclave per the above.
Measuring the Tutorial Password Manager
The Tutorial Password Manager is a mixed-mode application: The front-end is written in C#, and it connects to the enclave via C++/CLI. Since the EMMT can’t be run on mixed-mode applications, we need to write a native application to call the enclave interface functions in order to measure the memory usage.
To do this, we’ll embed our reference password vault in the application directly as byte arrays (the disk input/output routines in the Tutorial Password Manager are in the C++ .NET layer, so they aren’t available to us to use) and call the wrapper functions in the EnclaveBridge DLL. For demonstration purposes, we’ll perform the following tasks:
- Unlock the vault
- Get account info for each account
- Get the password for each account
- Add an account
- Change all the account passwords, randomly generating a new password each time
- Change the master password
The main program is shown in Figure 8.
int main()
{
int rv;
uint32_t count;
rv = ew_initialize();
if (rv != 0) {
fprintf(stderr, "ew_initialize: 0x%08x\n", rv);
Exit(1);
}
rv = ew_initialize_from_header(header, header_size);
if (rv != 0) {
fprintf(stderr, "ew_initialize_from_header: 0x%08x\n", rv);
Exit(1);
}
rv = ew_unlock(mpass);
if (rv != 0) {
fprintf(stderr, "ew_unlock: 0x%08x\n", rv);
Exit(1);
}
rv = ew_load_vault(vault_data);
if (rv != 0) {
fprintf(stderr, "ew_load_vault: 0x%08x\n", rv);
Exit(1);
}
rv = ew_accounts_get_count(&count);
if (rv != 0) {
fprintf(stderr, "ew_accounts_get_count: 0x%08x\n", rv);
Exit(1);
}
for (uint32_t i = 0; i < count; ++i) {
uint16_t sname, slogin, surl, spass;
char *name, *login, *url, *pass;
printf("Account %u:\n", i);
rv = ew_accounts_get_info_sizes(i, &sname, &slogin, &surl);
if (rv != 0) {
fprintf(stderr, "ew_accounts_get_info_sizes[%u]: 0x%08x\n", i, rv);
continue;
}
name = new char[sname+1];
login = new char[slogin+1];
url = new char[surl+1];
rv = ew_accounts_get_info(i, name, sname, login, slogin, url, surl);
if (rv != 0) {
fprintf(stderr, "ew_accounts_get_info[%u]: 0x%08x\n", i, rv);
continue;
}
name[sname] = 0;
login[slogin] = 0;
url[surl] = 0;
printf("\tname= %s\n", name);
printf("\tlogin= %s\n", login);
printf("\turl= %s\n", url);
rv = ew_accounts_get_password_size(i, &spass);
if (rv != 0) {
fprintf(stderr, "ew_accounts_get_password_size[%u]: 0x%08x\n", i, rv);
continue;
}
pass = new char[spass + 1];
rv = ew_accounts_get_password(i, pass, spass);
if (rv != 0) {
fprintf(stderr, "ew_accounts_get_password[%u]: 0x%08x\n", i, rv);
continue;
}
pass[spass] = 0;
printf("\tpass= %s\n", pass);
delete[] name;
delete[] login;
delete[] url;
delete[] pass;
}
// Start changing things
// Add an account
rv = ew_accounts_set_info(3, "Acme", 4, "wileye", 6, "http://acme.nodomain/", 21);
if (rv != 0) {
fprintf(stderr, "ew_accounts_set_info[3]: 0x%08x\n", rv);
Exit(1);
}
// Change the passwords many times
for (int i = 0; i < 4; ++i) {
uint32_t idx = i % 4;
char newpass[32];
int rv;
rv = ew_accounts_generate_password(31, 0xffff, newpass);
if (rv != 0) {
fprintf(stderr, "ew_accounts_generate_password[%d]: 0x%08x\n", i, rv);
Exit(1);
}
rv = ew_accounts_set_password(idx, newpass, 31);
if (rv != 0) {
fprintf(stderr, "ew_accounts_set_password[%d]: 0x%08x\n", i, rv);
Exit(1);
}
}
// Change the master password
rv = ew_change_master_password(mpass, new_mpass);
if (rv != 0) {
fprintf(stderr, "ew_change_master_password: 0x%08x\n", rv);
Exit(1);
}
Exit(0);
}
Figure 8. Program listing for the native test application.
Initially, we’ll change each account password once, just to get some output and see where our memory usage stands.
After building and executing the test application under the EMMT, we get the following:
> sgx_emmt "CLI Native Test App.exe"
The command line is: "cli native Test App.exe ".
Enclave: "Enclave.signed.dll"
[Peak stack use]: 0x2KB
[Peak heap use]: 0x4KB
Figure 9. EMMT output for the Tutorial Password Manager's enclave.
Because the size of the password vault is fixed, we don’t expect the Tutorial Password Manager to use a lot of enclave memory, and according to the EMMT it doesn’t. However, changing passwords just once isn’t much of a stress test, and the encryption and decryption functions may need more memory than is shown in this simple test. So, we’ll modify the test program to change each password 1000 times and see how that affects our results.
After recompiling and executing, we get the output shown in Figure 10:
The command line is: "cli native Test App.exe ".
Enclave: "Enclave.signed.dll"
[Peak stack use]: 0x2KB
[Peak heap use]: 0xc0KB
Figure 10. EMMT output after increasing the number of password changes.
This result, however, is unexpected. Originally, we used only 4 KB of stack space, but it has jumped to 192 KB. If there was a small increase that would possibly make sense, but a 48x increase does not! Increasing the number of password changes to 5000 causes yet another increase in the heap usage:
[Peak heap use]: 0x3acKB
Figure 11. EMMT output after further increasing the number of password changes.
This suggests we have a memory leak in the enclave.
The loop responsible for changing the password is invoking these two ECALLs:
ve_accounts_generate_password()ve_accounts_set_password()
A quick review of E_Vault.cpp turns up the culprit. In E_Vault::accounts_set_password(), we are calling new without a corresponding delete. Thus, each call to change a password leaks a few bytes of memory. Since this code was derived from the non-Intel SGX code path, a review of Vault.cpp reveals the exact same problem.
After fixing the memory leak, we run our test again and obtain the results shown in Figure 12:
The command line is: "cli native Test App.exe ".
Enclave: "Enclave.signed.dll"
[Peak stack use]: 0x2KB
[Peak heap use]: 0x4KB
Figure 12. EMMT output after fixing the memory leak.
This takes us back to our original heap usage of 4 KB.
We aren’t done, however. This is the base usage for a handful of entries, all with reasonable string lengths, but our vault can actually store much longer strings—up to 64 KB each, for the account name, URL, login name, and password. That means each account could require 256 KB, and with eight accounts total that means the vault could be up to 2 MB in memory.
How much memory we want to allocate to the enclave is now a judgement call. Realistically, users are probably not going to enter 65,000+ characters into any of the fields, so 2 MB of heap space for the highly unlikely, worst-case scenario seems excessive. If each field only stored 80 bytes, which is still a lot, the total storage need would be under 3 KB. To be safe, we’ll allocate an additional 4 KB to the heap above what the EMMT measures, which brings it to 8 KB. The stack is given the minimum of 4 KB. The final enclave settings are shown in Figure 13:
Figure 13. Final enclave settings for the Tutorial Password Manager.
Our enclave is now using only a fraction of the EPC that was allocated to it by the default configuration.
If we were building a commercially viable password manager, one that could have an unlimited number of accounts, we’d need a different design strategy for the application since the fixed heap size means an enclave’s memory footprint cannot grow without bound. Our enclave would not be able to store the entire password database unencrypted in memory, and would instead have to implement some form of just-in-time processing, where accounts were only decrypted as they were needed.
Summary
The Intel SGX SDK includes tools to aid in the development and debugging of Intel SGX enclaves, and in particular to address the unique challenges presented by the security model. Production applications must not be inspectable, but debuggers are a critical tool in the software development lifecycle, and Intel provides a solution in the Intel SGX Debugger. Special instructions in the Intel SGX instruction set allow developers to use the Intel SGX Debugger on debug-mode enclaves while denying the same to production applications. Additionally, developers are encouraged to minimize the memory footprint of their enclaves because the EPC is a shared resource. The EMMT gives developers the information they need to appropriately size their stack and heap allocations. The Intel SGX SDK provides a complete development ecosystem with libraries, APIs, and tools for aiding development, debugging, and management of enclaves.
Sample Code
The code sample for this part of the series builds against the Intel SGX SDK version 1.8 using Microsoft Visual Studio 2015.
Coming Up Next
In Part 11 of the series, we’ll prepare our Tutorial Password Manager for deployment by creating an installation package. Stay tuned!