Intel® Software Guard Extensions Tutorial Series: Part 4, Enclave Design

In Part 4 of the Intel® Software Guard Extensions (Intel® SGX) tutorial series we’ll be designing our enclave and its interface. We’ll take a look at the enclave boundary that was defined in Part 3 and identify the necessary bridge functions, examine the impact the bridge functions have on the object model, and create the project infrastructure necessary to integrate the enclave into our application. We’ll only be stubbing the enclave ECALLS at this point; full enclave integration will come in Part 5 of the series.

You can find the list of all of the published tutorials in the article Introducing the Intel® Software Guard Extensions Tutorial Series.

There is source code provided with this installment of the series: the enclave stub and interface functions are provided for you to download.

Application Architecture

Before we jump into designing the enclave interface, we need to take a moment and consider the overall application architecture. As discussed in Part 1, enclaves are implemented as dynamically loaded libraries (DLLs under Windows* and shared libraries under Linux*) and they can only link against 100-percent native C code.

The Tutorial Password Manager, however, will have a GUI written in C#. It uses a mixed-mode assembly written in C++/CLI to get us from managed to unmanaged code, but while that assembly contains native code it is not a 100-percent native module and it cannot interface directly with an Intel SGX enclave. Attempts to incorporate the untrusted enclave bridge functions in C++/CLI assemblies will result in a fatal error:

	Command line error D8045: cannot compile C file 'Enclave_u.c' with the /clr option

That means we need to place the untrusted bridge functions in a separate DLL that is all native code. As a result, our application will need to have, at minimum, three DLLs: the C++/CLI core, the enclave bridge, and the enclave itself. This structure is shown in Figure 1.

Figure 1. Component makeup for a mixed-mode application with enclaves.

Further Refinements

Since the enclave bridge functions must reside in a separate DLL, we’ll go a step further and place all the functions that deal directly with the enclave in that same DLL. This compartmentalization of the application layers will not only make it easier to manage (and debug) the program, but also to ease integration by lessening the impact to the other modules. When a class or module has a specific task with a clearly defined boundary, changes to other modules are less likely to impact it.

In this case, the PasswordManagerCoreNative class should not be burdened with the additional task of instantiating enclaves. It just needs to know whether or not Intel SGX is supported on the platform so that it can execute the appropriate function.

As an example, the following code block shows the unlock() method:

int PasswordManagerCoreNative::vault_unlock(const LPWSTR wpassphrase)
{
	int rv;
	UINT16 size;

	char *mbpassphrase = tombs(wpassphrase, -1, &size);
	if (mbpassphrase == NULL) return NL_STATUS_ALLOC;

	rv= vault.unlock(mbpassphrase);

	SecureZeroMemory(mbpassphrase, size);
	delete[] mbpassphrase;

	return rv;
}

This is a pretty simple method that takes the user’s passphrase as a wchar_t, converts it to a variable-length encoding (UTF-8), and then calls the unlock() method in the vault object. Rather than clutter up this class, and this method, with enclave-handling functions and logic, it would be best to add enclave support to this method through a one-line addition:

int PasswordManagerCoreNative::vault_unlock(const LPWSTR wpassphrase)
{
	int rv;
	UINT16 size;

	char *mbpassphrase = tombs(wpassphrase, -1, &size);
	if (mbpassphrase == NULL) return NL_STATUS_ALLOC;

	// Call the enclave bridge function if we support Intel SGX
	if (supports_sgx()) rv = ew_unlock(mbpassphrase);
	else rv= vault.unlock(mbpassphrase);

	SecureZeroMemory(mbpassphrase, size);
	delete[] mbpassphrase;

	return rv;
}

Our goal will be to put as little enclave awareness into this class as is feasible. The only other additions the PasswordManagerCoreNative class needs is a flag for Intel SGX support and methods to both set and get it.

class PASSWORDMANAGERCORE_API PasswordManagerCoreNative
{
	int _supports_sgx;

	// Other class members ommitted for clarity

protected:
	void set_sgx_support(void) { _supports_sgx = 1; }
	int supports_sgx(void) { return _supports_sgx; }

Designing the Enclave

Now that we have an overall application plan in place, it’s time to start designing the enclave and its interface. To do that, we return to the class diagram for the application core in Figure 2, which was first introduced in Part 3. The objects that will reside in the enclave are shaded in green while the untrusted components are shaded in blue.

Figure 2. Class diagram for the Tutorial Password Manager with Intel® Software Guard Extensions.

The enclave boundary only crosses one connection: the link between the PasswordManagerCoreNative object and the Vault object. That suggests that the majority of our ECALLs will simply be wrappers around the class methods in Vault. We’ll also need to add some additional ECALLs to manage the enclave infrastructure. One of the complications of enclave development is that the ECALLs, OCALLs, and bridge functions must be native C code, and we are making extensive use of C++ features. Once the enclave has been launched, we’ll also need functions that span the gap between C and C++ (objects, constructors, overloads, and others).

The wrapper and bridge functions will go in their own DLL, which we’ll name EnclaveBridge.dll. For clarity, we’ll prefix the wrapper functions with ew_ (for “enclave wrapper”), and the bridge functions that make the ECALLs with ve_ (for “vault enclave”).

Calls from PasswordManagerCoreNative to the corresponding method in Vault will follow the basic flow shown in Figure 3.

Figure 3. Execution flow for bridge functions and ECALLs.

The method in PasswordManagerCoreNative will call into the wrapper function in EnclaveBridge.dll. That wrapper will, in turn, invoke one or more ECALLs, which enter the enclave and invoke the corresponding class method in the Vault object. Once all ECALLs have completed, the wrapper function returns back to the calling method in PasswordManagerCoreNative and provides it with a return value.

Enclave Logistics

The first step in designing the enclave is working out a system for managing the enclave itself. The enclave must be launched and the resulting enclave ID must be provided to the ECALLs. Ideally, this should be transparent to the upper layers of the application.

The easiest solution for the Tutorial Password Manager is to use global variables in the EnclaveBridge DLL to hold the enclave information. This design decision comes with a restriction: only one thread can be active in the enclave at a time. This is a reasonable solution because the password manager application would not benefit from having multiple threads operating on the vault. Most of its actions are driven by the user interface and do not consume a significant amount of CPU time.

To solve the transparency problem, each wrapper function will first call a function to check to see if the enclave has been launched, and launch it if it hasn’t. This logic is fairly simple:

#define ENCLAVE_FILE _T("Enclave.signed.dll")

static sgx_enclave_id_t enclaveId = 0;
static sgx_launch_token_t launch_token = { 0 };
static int updated= 0;
static int launched = 0;
static sgx_status_t sgx_status= SGX_SUCCESS;

// Ensure the enclave has been created/launched.

static int get_enclave(sgx_enclave_id_t *eid)
{
	if (launched) return 1;
	else return create_enclave(eid);
}

static int create_enclave(sgx_enclave_id_t *eid)
{
	sgx_status = sgx_create_enclave(ENCLAVE_FILE, SGX_DEBUG_FLAG, &launch_token, &updated, &enclaveId, NULL);
	if (sgx_status == SGX_SUCCESS) {
		if ( eid != NULL ) *eid = enclaveId;
		launched = 1;
		return 1;
	}

	return 0;
}

Each wrapper function will start by calling get_enclave(), which checks to see if the enclave has been launched by examining a static variable. If it has, then it (optionally) populates the eid pointer with the enclave ID. This step is optional because the enclave ID is also stored as a global variable, enclaveID, which can of course just be used directly.

What happens if an enclave is lost due to a power event or a bug that causes it to crash? For that, we check the return value of the ECALL: it indicates the success or failure of the ECALL operation itself, not of the function being called in the enclave.

sgx_status = ve_initialize(enclaveId, &vault_rv);

The return value of the function being called in the enclave, if any, is transferred via the pointer which is provided as the second argument to the ECALL (these function prototypes are generated for you automatically by the Edger8r tool). You must always check the return value of the ECALL itself. Any result other than SGX_SUCCESS indicates that the program did not successfully enter the enclave and the requested function did not run. (Note that we’ve defined sgx_status as a global variable as well. This is another simplification stemming from our single-threaded design.)

We’ll add a function that examines the error returned by the ECALL and checks for a lost or crashed enclave:

static int lost_enclave()
{
	if (sgx_status == SGX_ERROR_ENCLAVE_LOST || sgx_status == SGX_ERROR_ENCLAVE_CRASHED) {
		launched = 0;
		return 1;
	}

	return 0;
}

These are recoverable errors. The upper layers don’t currently have logic to deal with these specific conditions, but we provide it in the EnclaveBridge DLL in order to support future enhancements.

Also notice that there is no function provided to destroy the enclave. As long as the user has the password manager application open, the enclave is in place even if they choose to lock their vault. This is not good enclave etiquette. Enclaves draw from a finite pool of resources, even when idle. We’ll address this problem in a future segment of the series when we talk about data sealing.

The Enclave Definition Language

Before moving on to the actual enclave design, we’ll take a few moments to discuss the Enclave Definition Language (EDL) syntax. An enclave’s bridge functions, both its ECALLs and OCALLs, are prototyped in its EDL file and its general structure is as follows:

enclave {
	// Include files

	// Import other edl files

	// Data structure declarations to be used as parameters of the function prototypes in edl

	trusted {
	// Include file if any. It will be inserted in the trusted header file (enclave_t.h)

	// Trusted function prototypes (ECALLs)

	};

	untrusted {
	// Include file if any. It will be inserted in the untrusted header file (enclave_u.h)

	// Untrusted function prototypes (OCALLs)

	};
};

ECALLs are prototyped in the trusted section, and OCALLs are prototyped in the untrusted section.

The EDL syntax is C-like and function prototypes very closely resemble C function prototypes, but it’s not identical. In particular, bridge function parameters and return values are limited to some fundamental data types and the EDL includes some additional keywords and syntax that defines some enclave behavior. The Intel® Software Guard Extensions (Intel® SGX) SDK User’s Guide explains the EDL syntax in great detail and includes a tutorial for creating a sample enclave. Rather than repeat all of that here, we’ll just discuss those elements of the language that are specific to our application.

When parameters are passed to enclave functions, they are marshaled into the protected memory space of the enclave. For parameters passed as values, no special action is required as the values are placed on the protected stack in the enclave just as they would be for any other function call. The situation is quite different for pointers, however.

For parameters passed as pointers, the data referenced by the pointer must be marshaled into and out of the enclave. The edge routines that perform this data marshalling need to know two things:

1. Which direction should the data be copied: into the bridge function, out of the bridge function, or both directions?
2. What is the size of the data buffer referenced by the pointer?

Pointer Direction

When providing a pointer parameter to a function, you must specify the direction by the keywords in brackets: [in], [out], or [in, out], respectively. Their meaning is given in Table 1.

Table 1. Pointer direction parameters and their meanings in ECALLs and OCALLs.

Note from the table that the direction is relative to the bridge function being called. For an ECALL, [in] means “copy the buffer to the enclave,” but for an OCALL it’s “copy the buffer to the untrusted function.”

(There is also the option called user_check that can be used in place of these, but it’s not relevant to our discussion. See the SDK documentation for information on its use and purpose.)

Buffer Size

The edge routines calculate the total buffer size, in bytes, as:

bytes = element_size * element_count

By default, the edge routines assume element_count = 1, and calculate element_size from the element referenced by the pointer parameter, e.g., for an integer pointer it assumes element_size is:

sizeof(int)

For a single element of a fixed data type, such as an int or a float, no additional information needs to be provided in the EDL prototype for the function. For a void pointer, you must specify an element size or you’ll get an error at compile time. For arrays, char and wchar_t strings, and other types where the length of the data buffer is more than one element you must specify the number of elements in the buffer or only one element will be copied.

Add either the count or size parameter (or both) to the bracketed keywords for the pointer as appropriate. They can be set to a constant value or one of the parameters to the function. For most cases, count and size are functionally the same, but it’s good practice to use them in their correct contexts. Strictly speaking, you would only specify size when passing a void pointer. Everything else would use count.

If you are passing a C string or wstring (a NULL-terminated char or wchar_t array), then you can use the string or wstring parameter in place of count or size. In this case, the edge routines will determine the size of the buffer by getting the length of the string directly.

function([in, size=12] void *param);
function([in, count=len] char *buffer, uint32_t len);
function([in, string] char *cstr);

Note that you can only use string or wstring if the direction is set to [in] or [in, out]. When the direction is set only to [out], the string has not yet been created so the edge routine can’t know the size of the buffer. Specifying [out, string] will generate an error at compile time.

Wrapper and Bridge Functions

We are now ready to define our wrapper and bridge functions. As we pointed out above, the majority of our ECALLs will be wrappers around the class methods in Vault. The class definition for the public member functions is shown below:

class PASSWORDMANAGERCORE_API Vault
{
	// Non-public methods and members ommitted for brevity

public:
	Vault();
	~Vault();

	int initialize();
	int initialize(const char *header, UINT16 size);
	int load_vault(const char *edata);

	int get_header(unsigned char *header, UINT16 *size);
	int get_vault(unsigned char *edate, UINT32 *size);

	UINT32 get_db_size();

	void lock();
	int unlock(const char *password);

	int set_master_password(const char *password);
	int change_master_password(const char *oldpass, const char *newpass);

	int accounts_get_count(UINT32 *count);
	int accounts_get_info(UINT32 idx, char *mbname, UINT16 *mbname_len, char *mblogin, UINT16 *mblogin_len, char *mburl, UINT16 *mburl_len);

	int accounts_get_password(UINT32 idx, char **mbpass, UINT16 *mbpass_len);

	int accounts_set_info(UINT32 idx, const char *mbname, UINT16 mbname_len, const char *mblogin, UINT16 mblogin_len, const char *mburl, UINT16 mburl_len);
	int accounts_set_password(UINT32 idx, const char *mbpass, UINT16 mbpass_len);

	int accounts_generate_password(UINT16 length, UINT16 pwflags, char *cpass);

	int is_valid() { return _VST_IS_VALID(state); }
	int is_locked() { return ((state&_VST_LOCKED) == _VST_LOCKED) ? 1 : 0; }
};

There are several problem functions in this class. Some of them are immediately obvious, such as the constructor, destructor, and the overloads for initialize(). These are C++ features that we must invoke using C functions. Some of the problems, though, are not immediately obvious because they stem from the function’s inherent design. (Some of these problem methods were poorly designed on purpose so that we could cover specific issues in this tutorial, but some were just poorly designed, period!) We’ll tackle each problem, one by one, presenting both the prototypes for the wrapper functions and the EDL prototypes for the proxy/bridge routines.

The Constructor and Destructor

In the non-Intel SGX code path, the Vault class is a member of PasswordManagerCoreNative. We can’t do this for the Intel SGX code path; however, the enclave can include C++ code so long as the bridge functions themselves are pure C functions.

Since we have already limited the enclave to a single thread, we can make the Vault class a static, global object in the enclave. This greatly simplifies our code and eliminates the need for creating bridge functions and logic to instantiate it.

The Overload on initialize()

There are two prototypes for the initialize() method:

1. The method with no arguments initializes the Vault object for a new password vault with no contents. This is a password vault that the user is creating for the first time.
2. The method with two arguments initializes the Vault object from the header of the vault file. This represents an existing password vault that the user is opening (and, later on, attempting to unlock).

This will be broken up into two wrapper functions:

ENCLAVEBRIDGE_API int ew_initialize();
ENCLAVEBRIDGE_API int ew_initialize_from_header(const char *header, uint16_t hsize);

And the corresponding ECALLs will be defined as:

public int ve_initialize ();
public int ve_initialize_from_header ([in, count=len] unsigned char *header, uint16_t len);

get_header()

This method has a fundamental design issue. Here’s the prototype:

int get_header(unsigned char *header, uint16_t *size);

This function accomplishes one of two tasks:

1. It gets the header block for the vault file and places it in the buffer pointed to by header. The caller must allocate enough memory to store this data.
2. If you pass a NULL pointer in the header parameter, the uint16_t pointed to by size is set to the size of the header block, so that the caller knows how much memory to allocate.

This is a fairly common compaction technique in some programming circles, but it presents a problem for enclaves: when you pass a pointer to an ECALL or an OCALL, the edge functions copy the data referenced by the pointer into or out of the enclave (or both). Those edge functions need to know the size of the data buffer so they know how many bytes to copy. The first usage involves a valid pointer with a variable size which is not a problem, but the second usage has a NULL pointer and a size of zero.

We could probably come up with an EDL prototype for the ECALL that could make this work, but clarity should generally trump brevity. It’s better to split this into two ECALLs:

public int ve_get_header_size ([out] uint16_t *sz);
public int ve_get_header ([out, count=len] unsigned char *header, uint16_t len);

The enclave wrapper function will take care of the necessary logic so that we don’t have to make changes to other classes:

ENCLAVEBRIDGE_API int ew_get_header(unsigned char *header, uint16_t *size)
{
	int vault_rv;

	if (!get_enclave(NULL)) return NL_STATUS_SGXERROR;

	if ( header == NULL ) sgx_status = ve_get_header_size(enclaveId, &vault_rv, size);
	else sgx_status = ve_get_header(enclaveId, &vault_rv, header, *size);

	RETURN_SGXERROR_OR(vault_rv);
}

accounts_get_info()

This method operates similarly to get_header(): pass a NULL pointer and it returns the size of the object in the corresponding parameter. However, it is uglier and sloppier because of the multiple parameter arguments. It is better off being broken up into two wrapper functions:

ENCLAVEBRIDGE_API int ew_accounts_get_info_sizes(uint32_t idx, uint16_t *mbname_sz, uint16_t *mblogin_sz, uint16_t *mburl_sz);
ENCLAVEBRIDGE_API int ew_accounts_get_info(uint32_t idx, char *mbname, uint16_t mbname_sz, char *mblogin, uint16_t mblogin_sz, char *mburl, uint16_t mburl_sz);

And two corresponding ECALLs:

public int ve_accounts_get_info_sizes (uint32_t idx, [out] uint16_t *mbname_sz, [out] uint16_t *mblogin_sz, [out] uint16_t *mburl_sz);
public int ve_accounts_get_info (uint32_t idx,
	[out, count=mbname_sz] char *mbname, uint16_t mbname_sz,
	[out, count=mblogin_sz] char *mblogin, uint16_t mblogin_sz,
	[out, count=mburl_sz] char *mburl, uint16_t mburl_sz
);

accounts_get_password()

This is the worst offender of the lot. Here’s the prototype:

int accounts_get_password(UINT32 idx, char **mbpass, UINT16 *mbpass_len);

The first thing you’ll notice is that it passes a pointer to a pointer in mbpass. This method is allocating memory.

In general, this is not a good design. No other method in the Vault class allocates memory so it is internally inconsistent, and the API violates convention by not providing a method to free this memory on the caller’s behalf. It also poses a unique problem for enclaves: an enclave cannot allocate memory in untrusted space.

This could be handled in the wrapper function. It could allocate the memory and then make the ECALL and it would all be transparent to the caller, but we have to modify the method in the Vault class, regardless, so we should just fix this the correct way and make the corresponding changes to PasswordManagerCoreNative. The caller should be given two functions: one to get the password length and one to fetch the password, just as with the previous two examples. PasswordManagerCoreNative should be responsible for allocating the memory, not any of these functions (the non-Intel SGX code path should be changed, too).

ENCLAVEBRIDGE_API int ew_accounts_get_password_size(uint32_t idx, uint16_t *len);
ENCLAVEBRIDGE_API int ew_accounts_get_password(uint32_t idx, char *mbpass, uint16_t len);

The EDL definition should look familiar by now:

public int ve_accounts_get_password_size (uint32_t idx, [out] uint16_t *mbpass_sz);
public int ve_accounts_get_password (uint32_t idx, [out, count=mbpass_sz] char *mbpass, uint16_t mbpass_sz);

load_vault()

The problem with load_vault() is subtle. The prototype is fairly simple, and at first glance it may look completely innocuous:

int load_vault(const char *edata);

What this method does is load the encrypted, serialized password database into the Vault object. Because the Vault object has already read the header, it knows how large the incoming buffer will be.

The issue here is that the enclave’s edge functions don’t have this information. A length has to be explicitly given to the ECALL so that the edge function knows how many bytes to copy from the incoming buffer into the enclave’s internal buffer, but the size is stored inside the enclave. It’s not available to the edge function.

The wrapper function’s prototype can mirror the class method’s prototype, as follows:

ENCLAVEBRIDGE_API int ew_load_vault(const unsigned char *edata);

The ECALL, however, needs to pass the header size as a parameter so that it can be used to define the size of the incoming data buffer in the EDL file:

public int ve_load_vault ([in, count=len] unsigned char *edata, uint32_t len)

To keep this transparent to the caller, the wrapper function will be given extra logic. It will be responsible for fetching the vault size from the enclave and then passing it through as a parameter to this ECALL.

ENCLAVEBRIDGE_API int ew_load_vault(const unsigned char *edata)
{
	int vault_rv;
	uint32_t dbsize;

	if (!get_enclave(NULL)) return NL_STATUS_SGXERROR;

	// We need to get the size of the password database before entering the enclave
	// to send the encrypted blob.

	sgx_status = ve_get_db_size(enclaveId, &dbsize);
	if (sgx_status == SGX_SUCCESS) {
		// Now we can send the encrypted vault data across.

		sgx_status = ve_load_vault(enclaveId, &vault_rv, (unsigned char *) edata, dbsize);
	}

	RETURN_SGXERROR_OR(vault_rv);
}

A Few Words on Unicode

In Part 3, we mentioned that the PasswordManagerCoreNative class is also tasked with converting between wchar_t and char strings. Given that enclaves support the wchar_t data type, why do this at all?

This is a design decision intended to minimize our footprint. In Windows, the wchar_t data type is the native encoding for Win32 APIs and it stores UTF-16 encoded characters. In UTF-16, each character is 16 bits in order to support non-ASCII characters, particularly for languages that aren’t based on the Latin alphabet or have a large number of characters. The problem with UTF-16 is that a character is always 16-bits long, even when encoding plain ASCII text.

Rather than store twice as much data both on disk and inside the enclave for the common case where the user’s account information is in plain ASCII and incur the performance penalty of having to copy and encrypt those extra bytes, the Tutorial Password Manager converts all of the strings coming from .NET to the UTF-8 encoding. UTF-8 is a variable-length encoding, where each character is represented by one to four 8-bit bytes. It is backwards-compatible with ASCII and it results in a much more compact encoding than UTF-16 for plain ASCII text. There are cases where UTF-8 will result in longer strings than UTF-16, but for our tutorial password manager we’ll accept that tradeoff.

A commercial application would choose the best encoding for the user’s native language, and then record that encoding in the vault (so that it would know which encoding was used to create it in case the vault is opened on a system using a different native language).

Sample Code

As mentioned in the introduction, there is sample code provided with this part for you to download. The attached archive includes the source code for the Tutorial Password Manager bridge DLL and the enclave DLL. The enclave functions are just stubs at this point, and they will be filled out in Part 5.

Coming Up Next

In Part 5 of the tutorial we’ll complete the enclave by porting the Crypto, DRNG, and Vault classes to the enclave, and connecting them to the ECALLs. Stay tuned!