A Cup of TEE, Please. But How Do We Know It's The Right Flavor?

Remote attestation and reproducible builds are crucial in the context of Trusted Execution Environments (TEEs) because they ensure the sanctity of sensitive computing environments. Remote attestation is a mechanism that guarantees the integrity and trustworthiness of a TEE, while reproducible builds verify that software built from the same source code always produces identical binaries. Together they allow proof that a remote program corresponds to audited source code.

In a bit more detail, here's how these critical aspects of TEEs work. 

With VM-based TEE abstractions like Intel TDX, the whole process of measuring and attesting the TEE is a rather long chain of trust. One broken link and the TEE may not really be a TEE after all, or it could be a different flavor of TEE, one containing a backdoor.

The cryptographic identity of a TEE is a set of measurements. The measurements are stored in registers, which accumulate values using cryptographic hash functions. With TDX, the TEE identity is the set of five values: MRTD, RTMR{0-3}. But what are these magic measurement values, and how can you use them to verify anything? 

The measurements are established during the VM boot process, where early boot stages measure later stages before executing them. This ensures that the modification of any one component results in a different TEE identity. Okay, so how exactly does a TDX VM boot?

• First, the TDX Module running in privileged SEAM mode on the CPU is responsible for provisioning the TDX VMs, managing memory encryption keys, etc., serving as the trusted part of the hypervisor (the program that manages the life cycle and resources of a virtual machine), isolated from the host OS.

• Next, when a TDX VM is initialized by calling into the TDX Module, the first part to be loaded is the virtual firmware (called TDVF) which is responsible for early boot. It would usually be based on OVMF, but ideally, it would be something smaller written in a memory-safe language like Rust. The cryptographic hash of the loaded TDVF pages is recorded as MRTD, which is then locked before boot and cannot be extended further.

• The VM boot process then proceeds similarly to how secure boot works. The TDVF is responsible for measuring all of the untrusted inputs from the untrusted hypervisor – e.g., the VM memory configuration, the CPU topology, ACPI tables, the kernel image, the kernel command-line, etc. – into runtime measurement registers (the RTMRs) which can only be extended by hashing their current value together with additional data (similar to how PCRs work in TPMs/secure boot). 

• After extending the relevant RTMRs, the TDVF loads the Linux kernel. If, for some reason, there is a bug in the TDVF and some vital part is not measured or can be bypassed, this can compromise the security of the TEE.

 In Practice: Remote Attestation for ROFL Apps

To put this in a broader context, Oasis is building Runtime Offchain Logic (ROFL), a fun and useful framework that makes developing and deploying TEE-based applications simple, secure, efficient, and accessible to Web3 smart contracts. Each ROFL app runs in its own TEE, which can be a TDX VM, and uses Sapphire for decentralized remote attestation policy management, key manager access, and a compute marketplace.

How this works for ROFL apps: Default ROFL app images currently use a minimal build of OVMF, there are plans to reduce this further by using a small TDVF written in Rust. For the Linux kernel, the default ROFL app images use a minimal Linux kernel with bundled initramfs, which contains a simple stage 1 init that parses the kernel command line, which has previously been measured into one of the RTMRs. 

The kernel command line contains the layout of the stage 2 virtual disk image and also the root hash of the block device containing the read-only stage 2 filesystem. This information is used to set up the block device, mount the stage 2 filesystem, and run the stage 2 init. The reason for needing the root hash is the fact that access to the underlying virtual block device is untrusted and may be manipulated by the host. Since the root hash has been measured, any change in the root filesystem results in a different TEE identity.

The stage 2 init is responsible for further initialization of the TEE, performing remote attestation, establishing freshness and registration on Sapphire. To do that, it needs to generate a report that contains the set of measurements (MRTD, RTMR{0-3}). This report needs to be signed using a certificate chain that is eventually rooted in Intel Root CA – this signed report is a quote.

To generate a quote, the TDX VM issues instructions that securely communicate with an SGX enclave running on the host. Wait what!? Where did SGX come from? You might be wondering… Well, this SGX enclave is the TD quoting enclave, which has access to the attestation key, which is used to produce a signature of the report (after its careful validation). This makes it possible to reuse the existing remote attestation primitives used for SGX.

The attestation keys are generated during secure CPU provisioning, which happens on the first boot and subsequent TCB recoveries or CPU topology changes (e.g., in the case of multi-CPU platforms). During CPU provisioning, an Intel service (called the PCS) issues a signed certificate (called the PCK certificate) which is used in all the subsequent processes of remote attestations. The PCK certificate is embedded in the generated quote to aid verification.

The generated quote is submitted to Sapphire together with the required collateral for verification. Onchain logic on Sapphire knows the Intel Root CA and can decode the quote and verify the entire chain of signatures to get the verified report containing the set of measurements. Additionally, it checks that the host platform has the proper microcode and TDX Module security updates applied, which is reflected in other fields of the quote.

It then compares the set of measurements against the policy stored on Sapphire for the given ROFL app. Only if the set of measurements matches is the app instance considered authenticated and can access services like the decentralized key management system to derive secrets.

As you can see, the process involves many components that form the trusted computing base of the application running in a TEE. For this reason, it's critical that source code is available and that all components can be built in a reproducible way (e.g., such that one can get exactly the same binaries by building all of the code locally). Fortunately, all ROFL app components are built with reproducibility in mind, and one can also reproduce the entire set of measurements locally without even having TDX as simply as running oasis rofl build --verify!

Now we can finally enjoy our TEE 🍵

‍