SGX-LKL: Securing the Host OS Interface for Trusted Execution
Christian Priebe, Divya Muthukumaran, Joshua Lind, Huanzhou Zhu, Shujie Cui, Vasily A. Sartakov, Peter Pietzuch
SSGX-LKL: Securing the Host OS Interface for Trusted Execution
Christian Priebe Divya Muthukumaran Joshua Lind Huanzhou ZhuShujie Cui Vasily A. Sartakov Peter Pietzuch
Imperial College London
Abstract
Hardware support for trusted execution in modern CPUs en-ables tenants to shield their data processing workloads inotherwise untrusted cloud environments. Runtime systems fortrusted execution must rely on an interface to the untrustedhost OS to use external resources such as storage, network,and other functions. Attackers may exploit this interface toleak data or corrupt the computation.We describe
SGX-LKL , a system for running Linux bina-ries inside of Intel SGX enclaves that exposes only a minimal,protected and oblivious host interface: the interface is (i) min-imal because SGX-LKL uses a complete library OS insidethe enclave, including file system and network stacks, lead-ing to an interface with only 7 calls; (ii) protected becauseSGX-LKL transparently encrypts and integrity-protects alldata passed via low-level I/O operations; and (iii) oblivious because SGX-LKL performs host operations independentlyof the application workload: for oblivious disk I/O, it uses anencrypted ext4 file system with randomised disk blocks; foroblivious network I/O, it uses layer-3 encryption with dummypackets for constant bit rates between trusted nodes. We showthat SGX-LKL can protect TensorFlow training with only a21% overhead.
In healthcare [3,76,115] and finance [21,55], it is challengingto run sensitive workloads in public cloud environments due tosecurity concerns [24]. For example, when training a machinelearning model, both the training data and the model may beconfidential, and the integrity of the model must be ensured.In clouds, the data and computation may be disclosed or cor-rupted due to infrastructure bugs [45], misconfigurations [98],rogue administrators [37, 66], or external attacks [5, 44].Modern Intel [25], ARM [6] and AMD [4] CPUs offer hard-ware support for trusted execution environments (TEEs) [86].A TEE protects the confidentiality and integrity of compu-tation and data by shielding it from the rest of the system.Intel’s
Software Guard Extensions (SGX) [25] add new in-structions to execute code within isolated memory regions called enclaves . Enclaves are encrypted by the CPU.Cloud providers have begun to roll out support forTEEs [48, 68]. This makes a “lift-and-shift” model—in whichtenants move a whole application to a TEE—an attrac-tive proposition. TEE runtime systems, such as Haven [9],SCONE [7] and Graphene-SGX [101], have demonstrated thefeasibility of executing complete Linux applications insideTEEs with acceptable performance overheads.When applications inside a TEE require external resources,such as files, the network or other OS functions, they must relyon the untrusted host OS. TEE runtime systems therefore havea host interface, but its security implications are poorly un-derstood and handled: the host interface may (i) accidentallyexpose state from the TEE to the outside by leaking sensitivedata in calls [94]; (ii) act as a side channel where the exis-tence or absence of a call reveals application state [108]; and(iii) have a malicious implementation and thus compromiseapplication integrity inside the TEE [20].Existing TEE runtime systems vary in the types of inter-faces that they expose to hosts, and the nature of the interfaceis a consequence of their design. We explore the oppositeapproach: we begin our design of a TEE runtime systemwith a desired secure host interface, and then decide on thenecessary system support inside the TEE. We aim for a hostinterface with three properties: (1) minimality —only function-ality that cannot be provided inside the TEE should be partof it; (2) protection —all data that crosses the host interfacemust be encrypted and integrity-protected; and (3) oblivi-ousness —the presence or absence of host calls should notdisclose information about the application state.We describe
SGX-LKL , a new TEE runtime system thatexecutes unmodified Linux binaries inside SGX enclaveswhile exposing a minimal, data-protected and oblivious hostinterface. The design of SGX-LKL makes three contribu-tions: (1) Minimal host interface (§3).
SGX-LKL only requires7 host interface calls, which expose low-level functionality SGX-LKL is available as open-source software: https://github.com/lsds/sgx-lkl . a r X i v : . [ c s . O S ] J a n or block-level I/O for storage and packet-level I/O for net-working. Higher-level POSIX functionality is implementedin SGX-LKL by porting a complete Linux-based library OSto an SGX enclave. SGX-LKL uses the Linux Kernel Li-brary (LKL) [83] for a mature POSIX implementation with avirtual file system layer and a TCP/IP network stack. (2) Protected host interface (§4).
SGX-LKL ensures that allI/O operations across the host interface are encrypted andintegrity-protected transparently: (i) for file I/O, SGX-LKLuses an encrypted Linux ext4 root file system image storedoutside of the SGX enclave, which is accessed using the Linux device mapper subsystem [31]; and (ii) for network I/O, SGX-LKL creates a virtual private network (VPN) overlay thatsecures all network traffic. Layer-3 IP packets are encryptedby the in-kernel Wireguard [112] VPN implementation.To verify the integrity of a SGX-LKL instance and providecryptographic keys, SGX-LKL supports a runtime attestationand provisioning process: it first attests the integrity of theSGX-LKL implementation inside the SGX enclave and thenprovisions the instance with the keys required to access theencrypted root file system image and the VPN channels. (3) Oblivious host interface (§5).
To prevent any informa-tion leakage as part of the host interface, SGX-LKL makesthe calls independent of the application workload inside ofthe SGX enclave: (i) it executes host calls in fixed batches,and each batch includes the same number of calls by addingindistinguishable dummy calls; (ii) for block I/O requests,SGX-LKL uses an oblivious construction that reuses the ex-isting ext4 file system image format. The encrypted blocks inthe file system image are randomised and shuffled, which onlyexposes random access patterns to the host. Between shuf-fles, SGX-LKL reads a block at most once by relying on thein-enclave page cache; and (iii) for network I/O, SGX-LKLsends fixed-size packets and makes use of Linux’ traffic shap-ing capabilities to ensure a constant traffic bitrate betweentrusted nodes.Our experimental evaluation shows that SGX-LKL’s hostinterface has a reasonable performance overhead. With emu-lated SGX enclaves (to ignore current SGX memory limita-tions) and oblivious calls, SGX-LKL trains common deep neu-ral network models using TensorFlow [99] with an overheadof 14%–21% on one machine and 2.7 × –2.9 × in a distributeddeployment. With SGX hardware, SGX-LKL runs PARSECbenchmarks that fit into SGX memory with an overhead of3.1 × and 1.5 × with and without oblivious calls, respectively. We assume that a cloud provider supports CPU-implemented trusted execution environments (TEEs) . TEEs separateuserspace code from the rest of the system, including privi-leged software such as the OS [38]. Multiple TEE implemen-tations are available, e.g. Intel SGX [25], ARM TrustZone [6] and AMD SEV [4], with others under way [60, 77, 89].Intel’s
Software Guard Extensions (SGX) [25] provide newCPU instructions to create TEEs called enclaves . Enclavememory is encrypted and integrity-protected transparentlyby the CPU. The CPU controls transitions from untrustedto enclave code, and only enclave code can access enclavememory. Due to this isolation, enclave code has no I/O accessbut must use the untrusted host.Intel SGX also supports attestation , allowing a remote partyto validate the enclave code. For this, the CPU (i) measuresthe enclave’s contents by computing a hash; (ii) signs themeasurement hash; and (iii) provides it to the attesting party.The attestor can verify the signature and the hash.For the deployment of applications using TEEs, we focuson a lift-and-shift model [74]: users deploy unmodified Linuxbinaries. For binary compatibility, this requires a
TEE runtimesystem [7, 9, 101], which provides POSIX abstractions. TEEruntime systems must use functionality by the untrusted hostOS for operations outside of the trust domain of the TEE, e.g.when using I/O resources. We refer to the interface betweenthe TEE runtime system and the host as the host interface . We want to prevent an adversary from compromising the con-fidentiality and integrity of the application code and its inputand output data. We consider all software outside the TEE,including the host OS kernel, as under adversarial control.We assume that the TEE itself is trustworthy. Existingattacks against TEEs, e.g. Spectre [64], Foreshadow [104],Zombieload [88], and controlled-channel attacks [114] areorthogonal to our work. They exploit flaws in current TEE im-plementations and can be mitigated through hardware and/ormicrocode changes [52]. As the maturity of TEE implemen-tations, especially new open-source ones [4, 50, 60], growsover time, such attacks will become rarer. Cache side-channelattacks [15, 42, 71] are not specific to TEEs and are enabledby micro-architectural resource sharing. They require funda-mental mitigations through compiler techniques [14, 22, 75]and defensive programming [29, 78, 87].Our threat model instead focuses on attacks against the hostinterface because they are easy to carry out and do not assumeparticular micro-architectural behaviour. An adversary maycompromise (i) confidentiality : they can observe the param-eters, frequencies and sequences of host calls. For example,they can learn the disk I/O access pattern to determine theapplication or workload being run—linear scans and repeatedaccesses are visible via the host interface. This side channeldiscloses information about the TEE execution; and compro-mise (ii) integrity : an adversary can modify the input/outputparameters of host calls, repeatedly perform arbitrary hostcalls or interfere with their outside execution. For example,an adversary may trick an enclave into exposing confidentialdata via
Iago attacks [20] or hijack its control flow.2 able 1: Security breakdown of parameters in host interface calls for existing TEE runtime systems
TEEruntimesystem Function Number ofhost calls Out parameters (impact confidentiality)
More difficult to protect −−−−−−−−−−−−−−−−−−−−−−−−−−−−−→
In parameters (impact integrity)
More difficult to protect −−−−−−−−−−−−−−−−−−−−−−−−−−−−−→
Variablesize buffer Addressrange Pureidentifier Impureidentifier Semanticparameters Variablesize buffer Addressrange Pureidentifier Impureidentifier Semanticparameters
Panoply [90] I/O 239 11 17 96 49 139 10 2 22 24 74Events 22 – – 5 – 22 – – 2 – 8Time 12 – – – – 7 – – – – 10Threading 29 1 – 10 – 17 2 – 4 – 10
Graphene-SGX [101] I/O 29 3 1 18 6 19 3 2 2 4 7Events 1 – – 1 – 1 – – – – 1Time 2 – – – – 1 – – – – 2Threading 6 – – – 1 4 1 – 1 – 1
Haven [9] I/O 11 1 3 6 1 7 1 – 3 – 1Events 6 – – 5 – 1 – – 1 – 3Time 1 – – – – – – – – – 1Threading 6 – – 1 – 2 – – 1 – –
SGX-LKL
I/O 4 2 – – – 2 2 – – – –Events 2 – – – – 1 – 1 – – 2Time 1 – – – – – – – – – 1Threading – – – – – – – – – – –
Next we analyse the security of the host interfaces of fourexisting TEE runtime systems:(1)
Panoply [90] minimises the size of the trusted computingbase (TCB) inside the TEE. It places the glibc standard Clibrary [82] outside and implements a shim layer that forwardscalls. Panoply exposes the largest host interface because itdelegates all C library calls to the host.(2)
SCONE [7] provides a modified musl standard C li-brary [73] alongside the application inside the TEE. Com-pared to Panoply, SCONE has a larger TCB but a smaller hostinterface: it forwards only system calls and not all libc callsto the host. It uses a custom shim layer to protect I/O calls.(3)
Graphene-SGX [101] reduces the size of the host interfacecompared to Panoply and SCONE by implementing a partiallibrary OS inside the TEE. It relies on the host for the filesystem, network stack, and threading implementations.(4)
Haven [9] uses the Drawbridge library OS [81] to executeWindows applications. Drawbridge also provides file systemand network stack implementations inside the enclave.
Types of host interface parameters.
In our security analysis,we focus on the parameters of host calls because they can leakinformation to the host or compromise enclave integrity. Wecategorise parameters into five types, ordered by the difficultyof protection from easiest to hardest: (i) variable-sized buffers pass a user-defined byte array across the host interface. Theyare used in file/network I/O operations (e.g. the buf and count parameters in the read() call [85]); (ii) address ranges rep-resent parameters that refer to regions of untrusted or trustedmemory (e.g. the parameters passed to mmap() and the re-turn value of malloc() ); (iii) pure/impure identifiers point toentities: pure identifiers only identify an entity (e.g. a file de-scriptor) ; impure identifiers also disclose information aboutthe entity (e.g. a path name); and (iv) semantic parameters refer to parameters with opaque semantics specific to the hostcall (e.g. mode and flags for file access operations).Tab. 1 shows the number of host calls for each system, split by function (I/O, events, time, threading). Panoply has a largeinterface, with 302 calls in total; Graphene-SGX and Havenrequire 38 and 24 calls, respectively.We break down the parameters according to the abovetypes, distinguishing between out parameters, which arepassed to the host and may compromise confidentiality, and in parameters, which are passed into the TEE and may af-fect integrity. Return values of calls out of the enclave areconsidered to be the same as in parameters. Confidentiality attacks.
To learn sensitive information, anadversary may observe out parameters:(i)
Variable-sized buffers may contain security-sensitive data.To ensure confidentiality, such data must be encrypted andpadded to a fixed size. If the true size is exposed, an adversarymay infer size-dependent secrets. For example, in an imageclassification application [58], an adversary may learn theclassification by considering different result buffer lengths.Tab. 1 shows that Panoply, Graphene-SGX and Haven use11, 3 and 1 buffers, respectively, for I/O operations; Panoplyalso uses a buffer to share messages between threads. Buffersare encrypted but their sizes are disclosed. Haven has anin-enclave file system and writes data to the host disk as fixed-sized blocks; SCONE divides files into fixed-size chunks forauthentication but reveals the number of chunks; Graphene-SGX and Panoply do not provide transparent file encryption.SCONE, Panoply, and Graphene all reveal file sizes.(ii)
Address ranges passed from the TEE to the host point tocontinuous regions of untrusted memory. For I/O operations,Panoply, Graphene-SGX and Haven expose 17, 1 and 3 ad-dress ranges, respectively. An adversary may observe theirusage pattern. For example, Panoply uses untrusted memoryfor communication between enclaves that isolate applicationcompartments. The usage pattern reveals application-specificcontrol flow, e.g. secret-dependent inter-enclave calls. For Panoply and Graphene-SGX, the host calls are taken from the GitHubsource code [43, 79]; for Haven, they are obtained from the paper [9]. We donot breakdown SCONE’s interface because its implementation is not public.As SCONE forwards system calls, its host interface is similar to Panoply’s.
Pure/impure identifiers.
Panoply, SCONE, Graphene-SGX, and Haven use a large number of identifiers for I/Ooperations, event handling and threading. Similar to addressranges, passing identifiers to the host reveals usage patterns.For example, Cash et al. [16] show that an adversary can learndatabase queries by observing record accesses, even if thequeries are executed within a TEE.Generally, pure identifiers reveal less information than im-pure identifiers (e.g. file names), as pure identifiers can bechosen randomly. Preventing information leakage by impureidentifiers can only be done on a case-by-case basis (e.g. byreplacing file names with hashes). The above systems do notprovide any protection for impure identifiers.(iv)
Semantic parameters have a context-specific meaning andtherefore cannot be encrypted transparently. Existing runtimesystems have a large number of unprotected semantic param-eters. For example, Graphene-SGX, Panoply, and Haven allrely on the host for thread synchronisation. Graphene-SGXexposes address, operation, value and timeout parameters of a futex host call. This gives an adversary detailed informationof the application state and enables attacks such as
Async-Shock [110], which exploit host control of enclave threads.
Integrity attacks.
To compromise integrity, an adversary maytamper with parameters. Integrity protecting in parametersvaries in difficulty according to the parameter type:(i) Variable-sized buffers passed to the TEE must have theirintegrity protected. For I/O calls, Panoply, Graphene-SGX,and Haven use 10, 3 and 1 parameters with variable-sizedbuffers, respectively. Panoply and Graphene-SGX also usevariable-sized buffers for inter-enclave communication.This exposes two means of integrity attacks: an adversarymay modify (i) the buffer contents to violate data integrity;and (ii) the buffer count to trigger an overflow. To ensurebuffer integrity, the contents must be protected by an HMAC.The TEE must ensure the freshness of the HMAC, otherwisean adversary can swap two valid buffers or perform a roll-back attack. Panoply protects the integrity of the inter-enclavecommunication buffer using TLS but does not protect file I/O;Graphene-SGX ensures the integrity of file I/O by maintaininga Merkle tree of file chunk hashes inside the enclave. NeitherPanoply, Graphene-SGX nor Haven protect the integrity ofnetwork I/O for applications without TLS support.To prevent buffer overflow attacks, the TEE must checkbuffer lengths. All analysed runtime systems do this.(ii)
Address ranges are difficult to integrity check. For I/Ooperations, Panoply and Graphene-SGX use 2 address rangeseach. They check that the ranges are fully inside or outside ofTEE memory, preventing control flow hijacking [11, 61].An adversary, however, may still manipulate addresses byreusing old ranges to roll back data, swapping ranges, or mod-ifying ranges to corrupt data. The three runtime systems arevulnerable to such attacks, and the only effective mitigationis to avoid relying on untrusted address ranges.(iii)
Pure/impure identifiers are commonly used as in parame- ters: Panoply uses 46, 2, and 4 identifiers for I/O, events, andthreading, respectively; Graphene-SGX uses 6 identifiers forI/O and 1 for threading; Haven uses 3, 1, and 1 identifiers forI/O, events and threading, respectively.Adversaries may pass invalid or manipulated identifiers: iftwo file descriptors are swapped, the enclave may access incor-rect files. Such malicious activity can be detected: Graphene-SGX maintains per-file HMACs, revealing wrong file descrip-tors, and incorrect sockets are revealed by TLS.(iv) Semantic parameters must have their integrity verified ona case-by-case basis. All three systems use many semantic in parameters for I/O, events, time and threading operations.For semantic parameters with few valid values, such as errorcode and signum , the TEE can perform explicit checks;for ones with a larger domain, such as size , bounds checks arepossible. None of these checks establish semantic correctnessthough. Since both Panoply and Graphene-SGX rely on thehost file system, they can only check for the plausibility ofreturned file meta-data, such as st_size and st_blocks . While host interfaces of existing runtime systems as wellas SDKs have also been shown to contain implementationflaws [105], the above issues related to the host interface aremore fundamental. We conclude that it is non-trivial (andat worst impossible) to protect out parameters from leakinginformation and exposing access patterns and to verify thecorrectness of in parameters. Therefore, the first step in design-ing a secure host interface is to keep it narrow and minimisethe number of parameters. Only functionality that cannot beprovided within an enclave should be delegated to the host.In addition, we want to reduce the number and complexityof host call parameters. By building on low-level calls insteadof high-level POSIX abstractions, the parameter types becomesimpler. For example, Panoply, SCONE, and Graphene-SGXrely on the host file system and thus must expose impureidentifiers such as file names. A secure host interface shouldavoid delegating high-level resource management to the host.Based on these principles, Tab. 1 shows the host calls andparameter types of SGX-LKL , our TEE runtime system. Itavoids address range and pure/impure identifiers and onlyrequires a few simple semantic parameters of which all butone ( time ) have a fixed set of valid values or can be bounds-checked.
Tab. 2 shows SGX-LKL’s host interface, which has only7 calls with functionality that cannot be provided inside en-claves.
SGX-LKL uses a low-level I/O interface:for disk I/O, disk_read() and disk_write() read/write a diskblock from/to a persistent block device, respectively. Each call4 able 2: SGX-LKL host interface
Call DescriptionI/O disk_read(offset, buf)
Reads 4 KB block from disk image file at offsetdisk_write(offset, buf)
Writes 4 KB block from buf to disk image file at offsetnet_read(buf)
Reads MTU-sized Ethernet frame from network device into bufnet_write(buf)
Writes MTU-sized Ethernet frame from buf to network device
Events net_poll(eventmask) → reventmask Waits for eventmask events on network device; returns occurred events forward_signal(num, code, addr)
Forwards signal num with description code occurred at addr to enclave
Time time_read() → time Reads time from untrusted vDSO memory region ext4 file systemLKLmusl
Unmodified application/runtime
EnclaveHost
System call interface libc TAP device Virtual diskSignal handler Virtio networkbackend Virtio block backend
LKL host operations Network/block device operations
File systemNetwork stackWork QueuesPage Cache … Crypto libc API
Host OS Library … libsgxlkl.so HostInterface disk_read/disk_writetime_readforward_signal sgx-lkl-run
Device mapperWireguardOblivious I/O net_read/net_write/net_poll
ThreadingMemory TimeSignal handler
Fig. 1: SGX-LKL architecture takes an offset into the block device and a pointer buf to abuffer; for network I/O, net_read() and net_write() receiveand send fixed-size network packets using buffer buf . Events. A net_poll() call passes an eventmask to the hostwith the network events that SGX-LKL is waiting for. Thecall blocks until network packets are available to be read oroutgoing packets can be sent. It returns which events haveoccurred.SGX-LKL must handle hardware exceptions, such as pageaccess violations or illegal instructions. An exception causesan enclave exit and transfers control to the host kernel. A forward_signal() call provides the signal description to theenclave: the signal number num , the cause code , and the as-sociated memory address addr . The exception is then eitherprocessed by SGX-LKL directly or forwarded to the applica-tion if it has registered a corresponding signal handler. Time.
The call time_read() reads time from different clocksources. It is used by application code and by SGX-LKL, e.g.to generate timer interrupts required by LKL (see §3.2).
Its host interface allows SGX-LKL to access low-level re-sources, but Linux applications require higher-level POSIXabstractions. To bridge this gap, SGX-LKL provides the fol-lowing OS functions inside the enclave: (i) file system im-plementations; (ii) a TCP/IP network stack; (iii) threadingand scheduling; (iv) memory management; (v) signal han- dling; and (vi) time. Rather than invoking these OS functionsthrough systems calls directly, applications link against a stan-dard C library ( libc ). Similar to other systems [7, 101], SGX-LKL includes a libc implementation to support unmodifieddynamically-linked binaries.Fig. 1 shows the SGX-LKL architecture. Next we describethe OS functionality provided inside the enclave in detail.Components covered in this section are shown in blue; theparts responsible for protecting the host interface (shownin green) and for making the interface oblivious (shown inorange) are described in §4 and §5, respectively.An untrusted loader, sgx-lkl-run , creates the SGX enclaveand loads the enclave library libsgxlkl.so , which runs along-side the application within the enclave. libsgxlkl.so includes amodified musl [73] C standard library that redirects systemcalls to an in-enclave library OS provided by the
Linux Ker-nel Library (LKL) [83]. LKL is an architecture port of theLinux kernel to userspace. It enables SGX-LKL to make com-ponents such as the Linux kernel page cache, work queues,file system and network stack implementations, and crypto li-braries available inside the enclave. As it is intended to run inuserspace, LKL expects a set of
LKL host operations , e.g. tocreate threads or to allocate memory. Therefore, libsgxlkl.so includes further components for memory management, user-level threading, signal handling and time.
File systems.
Existing TEE runtime systems [7, 90, 101] for-ward POSIX file operations to the host. SGX-LKL cannotadopt this approach because it would expose security-sensitivemetadata, such as file names, file sizes and directory structures.Instead, it provides complete in-enclave file system implemen-tations via the Linux virtual file system (VFS) layer [13]. TheVFS layer only requires two host operations for block-leveldisk I/O ( disk_read() and disk_write() , see Table 2).Applications operate on files through file descriptors asusual, which are handled by the ext4 file system implementa-tion of LKL. LKL forwards block-level I/O requests to a virtioblock device backend implemented by libsgxlkl.so , which is-sues disk_read() and disk_write() calls. On the host, thereads and writes are made to a single ext4 disk image file. Theimage is mapped into memory by sgx-lkl-run . Since the imagefile has a fixed size, the read/write operations can be imple-mented efficiently by memory-mapping the file and directlyaccessing the mapped region from within the enclave.This approach has three advantages: (i) it maintains a small5ost interface with only 2 disk I/O calls; (ii) it ensures thatindividual file accesses are not visible to the host, which canonly observe reads/writes to disk block offsets; and (iii) the in-enclave VFS implementation supports different file systems,such as the /tmp , /proc and /dev in-memory file systems. TCP/IP network stack.
To provide POSIX sockets, SGX-LKL uses LKL’s TCP/IP stack for in-enclave packet process-ing. This: (i) minimises the host interface because it onlyaccesses a virtual network device to send/receive Ethernetframes; (ii) enables SGX-LKL to support any transport pro-tocol (e.g. UDP) without extra host calls; and (iii) exposesLinux features such as packet encryption (see §4.2).To send/receive network traffic, sgx-lkl-run sets up a layer-2 TAP device. SGX-LKL implements a corresponding vir-tio network device backend inside the enclave. To be noti-fied about incoming/outgoing packets, the backend issues a net_poll() request. The return value indicates if the deviceis ready for reading/writing packets using net_read() and net_write() . Memory management.
SGX-LKL does not interact withthe host for memory allocations/deallocations but a limita-tion of SGX version 1 is that the enclave size must be fixedat initialisation time. SGX-LKL therefore pre-allocates en-clave memory and provides low-level memory managementprimitives inside the enclave. When an enclave is created,it initially contains libsgxlkl.so , and an uninitialised heaparea. The heap area is exposed through both LKL functions( mmap() , mremap() , munmap() ) and higher-level libc functions( malloc() , free() ).SGX-LKL supports both variable- and fixed-address anony-mous mappings, and tracks free pages via a heap allocationbitmap. It implements mmap() by scanning the bitmap for con-secutive free pages large enough for the requested allocation.To support private file mappings, files are loaded into enclavebecause SGX enclaves are bound to a linear address space.SGX-LKL must support changes to page permissions, e.g.when loading libraries. Applications may also modify per-missions directly: e.g. the Java Virtual Machine (JVM) re-quires executable pages for just-in-time (JIT) compilation andchanges of permissions for guard pages during garbage col-lection. While SGX pages have their own permissions, SGXversion 1 requires permissions to be fixed on enclave creation.As a workaround, SGX-LKL uses an extra mem_protect() host call for SGX version 1 execution. All enclave pages arecreated with full SGX page permissions and the actual permis-sions are set via the host-controlled page table permissions.Since relying on the host to manage page permissions is asecurity risk, SGX version 2 adds the ability to control pagepermissions from within the enclave. Thread management.
SGX allows multiple host threads toenter an enclave, The maximum number of enclave threads ,however, must be specified at enclave creation time, whichprevents dynamic thread creation. In addition, a one-to-onemapping between enclave and application threads means that the creation, joining as well as synchronisation of threadsrequires host OS support, posing a security risk [110].Therefore, SGX-LKL implements user-level threadingbased on the lthread library [65] and provides synchroni-sation primitives inside the enclave. A fixed number of hostthreads are assigned to enclave threads, which enter the en-clave at startup and only leave when idle. Application threadsand LKL kernel threads are lthreads , managed via the stan-dard pthreads interface. SGX-LKL implements futex callsin-enclave to provide synchronisation primitives such as mu-texes and semaphores.
Signals.
Applications can register custom signal handlers tohandle exceptions, interrupts or user-defined signals. Somesignals (
SIGALRM ) are handled entirely within the enclave;others (
SIGSEGV , SIGILL ) are caused by hardware exceptions,exit the enclave and return control to the host OS. SGX-LKLforwards these signals from the host to in-enclave handlers.During initialisation, sgx-lkl-run registers signal handlersfor all catchable signals with the host. All signals areforwarded via forward_signal() , which hides application-specific handlers. SGX-LKL then checks for a correspondingapplication-registered signal handler and, if present, deliversthe signal. Since application-registered signal handlers aremanaged within the enclave, calls such as sigaction() and sigprocmask() are supported without host interaction.
Time.
Applications, libraries and LKL frequently access timeinformation. In addition, SGX-LKL reads the current timebetween context switches to reschedule blocked threads totrigger timer interrupts. Current SGX implementations, how-ever, do not offer a high-performance in-enclave time source,and SGX-LKL relies on time provided by the host.Instead of issuing expensive individual host calls, SGX-LKL uses the virtual dynamic shared object (vDSO) [107]mechanism of the Linux kernel. The kernel maps a smallshared library to the address space outside of the enclave.On each clock tick, the host kernel updates a shared memorylocation with the current time for various clock sources, whichare read from within the enclave when a time-related call ismade.For high precision, the vDSO mechanism requires the
RDTSCP instruction to adjust for the time passed since the lastvDSO update. In SGX version 1, this instruction is not per-mitted inside enclaves. The accuracy of clock_gettime() inSGX-LKL thus depends on the frequency of vDSO updates;SGX version 2 does not have this limitation.
Illegal instructions.
Applications may use the
RDTSC in-struction to read the timestamp counter or
CPUID to obtainCPU features. These instructions, however, are illegal insideSGX version 1 enclaves. SGX-LKL catches the resulting
SIGILL exception and emulates the instructions:
RDTSC is exe-cuted outside the enclave, and the result is forwarded via the forward_signal() call; for
CPUID , SGX-LKL caches all
CPUID information during enclave set-up. This eliminates the needfor an extra host call and also hides the
CPUID request by the6pplication.
Host disk blocks must have their confidentiality and integrityprotected. SGX-LKL uses the Linux device mapper subsys-tem [31], which maps virtual block devices, such as SGX-LKL’s virtio backend, to higher-level devices, and allows I/Odata to be transformed. File systems such as ext4 use a virtualblock device, and data is encrypted and integrity protectedtransparently before it reaches the underlying device.SGX-LKL uses different device mapper targets: (i) dm-crypt [33] provides full-disk encryption using AES in XTSmode with the sector number as an initialisation vector;(ii) dm-verity [35] offers volume-level read-only integrityprotection through a Merkle tree of disk block hashes, withthe root node stored in memory; and (iii) dm-integrity [34]provides key-based block-level read/write integrity protec-tion. dm-crypt and dm-integrity can be combined for fulldisk encryption with read/write integrity via AES-GCM. ForAES-XTS/AES-GCM, SGX-LKL uses hardware AES-NI in-structions.SGX-LKL combines the different targets to provide bothconfidentiality and integrity for block reads/writes, dependingon the security requirements of the application. For example,for an in-memory key-value store such as
Memcached [67], itis sufficient to protect the integrity of a read-only disk with dm-verity : the Memcached binary stored on disk must have itsintegrity protected, but no further sensitive data is stored onthe disk. If the application itself is confidential or other appli-cation data is written to disk, dm-crypt can be used with eitherof the integrity protection targets. SGX-LKL’s use of LKLalso allows it to support other device mapper based protectiontargets such as dm-x [17], which provides full volume-levelread/write integrity and replay protection. The use of differenttargets affects I/O performance, as we explore in §6.1.
SGX-LKL must guarantee the confidentiality and integrity ofall network data. Existing TEE runtime systems either requireapplications to have built-in support for network encryption [9,90, 101] or use TLS [7]. Some applications though do notsupport TLS (e.g. Tensorflow) or use other transport protocols(e.g. UDP-based streaming).Instead, SGX-LKL exploits its TCP/IP network stack in-side the enclave, which enables it to provide transparent low-level network encryption. All data received and sent via the net_read() and net_write() host calls is automatically en-crypted, authenticated and integrity-protected. To protect allnetwork traffic, SGX-LKL uses
Wireguard [112], a layer 3VPN protocol, currently proposed for inclusion in the Linuxkernel [63].SGX-LKL sets up Wireguard at initialisation time and ex-poses the VPN to the application through a network interface with its own IP address. An application binding to this IPaddress is only reachable by trusted nodes in the VPN. EachWireguard peer has a public/private key pair, which is boundto a VPN IP address and an endpoint, an (IP, port)-pair throughwhich the VPN is accessible. Wireguard uses the asymmetrickey pairs to establish ephemeral symmetric session keys toprotect messages using authenticated encryption, and noncesto prevent replay attacks. In contrast to TLS, which uses cer-tificates, Wireguard identifies parties through public keys. Itdoes not perform key distribution—SGX-LKL binds keys toenclave identities and supports provisioning of peers’ keys(see §4.4).
For the remaining calls in Table 2, SGX-LKL must ensure thatan adversary cannot learn confidential data or compromiseintegrity by providing invalid data. net_poll() . While the eventmask reveals if the enclave wantsto receive/send packets, this is already disclosed by the pres-ence of net_read and net_write calls. An adversary can re-turn a wrong eventmask : as a result, either the net_read callfails, which can be handled transparently, or an invalid packetis read that fails Wireguard’s integrity protection (see §4.2). forward_signal() . SGX-LKL must ensure that signals corre-spond to genuine events with valid signal descriptions; other-wise an adversary can cause an application signal handler toexecute with invalid signal data. For signals due to hardwareexceptions, SGX-LKL ensures that the address lies withinthe enclave range (e.g.
SIGSEGV ) or replaces the address withthe current instruction pointer for signals that refer to a fault-ing instruction (e.g.
SIGILL and
SIGFPE ). SGX-LKL can beconfigured to ignore user-controlled signals (e.g.
SIGINT ). time_read() . A challenge to integrity is that the time can-not be trusted. An adversary may return a timestamp that isnot monotonically increasing and thus causes an underflowwhen an application calculates a timespan. SGX-LKL there-fore checks for monotonicity for
CLOCK_MONOTONIC_* clocksources. Future TEE implementations may provide practicaltrustworthy time sources, which SGX-LKL could use.
SGX-LKL must execute securely in an untrusted and poten-tially malicious environment. For this, parties must (i) re-motely attest that they execute a trustworthy version of SGX-LKL in an SGX enclave; (ii) deploy applications securely, i.e.guaranteeing the confidentiality and integrity of applicationcode; and (iii) provision applications with secrets such ascryptographic keys and sensitive application configuration.The above requirements go beyond attestation in currentSGX SDKs [51,69], which assume that all application code iscompiled into the library, and a single measurement sufficesto verify integrity. They also do not protect application confi-dentiality, and leave it to applications to implement custommechanisms for secret provisioning, which is cumbersome.7 gx-lkl-run
Application binary
Libraries
Ext4 disk image
ApplicationBinary
LibrariesStaticdata run :/usr/bin/python args:/app/myapp.py } libsgxlkl.so Create enclave /load libsgxlkl.soCreate disk image u v y
Send encrypted disk image
Root hashEnc. key
Provide app config& secrets | Peer Pub. Keys
EnclaveHost (Cloud Provider) ext4 file system
Service ProviderTrusted Client wg genkey
Generate attestation key w Attestation key
Request launch x Disk IDAttestation key
Return enclave quote z QuoteEnclave Pub.
Key(signed)
Trusted Peers
App Config
Intel Attestation Service sgx-lkl-create./Dockerfile--encrypt --verity
Root hashEnc. keyPeer Pub. KeysApp Config { Verify Quote Decrypt & mount disk / load & run application
Fig. 2: Deployment workflow for SGX-LKL
SGX-LKL addresses these issues as part of three phases:(i) application provisioning , (ii) remote attestation and (iii) se-cret provisioning . Fig. 2 shows the deployment workflow, in-volving three parties: (i) a service provider (SP) that wants todeploy an application and has a trusted client. For a distributedapplication, this may involve deploying multiple trusted peers;(ii) an untrusted host controlled by a cloud provider (CP) pro-vides enclaves; and (iii) the
Intel Attestation Service (IAS),which allows the SP to verify an enclave measurement. (1) Application provisioning. In (cid:182) , the trusted client createsa disk image with the application binary and its dependencies,e.g. by exporting a Docker container [36]. SGX-LKL providesa sgx-lkl-create tool to simplify image creation. The tool isbased on cryptsetup [28], which configures dm-crypt for diskencryption/integrity protection. It outputs the encryption keyand the root hash of the Merkle tree for dm-verity protection.In (cid:183) , the disk image is sent to the CP. For attestation, theclient generates a Wireguard asymmetric key pair in (cid:184) . (2) Remote attestation. In (cid:185) , the public attestation key andthe image disk ID are sent to the cloud host. The host createsan enclave with libsgxlkl.so ( (cid:186) ). libsgxlkl.so boots LKL, setsup networking and generates its Wireguard key pair.Now the enclave can be attested: libsgxlkl.so creates areport with an enclave measurement. SGX allows customdata to be included with the report, which SGX-LKL usesto add the generated public key. The report is signed by a quoting enclave , and the quote, together with the enclave’spublic key, is returned to the SP ( (cid:187) ). In (cid:188) , the SP sends thequote to the IAS, which returns a verification report that ischecked. (3) Secret provisioning. After successful attestation, the SPestablishes a secure channel with the enclave through its pub-lic key. libsgxlkl.so accepts only one Wireguard peer so thatno other party can communicate with the enclave.In (cid:189) , the SP sends to the enclave: (i) the disk encryptionkey; (ii) the root hash; (iii) public keys of other trusted peers;and (iv) configuration data, including the executable path andits arguments. In (cid:190) , libsgxlkl.so mounts the disk and setsup device mapper for decryption/integrity. It adds the newWireguard peers, loads the application, and begins execution. As explained in §2.3, an adversary can compromise appli-cation confidentiality by observing: (i) frequencies of calls,i.e. their number and time intervals. For example, there maybe more host calls when an application processes data ondisk, thus revealing execution information; (ii) sequences ofcalls, i.e. the call order and the relationships between them.For example, an application may always execute reads be-fore writes, allowing an adversary to make inferences aboutthe execution; and (iii) parameters of calls. Although SGX-LKL encrypts data blocks (§4.1), the block offsets reveal filelocations. Observing the same offset discloses the data layout.The main idea for mitigation against these side-channels isto make the host call interface oblivious [40], i.e. the sequenceand frequency of calls as well as observable parameters be-come workload independent. We focus on the disk (§5.1) andnetwork (§ ?? ) I/O host calls and discuss other calls in §5.2. SGX-LKL exposes two disk I/O calls to the host: disk_read() and disk_write() . These read and write fixed-size encryptedblocks at the specified offset (see §4.1). To prevent these callsfrom revealing information, SGX-LKL employs several tech-niques: (i) regarding frequencies, SGX-LKL discretises theexecution of read/write calls into fixed time-interval rounds.In each round, it executes a fixed number of calls in a batch ,potentially adding indistinguishable dummy calls; (ii) regard-ing sequences, SGX-LKL makes the order of read/write calls deterministic per batch by issuing them in a predefined order,e.g. always executing reads before writes; and (iii) regardingparameters, SGX-LKL ensures that all call parameters appear random , obscuring patterns. SGX-LKL makes accesses obliv-ious by using seemingly random offset parameters. Repeatedaccesses to the same disk block appear indistinguishable.
Hiding disk I/O accesses. sys discretises the execution of allcalls into fixed interval batches. Every t time units, SGX-LKLperforms a single disk_read() followed by a disk_write() .When the LKL filesystem layer (see §3.2) issues a disk I/Ocall, instead of submitting it directly to the host, it is addedto a queue. SGX-LKL’s lthread scheduler checks if enoughtime has elapsed, and then issues the next call from the queue.If there are too many read/write calls in the queue, theremaining calls are delayed until the next batch. Conversely,if there are not enough read/write calls from the application,SGX-LKL issues dummy calls to random blocks in order topad that batch. This is done through dummy files in the filesystem and reading/writing blocks of those files.The timing parameter t must be tuned for good performancebecause applications issue system calls at different rates: if t is lower than the application’s call rate, SGX-LKL will issuemore batches with dummy calls; if t is too high, the appli-cation will make slow progress. In our experiments, we use t = . lgorithm 1: Oblivious ext4 file system shuffle
Data:
FDs: file descriptors of files to be shuffledLP: logical to physical block mapping for file system
Result:
LP_n: new logical to physical block mapping for file system for fd in FDs do // find file with largest size file_blk := num_blocks(fd); max_blk := max(max_blk, file_blk); num_shuff_blk := num_shuff_blk + file_blk; /* create donor files to be used for the swap */ free_blk := get_num_free_blocks_in_fs(); num_donors := free_blk / max_blk; donor_files := init_donors(num_donors); /* generate new random permutation of source blocks */ src_blks_perm := map_src_blk_to_num(FDs); src_blk_new_perm := FisherYatesShuffle(src_blk_perm); for id in (0 .. num_shuff_blk) do // get fd & block no. for id fd := fd_for_id(src_blk_new_perm, id); block_no := blk_no_for_id(src_blk_new_perm, id); donor = pick_rand_donor(num_donors); /* swap source block with donor block */ fetch_source_block(block_no); ioctl( EXT4_IOC_MOVE_EXT , fd, donors_files[donor], block_no); unlink_all_files(donor_files); Hiding disk I/O parameters.
The call parame-ters (see Tab. 2) still leak information: (i) the data in buf ; (ii) the disk offset ; and (iii) the buffer length len . Tohide buf , SGX-LKL adopts probabilistic encryption [41],which re-encrypts data before writing it (see §4.1). Thisprevents an adversary from discerning if written data is old ornew.To hide the offset , two traces of block accesses must beindistinguishable. File I/O calls from the application passthrough the file system layer, which may handle them from thein-enclave LKL page cache, or expose them to the host. Thefile system layer stores the mapping from logical to physicalblocks. To hide repeated block accesses, SGX-LKL must:(i) randomise the disk layout before application execution,hiding the initial block mapping; and (ii) shuffle blocks bychanging the mapping in a manner opaque to the adversary.Existing oblivious RAM (ORAM) [30, 40, 62, 92, 93, 109]constructions can be used to hide block accesses. LKL’s pagecache is well-suited to employ square root ORAM [40]—thepage cache is analogous to a “shelter”. As it is private, itprecludes the need to scan it in its entirety for each access andallows data to be accessed directly. After a block was addedto the page cache, subsequent accesses to the same block areserved directly, until the page is removed, or flushed to disk.If evicted blocks are requested again, the adversary wouldobserve this. Therefore, SGX-LKL must obliviously reshuffle blocks before continuing execution [40].An oblivious shuffle moves blocks around so that an adver-sary cannot correlate the blocks before and after the shuffle.SGX-LKL uses a k-oblivious shuffle [80]. This assumes that k of the n source blocks are in the private cache. The algorithmthen sequentially moves the n blocks from the source (eitherfrom the disk or from the cache) to the destination accordingto a randomly generated permutation of the blocks. Alg. 1 describes the shuffle algorithm. We assume a listof files that must be shuffled. To protect a single applicationworkload, this list can be obtained from a trace of openedfiles; otherwise all files on the disk can be considered. Wedivide the disk blocks into: (i) blocks that belong to files thatmust be shuffled, i.e. the source array of size n ; and (ii) adestination array of n unallocated blocks.First, the algorithm determines the size max _ blk of thelargest file to be shuffled (lines 1–4) and the number offree blocks (line 6). It uses this information to create donorfiles (line 8), each of size max _ blk , to fill up the free blocks.These donor files are used for swapping blocks with the sourcefiles. Next, the algorithm assigns an ordering to the source fileblocks that need to be shuffled (line 10), mapping the set ofblocks to natural numbers. It uses a Fisher-Yates shuffle [56]to produce a random permutation of the blocks (line 11). Thealgorithm then performs an iteration per source block to beshuffled (line 12). At each iteration, it determines the sourcefile descriptor and block number that maps to the current in-dex according to the new permutation (lines 13–14). If thesource block is already in the page cache, an unread sourceblock, if any, is accessed and brought into the cache (line 17).Finally, a random donor file is selected, and the sourceblock is swapped with the block from the donor file (line 18).The algorithm is implemented at the LKL ext4 layer becausethe shuffled blocks must be decrypted/encrypted by dm-crypt ,as discussed in §4.1. The ext4 layer also stores the block map-ping, which needs to be modified when blocks are swapped.SGX-LKL uses the
EXT4_IOC_MOVE_EXT ioctl() call for this,which exchanges the blocks belonging to two different files,while also updating the underlying metadata. Once all theblock have been swapped, the dummy files are unlinked, andthe corresponding blocks become unallocated again.Note that while the creation/unlinking of donor files isdeterministic, the locations of donor file blocks are randomand thus indistinguishable from regular file blocks. Accessesfor both types are intermixed and thus also indistinguishable.SGX-LKL exposes two network I/O calls to the host: net_read() and net_write() . These read and write a fixednumber of bytes from the network device (see §4.2). Tohide the network communication pattern, SGX-LKL usesfixed-size network packets and adds dummy packets to gen-erate a constant traffic rate between participating trustednodes [102, 106].To achieve this, SGX-LKL uses the set of Wireguard peersas the participating trusted nodes (see §4.2). For each peer,SGX-LKL creates a thread to send dummy packets with ran-dom payloads. These packets are probabilistically encryptedby Wireguard and indistinguishable from regular packets.To ensure a constant traffic rate, SGX-LKL uses Linux’traffic shaping capabilities [97]. It sets the queueing disci-pline ( qdisc ) for the the Wireguard interface to hierarchytoken bucket (HTB) [95], which supports rate limits for net-work connections. For each peer, SGX-LKL adds an HTB9lass filter for a per-peer rate limit. To optimise throughput ofregular packets, SGX-LKL only sends dummy packets whennecessary to reach the rate limit. It adds a
PRIO qdisc [96]below each HTB class, which prioritises packets with higherpriority, and sets dummy packets’ priority as lowest. SGX-LKL pads all network packets to the MTU size to obtainfixed-size packets.This approach has an overhead proportional to the numberof trusted nodes. To scale to more clients and to hide the setof potential recipients, traffic could alternatively be routedthrough a single trusted proxy [12, 18, 59]. caused by the application disclose information. Forexample, if an application executes an illegal instruction, theresulting signal is observable. SGX-LKL hides what instruc-tion caused the signal but cannot hide the exception (see §4.3).This is a limitation of SGX—future implementations mayhandle exceptions without host involvement.
Time.
The host kernel updates a vDSO memory region withthe current time (see §3.2). It is read by the enclave, which isobservable by an adversary. However, the SGX-LKL sched-uler accesses time frequently on each context switch, and anadversary cannot distinguish between accesses from SGX-LKL and the application, hiding application-specific accesspatterns.
To understand the performance impact of SGX-LKL’s designchoices, we evaluate it with real application workloads andmicro-benchmarks. All experiments use machines with IntelXeon E3-1280 4-core CPUs with 8 MB LLC and SGX 1support, 64 GB RAM and a 10-Gbps NIC. The machines runUbuntu Linux 18.04 (kernel 4.15.0-46) with SGX driver v2.5.The EPC size is 128 MB, and around 90 MB are availableto user applications. Since this limit is too low for some ofour workloads but will increase substantially in the futurewith new SGX implementations [54], we first evaluate SGX-LKL in software-only mode to ignore the overhead dueto EPC paging. After that, we consider SGX-LKL in hard-ware mode to validate how real-world SGX hardware impactsperformance. When using the oblivious host interface, werandomise the disk image and use a large page cache size. We evaluate the performance of SGX-LKL with two data-intensive workloads, TensorFlow [1] and PARSEC [10].We use
TensorFlow (TF) [1] for training and inference.The models are selected from TF’s benchmark suite [100],representing different type of networks, including small(ResNet-34), large (AlexNet, ResNet-101) and deep andlow-dimensional (ResNet-50). The input datasets are CI-FAR10 [57] and a subset of ImageNet [49]. In SGX-LKL, all SGX-LKL executes the same code in both software and hardware modes. ImageNet AlexNet
CIFAR-10 ResNet -34 ResNet -50 ResNet -101 T h r oughpu t ( I m age s / s e c ; l og sc a l e ) NativeSGX-LKL (sim)SGX-LKL (sim+enc/int)SGX-LKL (sim+obliv)
Fig. 3: Training throughput with TensorFlow ImageNet AlexNet
CIFAR-10 ResNet -34 ResNet -50 ResNet -101 T h r oughpu t ( I m age s / s e c ; l og sc a l e ) NativeSGX-LKL (sim)SGX-LKL (sim+enc/int)SGX-LKL (sim+obliv)
Fig. 4: Inference throughput with TensorFlow experiments are executed with and without disk encryptionand oblivious host calls. To avoid SGX paging effects, we runin software mode with an enclave size of 12.5 GB.In Fig. 3, we report the training throughput with SGX-LKLin different modes compared to native execution. SGX-LKLoutperforms native in software mode: the training throughputof ResNet-34, ResNet-50 and ResNet-101 is higher than na-tive by 28%, 33% and 32%, respectively. SGX-LKL achievesbetter performance due to LKL’s implementation of systemcalls as standard function calls. The results also show thatthe overhead of disk encryption and integrity checking isnegligible.With oblivious host calls with disk encryption, SGX-LKLshows a slow-down: the throughput for AlexNet decreases by24% (ImageNet) and 37% (CIFAR10) compared to softwaremode, or 14% and 21% compared to native execution. Theimpact of oblivious host calls on the ResNet models is less:SGX-LKL’s performance is almost equal to native. This is dueto the different I/O characteristics: AlexNet processes moreimages than ResNet per second, thus stressing the oblivioushost calls and requiring more shuffle operations.In Fig. 4, we also explore inference throughput with SGX-LKL in different modes compared to native. The results showthe same trend as training: the inference throughput is higher
120 1 2 3 4 T h r oughpu t ( I m age s / s e c ) WorkersNativeSGX-LKL (sim)SGX-LKL (sim+wireguard)SGX-LKL (sim+obliv)
Fig. 5: Distributed training throughput with TensorFlow
25 blackscholes bodytrack canneal fuidanimate streamcluster S l o w do w n ( X ) SGX-LKL (sim)SGX-LKL (sim+enc/int)SGX-LKL (sim+obliv)SGX-LKL (hw)SGX-LKL (hw+enc/int)SGX-LKL (hw+obliv)
Fig. 6: Execution overhead for PARSEC workloads than the native system by 22% for AlexNet with ImageNet,42% for AlexNet with CIFAR10, 29% for ResNet-34, 26% forResNet 50, and 20% for ResNet 101. The throughput is lowerthan native with oblivious host calls by 79% (AlexNet withImageNet), 15% (AlexNet with CIFAR-10), 53% (ResNet-101), 50% (ResNet-50), and 43% (ResNet-34). We concludethat CPU-intensive networks are less affected by the obliviouscall interface, while I/O-intensive networks are more affected.To evaluate performance for distributed applications, wedeploy TF with one parameter server and a varying numberof workers, using a batch size of 256.Fig. 5 shows SGX-LKL’s performance compared to native:without network encryption, SGX-LKL’s throughput is com-parable to native: with 4 workers, SGX-LKL’s throughputis 3% slower; with the Wireguard VPN, the throughput de-creases to 52% for 4 workers; and with oblivious network I/Ousing a constant rate of 200 Mbps per peer, the throughputreduces to 34%. We believe that this is an acceptable securityoverhead that can be compensated by scale out.To explore the performance of SGX-LKL with SGX hard-ware, we use five
PARSEC benchmarks [103], ranging fromcomputer vision algorithms to machine learning, with differ-ent working set sizes. We use four threads, and the input sizeis “simlarge”.Fig. 6 shows the execution of SGX-LKL with different se-curity guarantees. Without oblivious host calls, SGX-LKLin software mode is 1.05 × slower than native. The overheadfor streamcluster is 1.07 × ; fluidanimate has the same perfor-mance as native; for canneal and blackscholes , the slowdownis around 1.1 × . We note that, without EPC limits, SGX-LKL’s performance without oblivious calls is comparable tonative.In hardware mode, the overhead of workloads with thesmall to medium working set sizes is 1.3 × ( streamcluster ),1.4 × ( blackscholes ) and 1.7 × ( bodytrack ). With large work-ing sets, fluidanimate and canneal are slower by 4 × and 15 × ,respectively, which is due to the limited EPC size.With oblivious host calls, streamcluster shows similar over-heads in software (2.4 × ) and hardware modes (2.8 × ). Since streamcluster does not have input files, no shuffling is re-quired, and the overhead comes from the dummy calls; forbenchmarks with input files, e.g. blackscholes , the overheadin hardware mode is higher (3.6 × ) due to shuffling. For flu-idanimate and canneal , which have large memory footprints,
600 Native No
Enc FDE FDE+Int T h r oughpu t ( M B / s e c ) NativeSGX-LKL (sim)1.00x SGX-LKL (hw)0.90x 0.93x0.62x 0.68x0.45x
Fig. 7: Disk performance with encryption/integrity protection
256 512 1
K B K B K B K B K B K B K B
K B
K B T h r oughpu t ( G bp s ) Buffer size (bytes)NativeSGX-LKL (hw)SGX-LKL (enc) SGX-LKL (sim)SGX-LKL (sim+enc)
Fig. 8: Network I/O throughput for different buffer sizes the overhead is up to 2 × in software; in hardware mode, itrises to 10 × and 21 × , respectively, due to SGX paging. We evaluate disk I/O performance by measuring the sequen-tial read throughput for an uncached 1 GB file on SSD. Weconsider: (i) unencrypted; (ii) full disk encryption (
FDE ) via dm-crypt using AES-XTS; and (iii) FDE and integrity protec-tion via dm-crypt and dm-integrity using AES-GCM.Fig. 7 shows that SGX-LKL achieves near native perfor-mance without encryption or integrity protection, fully saturat-ing the SSD bandwidth of approx. 510 MB/sec. With
FDE , thethroughput decreases to 320 MB/sec (62%) of native through-put in hardware mode. Enabling integrity protection furtherreduces throughput to around 230 MB/sec (45%). This showsthe benefit of using x86-specific cryptographic instructions.
We evaluate network performance by measuring throughputfor different buffer sizes with iperf v3.1.3 [53].Fig. 8 shows that throughput increases with larger buffersizes: for 256 byte packets, SGX-LKL reaches about 0.7 × and0.9 × of native throughput in hardware and software modes,respectively; for small buffers, the throughput is client-bound.Native saturates the network bandwidth of about 9.4 Gbpswith 4 KB buffers. Here SGX-LKL reaches 5.1 Gbps (0.5 × )in hardware and 5.7 Gbps (0.6 × ) in software mode.Due to its support for TCP segmentation offloading, SGX-LKL performs better with larger buffers: with 64 KB buffers,it reaches a throughput of 8.4 Gbps (0.9 × ) and 8.8 Gbps(0.9 × ) for hardware and software modes, respectively.With Wireguard and 256 byte buffers, throughput for hard-ware and software mode is 0.44 Gbps and 0.54 Gbps, respec-11ively; with buffers larger than 4 KB, the throughput increasesto 2 Gbps, and decryption becomes the bottleneck. TEE runtime systems.
As mentioned in §2.3, Haven [9],SCONE [7], Graphene-SGX [101], and Panoply [90] are exist-ing TEE runtime systems, but they have wider host interfacesthan SGX-LKL and do not support an oblivious host interface.Ryoan [47] uses SGX enclaves to sandbox data processing.It makes the data flow independent from the content of theinput data but does not make the host interface oblivious.
ORAM with TEEs can conceal access patterns. Zero-Trace [87] combines SGX with ORAM to create obliviousmemory primitives. ObliDB [39] and Oblix [70] use PathORAM [92] with SGX to hide the access patterns of SQLqueries. In Raccoon [84], Path ORAM hides array accessesthat depend on secrets. By using ORAM, these approachescan hide access pattern but hosts can still mount Iago attackswhen returning data blocks to the enclave.
Oblivious file systems.
Obliviate [2] is an oblivious file sys-tem for SGX enclaves. It adapts an ORAM protocol to read-/write a data file. Privatefs [111] is an ORAM-based paral-lel oblivious file system, which suports concurrent accesses.Oladi [27] is a cloud-based key/value store that supports obliv-ious transactions while protecting access patterns from cloudproviders. Bittau et al. [12] introduce an oblivious architec-ture for monitoring client software behaviour while protectinguser privacy. In constrast to SGX-LKL, these approaches donot offer a file system abstraction with POSIX semantics.
Unobservable communication.
For message systems, hid-ing which users communicate is challenging. Anonymousnetworks such as Tor [32] are susceptible to traffic anal-ysis attacks [46, 72]. Dining Cryptographers (DC) net-works [19, 91, 113] provide stronger guarantees by broad-casting messages to all users, but incur a high overhead. Morerecently, Private Information Retrieval (PIR) [23, 26] and dif-ferential privacy techniques [8, 102, 106] add noise to hidemetadata. SGX-LKL can leverage this to blind communica-tions.
We described SGX-LKL, a TEE runtime system that is de-signed around a minimal and oblivious host interface. SGX-LKL executes Linux binaries by using the Linux kernel toprovide POSIX abstractions inside of SGX enclaves, and itshuffles disk blocks to hide access patterns from the host.Ignoring SGX paging effects, SGX-LKL’s performance over-head is low, even for complex applications such as Tensor-Flow.
References [1] Martín Abadi, Paul Barham, Jianmin Chen, ZhifengChen, Andy Davis, Jeffrey Dean, Matthieu Devin, San-jay Ghemawat, Geoffrey Irving, Michael Isard, Man- junath Kudlur, Josh Levenberg, Rajat Monga, SherryMoore, Derek Gordon Murray, Benoit Steiner, Paul A.Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke,Yuan Yu, and Xiaoqiang Zheng. Tensorflow: A sys-tem for large-scale machine learning. In , pages 265–283, 2016.[2] Adil Ahmad, Kyungtae Kim, Muhammad IhsanulhaqSarfaraz, and Byoungyoung Lee. OBLIVIATE: A DataOblivious Filesystem for Intel SGX. In
Symposiumon Network and Distributed System Security (NDSS) ,2018.[3] Moshaddique Al Ameen, Jingwei Liu, and KyungsupKwak. Security and Privacy Issues in Wireless Sen-sor Networks for Healthcare Applications.
Journal ofmedical systems , 36(1):93–101, 2012.[4] AMD. AMD Secure Encrypted Virtualization (SEV). https://developer.amd.com/sev/ . Last accessed:August 7, 2019.[5] Apriorit. Cloud Computing: A New Vector for CyberAttacks. . Last ac-cessed: August 7, 2019.[6] ARM. Technologies TrustZone for Cortex-A. . Last accessed: August 7, 2019.[7] Sergei Arnautov, Bohdan Trach, Franz Gregor, ThomasKnauth, Andre Martin, Christian Priebe, Joshua Lind,Divya Muthukumaran, Dan O’Keeffe, Mark L. Still-well, David Goltzsche, Dave Eyers, Rüdiger Kapitza,Peter R. Pietzuch, and Christof Fetzer. SCONE: SecureLinux Containers With Intel SGX. In , pages 689–703, 2016.[8] Michael Backes, Aniket Kate, Praveen Manoharan, Se-bastian Meiser, and Esfandiar Mohammadi. AnoA: AFramework for Analyzing Anonymous Communica-tion Protocols. In , pages 163–178, 2013.[9] Andrew Baumann, Marcus Peinado, and Galen Hunt.Shielding Applications from an Untrusted Cloud withHaven. In
Proceedings of the 11th USENIX Confer-ence on Operating Systems Design and Implementa-tion , OSDI, pages 267–283, 2014.[10] Christian Bienia, Sanjeev Kumar, Jaswinder Pal Singh,and Kai Li. The PARSEC Benchmark Suite: Character-ization and Architectural Implications. In
Proceedings f the 17th International Conference on Parallel Archi-tectures and Compilation Techniques , October 2008.[11] Andrea Biondo, Mauro Conti, Lucas Davi, TommasoFrassetto, and Ahmad-Reza Sadeghi. The guard’sdilemma: Efficient code-reuse attacks against intelSGX. In , pages 1213–1227, 2018.[12] Andrea Bittau, Úlfar Erlingsson, Petros Maniatis,Ilya Mironov, Ananth Raghunathan, David Lie, MitchRudominer, Ushasree Kode, Julien Tinnes, and Bern-hard Seefeld. Prochlo: Strong Privacy for Analytics inthe Crowd. In Proceedings of the 26th Symposium onOperating Systems Principles , pages 441–459, 2017.[13] Daniel P Bovet and Marco Cesati. Understanding theLinux Kernel. OReilly & Associates.
Inc. October2000 , 2002.[14] Ferdinand Brasser, Srdjan Capkun, AlexandraDmitrienko, Tommaso Frassetto, Kari Kostiainen,Urs Müller, and Ahmad-Reza Sadeghi. DR.SGX:Hardening SGX Enclaves against Cache Attacks withData Location Randomization.
CoRR , abs/1709.09917,2017.[15] Ferdinand Brasser, Urs Müller, Alexandra Dmitrienko,Kari Kostiainen, Srdjan Capkun, and Ahmad-RezaSadeghi. Software grand exposure: SGX cache attacksare practical. In , 2017.[16] David Cash, Paul Grubbs, Jason Perry, and ThomasRistenpart. Leakage-Abuse Attacks Against Search-able Encryption. In
Proceedings of the 22nd ACMSIGSAC Conference on Computer and Communica-tions Security , pages 668–679, 2015.[17] Anrin Chakraborti, Bhushan Jain, Jan Kasiak, TaoZhang, Donald Porter, and Radu Sion. Dm-x: Pro-tecting Volume-level Integrity for Cloud Volumes andLocal Block Devices. In
Proceedings of the 8th Asia-Pacific Workshop on Systems , APSys, pages 16:1–16:7,2017.[18] David Chaum. The dining cryptographers problem:Unconditional sender and recipient untraceability.
J.Cryptology , 1(1):65–75, 1988.[19] David Chaum. The Dining Cryptographers Problem:Unconditional Sender and Recipient Untraceability.
Journal of cryptology , 1(1):65–75, 1988.[20] Stephen Checkoway and Hovav Shacham. Iago At-tacks: Why the System Call API is a Bad UntrustedRPC Interface. In
Proceedings of the Eighteenth In-ternational Conference on Architectural Support for Programming Languages and Operating Systems , AS-PLOS, pages 253–264, 2013.[21] Ramnath K Chellappa and Paul A Pavlou. Perceived In-formation Security, Financial Liability and ConsumerTrust in Electronic Commerce Transactions.
LogisticsInformation Management , 15(5/6):358–368, 2002.[22] Guoxing Chen, Wenhao Wang, Tianyu Chen, SanchuanChen, Yinqian Zhang, XiaoFeng Wang, Ten-HwangLai, and Dongdai Lin. Racing in Hyperspace: ClosingHyper-Threading Side Channels on SGX with Con-trived Data Races. In , pages 178–194, 2018.[23] Benny Chor, Oded Goldreich, Eyal Kushilevitz, andMadhu Sudan. Private Information Retrieval. In
Pro-ceedings of IEEE 36th Annual Foundations of Com-puter Science , pages 41–50, 1995.[24] Cloud Security Alliance. Top Threats to CloudComputing Plus: Industry Insights. https://cloudsecurityalliance.org/download/top-threats-cloud-computing-plus-industry-insights . Last accessed: August 7, 2019.[25] Intel Corp. Software Guard Extensions Pro-gramming Reference, Ref. 329298-002US. https://software.intel.com/sites/default/files/managed/48/88/329298-002.pdf , 2014.[26] Henry Corrigan-Gibbs, Dan Boneh, and David Maz-ières. Riposte: An Anonymous Messaging SystemHandling Millions of Users. In , pages 321–338, 2015.[27] Natacha Crooks, Matthew Burke, Ethan Cecchetti,Sitar Harel, Rachit Agarwal, and Lorenzo Alvisi.Obladi: Oblivious Serializable Transactions in theCloud. In , pages727–743, 2018.[28] cryptsetup. https://gitlab.com/cryptsetup/cryptsetup . Last accessed: August 16, 2019.[29] Shujie Cui, Sana Belguith, Ming Zhang, Muham-mad Rizwan Asghar, and Giovanni Russello. Pre-serving Access Pattern Privacy in SGX-Assisted En-crypted Search. In ,pages 1–9. IEEE, 2018.[30] Srinivas Devadas, Marten van Dijk, Christopher WFletcher, Ling Ren, Elaine Shi, and Daniel Wichs.Onion ORAM: A Constant Bandwidth Blowup Obliv-ious RAM. In
Theory of Cryptography Conference ,pages 145–174. Springer, 2016.1331] Linux Kernel device mapper framework. . Last accessed: August 7, 2019.[32] Roger Dingledine, Nick Mathewson, and Paul Syver-son. Tor: The Second-Generation Onion Router.Technical report, Naval Research Lab Washington DC,2004.[33] DMCrypt. https://gitlab.com/cryptsetup/cryptsetup/wikis/DMCrypt . Last accessed: August16, 2019.[34] DMIntegrity. https://gitlab.com/cryptsetup/cryptsetup/wikis/DMIntegrity . Last accessed:August 16, 2019.[35] DMVerity. https://gitlab.com/cryptsetup/cryptsetup/wikis/DMVerity . Last accessed: Au-gust 16, 2019.[36] Docker. Enterprise Container Platform for High-Velocity Innovation. .Last accessed: August 16, 2019.[37] Adrian J Duncan, Sadie Creese, and Michael Gold-smith. Insider Attacks in Cloud Computing. In , pages857–862, 2012.[38] Jan-Erik Ekberg, Kari Kostiainen, and N. Asokan.Trusted execution environments on mobile devices. In , pages 1497–1498, 2013.[39] Saba Eskandarian and Matei Zaharia. ObliDB: Oblivi-ous Query Processing Using Hardware Enclaves. arXivpreprint arXiv:1710.00458 , 2017.[40] Oded Goldreich and Rafail Ostrovsky. Software pro-tection and simulation on oblivious rams.
J. ACM ,43(3):431–473, 1996.[41] Shafi Goldwasser and Silvio Micali. Probabilistic En-cryption.
Journal of computer and system sciences ,28(2):270–299, 1984.[42] Johannes Götzfried, Moritz Eckert, Sebastian Schinzel,and Tilo Müller. Cache Attacks on Intel SGX. In
Pro-ceedings of the 10th European Workshop on SystemsSecurity, EUROSEC , pages 2:1–2:6, 2017.[43] Graphene Library OS with Intel Registered SGX Sup-port. https://github.com/oscarlab/graphene .Last accessed: August 7, 2019. [44] Nils Gruschka and Meiko Jensen. Attack Surfaces:A Taxonomy for Attacks on Cloud Services. In , pages 276–279, 2010.[45] Haryadi S Gunawi, Mingzhe Hao, Tanakorn Leesa-tapornwongsa, Tiratat Patana-anake, Thanh Do, Jef-fry Adityatama, Kurnia J Eliazar, Agung Laksono, Jef-frey F Lukman, Vincentius Martin, et al. What BugsLive in the Cloud? A Study of 3000+ Issues in CloudSystems. In
Proceedings of the ACM Symposium onCloud Computing , pages 1–14, 2014.[46] Nicholas Hopper, Eugene Y Vasserman, and Eric Chan-Tin. How Much Anonymity Does Network LatencyLeak?
ACM Transactions on Information and SystemSecurity (TISSEC) , 13(2):13, 2010.[47] Tyler Hunt, Zhiting Zhu, Yuanzhong Xu, Simon Pe-ter, and Emmett Witchel. Ryoan: A Distributed Sand-box for Untrusted Computation on Secret Data.
ACMTransactions on Computer Systems (TOCS) , 35(4):13,2018.[48] IBM. IBM Cloud. .Last accessed: August 7, 2019.[49] imagenet. http://download.tensorflow.org/example_images/flower_photos.tgz . Last ac-cessed: August 7, 2019.[50] Intel. Intel Software Guard Extensions. https://software.intel.com/en-us/sgx . Last accessed:August 7, 2019.[51] Intel. Intel(R) Software Guard Extensions for Linux*OS. https://github.com/intel/linux-sgx . Lastaccessed: August 16, 2019.[52] Intel. Resources and Response to Side Channel L1TF. . Lastaccessed: August 14, 2019.[53] IPerf3. https://iperf.fr . Last accessed: August 7,2019.[54] Simon Johnson. Keynote: Scaling Towards Confi-dential Computing. https://systex.ibr.cs.tu-bs.de/systex19/slides/systex19-keynote-simon.pdf .[55] Dan J Kim, Donald L Ferrin, and H Raghav Rao.A Trust-Based Consumer Decision-Making Model inElectronic Commerce: The Role of Trust, PerceivedRisk, and Their Antecedents.
Decision support systems ,44(2):544–564, 2008.1456] Donald Ervin Knuth.
The art of computer program-ming, , Volume III, 2nd Edition . Addison-Wesley, 1998.[57] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.The CIFAR-10 dataset. . Last accessed: August 16,2019.[58] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hin-ton. ImageNet Classification with Deep ConvolutionalNeural Networks.
Commun. ACM , 60(6):84–90, 2017.[59] Stevens Le Blond, David Choffnes, Wenxuan Zhou,Peter Druschel, Hitesh Ballani, and Paul Francis. To-wards Efficient Traffic-Analysis Resistant AnonymityNetworks.
ACM SIGCOMM Computer Communica-tion Review , 43(4):303–314, 2013.[60] Dayeol Lee, David Kohlbrenner, Shweta Shinde, DawnSong, and Krste Asanovi´c. Keystone: A Framework forArchitecting TEEs. arXiv preprint arXiv:1907.10119 ,2019.[61] Jae-Hyuk Lee, Jin Soo Jang, Yeongjin Jang, NohyunKwak, Yeseul Choi, Changho Choi, Taesoo Kim, Mar-cus Peinado, and Brent ByungHoon Kang. Hackingin Darkness: Return-oriented Programming against Se-cure Enclaves. In , pages 523–539, 2017.[62] Liang Liu, Rujia Wang, Youtao Zhang, and Jun Yang.H-ORAM: A Cacheable ORAM Interface for EfficientI/O Accesses. In
Proceedings of the 56th Annual De-sign Automation Conference 2019 , page 33, 2019.[63] WireGuard: Secure Network Tunnel. https://lkml.org/lkml/2019/3/22/95 . Last accessed: August 16,2019.[64] LSDS. spectre-attack-sgx. https://github.com/lsds/spectre-attack-sgx . Last accessed: August7, 2019.[65] lthread. https://github.com/halayli/lthread .Last accessed: August 16, 2019.[66] McAfee. 5 Devious Instances of Insider Threat inthe Cloud. . Last accessed: August 7,2019.[67] Memcached. What is Memcached? https://memcached.org/ . Last accessed: August 16, 2019.[68] Microsoft. Azure Confidential Comput-ing. https://azure.microsoft.com/en-gb/blog/azure-confidential-computing . Lastaccessed: August 7, 2019. [69] Microsoft. Open Enclave SDK. https://openenclave.io/sdk/ . Last accessed: August 16,2019.[70] Pratyush Mishra, Rishabh Poddar, Jerry Chen, Alessan-dro Chiesa, and Raluca Ada Popa. Oblix: An EfficientOblivious Search Index. In , pages 279–296, 2018.[71] Ahmad Moghimi, Gorka Irazoqui, and Thomas Eisen-barth. CacheZoom: How SGX Amplifies the Powerof Cache Attacks. In
Cryptographic Hardware andEmbedded Systems - CHES 2017 - 19th InternationalConference , pages 69–90. Springer, 2017.[72] Steven J Murdoch and George Danezis. Low-CostTraffic Analysis of Tor. In , pages 183–195, 2005.[73] Musl. . Last ac-cessed: August 7, 2019.[74] NetApp. What Is a Lift and Shift Cloud Migra-tion? https://cloud.netapp.com/blog/what-is-a-lift-and-shift-cloud-migration . Last ac-cessed: August 14, 2019.[75] Oleksii Oleksenko, Bohdan Trach, Robert Krahn, MarkSilberstein, and Christof Fetzer. Varys: Protecting SGXEnclaves from Practical Side-Channel Attacks. In ,pages 227–240, 2018.[76] Nuria Oliver and Fernando Flores-Mangas.HealthGear: A Real-Time Wearable System forMonitoring and Analyzing Physiological Signals. In
International Workshop on Wearable and ImplantableBody Sensor Networks (BSN) , pages 4–pp. IEEE,2006.[77] OP-TEE. Open Portable Trusted Execution Environ-ment. . Last accessed:August 7, 2019.[78] Dag Arne Osvik, Adi Shamir, and Eran Tromer. CacheAttacks and Countermeasures: The Case of AES. In
Cryptographers’ Track at the RSA Conference , pages1–20. Springer, 2006.[79] Panoply Source Code. https://github.com/shwetasshinde24/Panoply . Last accessed: August7, 2019.[80] Sarvar Patel, Giuseppe Persiano, and Kevin Yeo.CacheShuffle: A Family of Oblivious Shuffles. In , pages 161:1–161:13, 2018.1581] Donald E Porter, Silas Boyd-Wickizer, Jon Howell,Reuben Olinsky, and Galen C Hunt. Rethinking theLibrary OS From the Top Down. In
ACM SIGARCHComputer Architecture News , volume 39, pages 291–304, 2011.[82] GNU project. The GNU C Library (glibc). . Last accessed: Au-gust 7, 2019.[83] Octavian Purdila, Lucian Adrian Grijincu, and NicolaeTapus. LKL: The linux kernel library. In , pages 328–333, 2010.[84] Ashay Rane, Calvin Lin, and Mohit Tiwari. Rac-coon: Closing Digital Side-Channels Through Obfus-cated Execution. In , pages 431–446, 2015.[85] Read(2) Linux Programmer’s Manual. http://man7.org/linux/man-pages/man2/read.2.html . Lastaccessed: August 14, 2019.[86] Mohamed Sabt, Mohammed Achemlal, and Abdelmad-jid Bouabdallah. Trusted Execution Environment:What It Is, and What It Is Not. In , pages 57–64, 2015.[87] Sajin Sasy, Sergey Gorbunov, and Christopher Fletcher.ZeroTrace: Oblivious Memory Primitives From IntelSGX. In
Symposium on Network and Distributed Sys-tem Security (NDSS) , 2018.[88] Michael Schwarz, Moritz Lipp, Daniel Moghimi,Jo Van Bulck, Julian Stecklina, Thomas Prescher, andDaniel Gruss. ZombieLoad: Cross-Privilege-BoundaryData Sampling. arXiv preprint arXiv:1905.05726 ,2019.[89] HEX-Five Security. Introducing MultiZone™ SecureIoT Stack - the first Secure IoT Stack for RISC-V . https://hex-five.com/ . Last accessed: August 7,2019.[90] Shweta Shinde, DL Tien, Shruti Tople, and PrateekSaxena. Panoply: Low-TCB Linux Applications WithSGX Enclaves. In
Proceedings of the Annual Networkand Distributed System Security Symposium (NDSS) ,page 12, 2017.[91] Emin Gün Sirer, Sharad Goel, Mark Robson, andDoˇgan Engin. Eluding Carnivores: File Sharing WithStrong Anonymity. In
Proceedings of the 11th Work-shop on ACM SIGOPS European Workshop , page 19,2004.[92] Emil Stefanov, Marten van Dijk, Elaine Shi, Christo-pher W. Fletcher, Ling Ren, Xiangyao Yu, and Srinivas Devadas. Path ORAM: an extremely simple obliviousRAM protocol. In , pages299–310, 2013.[93] Karin Sumongkayothin, Steven Gordon, AtsukoMiyaji, Chunhua Su, and Komwut Wipusitwarakun.Recursive M-ORAM: A Matrix ORAM for ClientsWith Constrained Storage Space. In
International Con-ference on Applications and Techniques in InformationSecurity , pages 130–141. Springer, 2016.[94] Richard Ta-Min, Lionel Litty, and David Lie. SplittingInterfaces: Making Trust Between Applications andOperating Systems Configurable. In ,pages 279–292, 2006.[95] tc-htb manpage. https://linux.die.net/man/8/tc-htb . Last accessed: January 11, 2020.[96] tc-htb manpage. https://linux.die.net/man/8/tc-prio . Last accessed: January 11, 2020.[97] tc manpage. http://man7.org/linux/man-pages/man8/tc.8.html . Last accessed: January 11, 2020.[98] Tech Republic. Dropbox and Box leak files in securitythrough obscurity nightmare. . Last accessed: August7, 2019.[99] TensorFlow. . Lastaccessed: August 7, 2019.[100] TensorFlow Benchmarks. https://github.com/tensorflow/benchmarks . Last accessed: August 7,2019.[101] Chia-Che Tsai, Donald E. Porter, and Mona Vij.Graphene-SGX: A Practical Library OS for Unmod-ified Applications on SGX. In
Proceedings of theUSENIX Annual Technical Conference (ATC) , page 8,2017.[102] Nirvan Tyagi, Yossi Gilad, Derek Leung, Matei Za-haria, and Nickolai Zeldovich. Stadium: A DistributedMetadata-Private Messaging System. In
Proceedingsof the 26th Symposium on Operating Systems Princi-ples , pages 423–440, 2017.[103] Princeton University. PARSEC. https://parsec.cs.princeton.edu/ . Last accessed: August 7, 2019.[104] Jo Van Bulck, Marina Minkin, Ofir Weisse, DanielGenkin, Baris Kasikci, Frank Piessens, Mark Silber-stein, Thomas F Wenisch, Yuval Yarom, and RaoulStrackx. Foreshadow: Extracting the Keys to the IntelSGX Kingdom With Transient Out-Of-Order Execu-tion. In , pages 991–1008, 2018.16105] Jo Van Bulck, David Oswald, Eduard Marin, AbdullaAldoseri, Flavio D. Garcia, and Frank Piessens. A Taleof Two Worlds: Assessing the Vulnerability of EnclaveShielding Runtimes. In
Proceedings of the 2019 ACMSIGSAC Conference on Computer and Communica-tions Security , CCS, page 1741–1758, 2019.[106] Jelle Van Den Hooff, David Lazar, Matei Zaharia, andNickolai Zeldovich. Vuvuzela: Scalable Private Mes-saging Resistant to Traffic Analysis. In
Proceedings ofthe 25th Symposium on Operating Systems Principles ,pages 137–152. ACM, 2015.[107] vDSO manpage. http://man7.org/linux/man-pages/man7/vdso.7.html . Last accessed: August7, 2019.[108] Jinwen Wang, Yueqiang Cheng, Qi Li, and Yong Jiang.Interface-Based Side Channel Attack Against IntelSGX. arXiv preprint arXiv:1811.05378 , 2018.[109] Xiao Wang, Hubert Chan, and Elaine Shi. CircuitORAM: On Tightness of the Goldreich-OstrovskyLower Bound. In
Proceedings of the 22nd ACMSIGSAC Conference on Computer and Communica-tions Security , pages 850–861, 2015.[110] Nico Weichbrodt, Anil Kurmus, Peter Pietzuch, andRüdiger Kapitza. AsyncShock: Exploiting Synchro- nisation Bugs in Intel SGX Enclaves. In
EuropeanSymposium on Research in Computer Security , pages440–457. Springer, 2016.[111] Peter Williams, Radu Sion, and Alin Tomescu. Pri-vatefs: A Parallel Oblivious File System. In
Proceed-ings of the 2012 ACM Conference on Computer andCommunications Security , pages 977–988, 2012.[112] WireGuard. . Lastaccessed: August 7, 2019.[113] David Isaac Wolinsky, Henry Corrigan-Gibbs, BryanFord, and Aaron Johnson. Dissent in Numbers: Mak-ing Strong Anonymity Scale. In
Presented as Part ofthe 10th USENIX Symposium on Operating SystemsDesign and Implementation (OSDI) , pages 179–182,2012.[114] Yuanzhong Xu, Weidong Cui, and Marcus Peinado.Controlled-Channel Attacks: Deterministic Side Chan-nels for Untrusted Operating Systems. In
Securityand Privacy (SP), 2015 IEEE Symposium On , pages640–656, 2015.[115] Rui Zhang and Ling Liu. Security Models and Require-ments for Healthcare Application Clouds. In2010IEEE 3rd International Conference on Cloud Comput-ing