Restoring Uniqueness in MicroVM Snapshots
Marc Brooker, Adrian Costin Catangiu, Mike Danilov, Alexander Graf, Colm MacCarthaigh, Andrei Sandu
RRestoring Uniqueness in MicroVM Snapshots
Marc Brooker
Amazon Web Services
Adrian Costin Catangiu
Amazon Web Services
Mike Danilov
Amazon Web Services
Alexander Graf
Amazon Web Services
Colm MacCarthaigh
Amazon Web Services
Andrei Sandu
Amazon Web Services
Abstract
Code initialization—the step of loading code, executing static code, filling caches, and forming re-used connections—tendsto dominate cold-start time in serverless compute systemssuch as AWS Lambda. Post-initialization memory snapshots,cloned and restored on start, have emerged as a viable solutionto this problem, with incremental snapshot and fast restoresupport in VMMs like Firecracker.Saving memory introduces the challenge of managinghigh-value memory contents, such as cryptographic secrets.Cloning introduces the challenge of restoring the uniquenessof the VMs, to allow them to do unique things like generateUUIDs, secrets, and nonces. This paper examines solutionsto these problems in the every microsecond counts context ofserverless cold-start, and discusses the state-of-the-art of avail-able solutions. We present two new interfaces aimed at solvingthis problem—
MADV_WIPEONSUSPEND and VmGenId—andcompare them to alternative solutions.
When AWS Lambda receives a request to invoke a server-less function, the system attempts to match the request to analready-running microVM containing a copy of the functioncode. At scale, this warm start succeeds with high probability–both because of the averaging effects of scale, and becauseserverless systems attempt to predict the number of incomingrequests using techniques such as reinforcement learning [6],LSTMs [16], and ARIMA models [26]. When warm startdoes not succeed, the system falls back on cold start , whichinvolves starting a Firecracker [5] microVM (or taking onefrom a pool), loading the function code, and initializing thecode to get it ready to handle the request. The last step ofthe the cold start, initialization, is typically the one that takeslongest. Du et al [11] found this initialization step to take be-tween 25% and 95% of total start-up time, mirroring our ownexperiences. As an additional challenge for cloud providers,this initialization step is almost entirely within the customer’s
MicroVM bootDownloadInvokeInitializeInvokeDeath… ~200ms20ms to60s“cold start”<10ms“Warm start”
Figure 1: Time line of the life cycle of a microVM in AWSLambdacontrol, leaving little opportunity to optimize or improve it onbehalf of the customer (beyond simply allocating additionalresources to its execution).Figure 1 shows a typical time line of the life cycle of amicroVM in Lambda. The microVM boot, booting a minimalLinux kernel in Firecracker, with a minimal set of userspacedaemons, and network setup, takes approximately 200ms.This is hidden from the customer through the use of a smallpool of pre-booted microVMs. The cold start portion, includ-ing downloading the function code into the microVM andinitialization, frequently takes up to 60s but can be as littleas 20ms for small static binaries. The download is typicallyshort: at 25Gb/s, downloading Lambda’s maximum packagesize of 250MB takes only 80ms. Finally, each invoke expe-riences a warm start delay caused by routing, typically lessthan 10ms. The same microVM is then re-used, for futureinvokes of the same version of the same function, to amortizethe cold start costs.The time and resources taken by initialization dependstrongly on the customer’s choice of language and frame-work, and on additional work the customer has chosen to doat process start time. Of the popular languages, Java typically1 a r X i v : . [ c s . CR ] F e b icroVM bootDownloadSnapshotInitializeRestore 200ms to60s(Hidden)Restore RestoreInvoke Invoke Invoke Figure 2: Time line of the life cycle of a microVM in AWSLambda with snapshot and cloneexhibits the longest cold-start times. This is driven by threemain factors: the start-up time of the JVM itself, the decom-pression and loading of class code, and the execution of static code in the loaded classes. It is typically the last of thesewhich drives long cold-start times, because Java allows forthe execution of effectively arbitrary code at static initializa-tion time. While Java code initialization and static executionis multithreaded, the specification [13] limits parallelism bydefining a strict recursive order on class initialization. Thisproblem is not only limited to Java. The same outcome is com-mon, but less ubiquitous, in workloads written in Javascript,Python, and C
Snapshotting and clone-on-restore introduce several system-level challenges. First, by their nature snapshots turn datain RAM into data on storage. This data can include crypto-graphic secrets and other high-value tokens which the pro-grammer wished to control carefully, irrespective of the secu-rity properties of the snapshot storage and distribution layer.Second, cloning duplicates memory state, which is a challengefor applications which want to generate unique data such as re-quest IDs, UUIDs, and cryptographic nonces. Third, cloning isnot compatible with existing session-layer network protocolssuch as TCP and TLS, which assume that the client is a singleunique entity with a single identity. Re-establishing these con-nections can add significantly to restore latency [17, 18], andcan even cause correctness issues in some protocols (a type ofreplay attack). This paper focuses on the second challenge—uniqueness—but some of the solutions we analyze also helpaddress the challenges posed by high-value secrets and net-work protocols.
Most modern distributed systems depend on the ability ofnodes to generate unique or random values. RFC4122 [20]version 4 UUIDs are widely used as unique database keys, andas request IDs used for log correlation and distributed tracing.In this context, duplicate IDs will lead to correctness issues,logical data corruption, incorrect traces, and confusing ormisleading logs. UUIDs are also used as tokens for API idem-potency, where multiple calls with the same token behave as asingle call. Here, duplicate IDs can lead to availability or cor-rectness issues. Many common distributed systems protocols,including consensus protocols like Paxos [19], and orderingprotocols like vector clocks, rely on the fact that participantscan uniquely identify themselves. Cloning without changingidentity could be seen as an unintentional Sybil [10] attack,which most distributed systems protocols don’t tolerate.Jitter, the intentional addition of pseudorandomness to val-ues like retry intervals, sampling intervals, timeouts and re-source limits are widely used to improve the resilience ofdistributed systems. For example, adding jitter to the retryintervals used by clients of an overloaded system reduce theclustering of retries in time, helping systems recover fasterfrom transient overload. Adding jitter to periodic jobs, like cron , helps avoid seasonalities like start-of-day and top-of-hour spikes. Systems that aren’t unique will tend to cluster,and clustering leads to correlated load, which leads to lowersystem utilization.Cryptography is the most critical application of uniquedata. Any predictability in the data used to generate crypto-graphic keys—whether long-lived keys for applications likestorage encryption or ephemeral keys for protocols like TLS—fundamentally compromise the confidentiality and authenti-2ation properties offered by cryptography. A potentially less-obvious vulnerability relates to initialization vectors (IVs)and nonces used in common block cipher modes like countermode (CTR) and Galois counter mode (GCM). Using thesame IV with the same key multiple times breaks the securityof these modes. In many protocols, the IV is translated in theclear, making it easy for an attacker to tell if an IV has beenre-used. Dworkin in NIST SP800-38D [12] says:The probability that the authenticated encryptionfunction ever will be invoked with the same IV andthe same key on two (or more) distinct sets of inputdata shall be no greater than 2 − .. . .In practice, this requirement is almost as importantas the secrecy of the key.These properties—the need for unique IVs and keyswith both low predictability and very small probabilityduplication—require that even cloned MicroVMs have ac-cess to high-quality entropy, and the means to use it. Insome cases, such as AES-GCM, these values need only beunique and not random, so access to a unique per-VM identi-fier is sufficient. Solutions to the entropy problem are well-understood, and widely deployed in virtualized environments.Typically, entropy is injected into the guest, where it can beaccessed using devices (like /dev/urandom ) or system calls(like getrandom ). Hardware RNGs, such as Intel’s RDSEED and
RDRAND [2] (itself the output of a DRBG seeded by ahardware RNG), can also provide high-quality entropy insidecloned MicroVMs. We aren’t the first to notice these kindsof cryptographic vulnerabilities. For example, Ristenpart andYilek [22] describe successful attacks against TLS 1.0 relyingon the-reuse of VM snapshots.Many applications don’t obtain their entropy directly fromthe kernel or hardware, and instead deploy userspace pseudo-random number generators (PRNGs) or deterministic randombit generators (DRBGs). Widely-deployed cryptographically-secure PRNGs include NIST’s CTR_DRBG [7], Java’s
SecureRandom , and Javascript’s getRandomValues . Most ofthese PRNGs are deterministic algorithms that get their seedsfrom the kernel or hardware RNGs, either at startup or peri-odically, and stretch them into an unpredictable sequence ofrandom bits.This presents a challenge to the serverless computeprovider: even if they follow best practices such as reseed-ing kernel randomness, and customers follow best practiceslike using a cryptographically secure PRNG (CSPRNG), thecombination is not secure. For platforms like AWS Lambdawhich allow customers to run arbitrary code, even arbitrarybinaries, addressing this challenge is not trivial.
Solving the uniqueness problem strongly enough for crypto-graphic purposes requires a mechanism which can determin-istically reseed userspace PRNGs with new entropy at restoretime. This mechanism must also support the high-throughputand low-latency use-cases that led programmers to pick auserspace PRNG in the first place (so simply reverting to getrandom is not acceptable); be usable by both applicationcode and libraries; allow transparent retrofitting behind exist-ing popular PRNG interfaces without changing applicationcode; it must be efficient, especially on restore; and be simpleenough for wide adoption.Efficiency is a particularly important concern. The defaultPRNGs in many languages, including C, Java, and Python,are not cryptographically secure, a choice typically drivenby performance. This frequently leads to programmers usingthese weak random number generators for cryptographic pur-poses. Over 350 CVEs have been allocated in 2020 alone forsecurity issues introduced by this class of bugs. Reducing theefficiency of CSPRNGs further would increase adoption ofnon-CS PRNGs, leading to more bugs and insecure cryptog-raphy implementations.
MADV_WIPEONFORK
A similar problem with userspace PRNGs is introduced by fork , as processes memory is duplicated in both the parentand child processes. A new madvise flag,
MADV_WIPEONFORK ,was introduced in Linux 4.14 in 2017. Memory pages marked
MADV_WIPEONFORK are set to all zeros in the child processafter the call to fork , clone and related calls. PRNGs whichmark their internal state this way, or put a guard variable in apage marked this way, can deterministically detect they havebeen forked, and reseed from kernel or hardware randomness.This does not require additional system calls, and adds onlythe overhead of a single memory access per call to the randomnumber generation library. BSDs support a similar approach,using the MAP_INHERIT_ZERO flag to the minherit systemcall.The same interface is useful in other ways too. A processwhich handles cryptographic keys or other high-value materialcan mark them as
MADV_WIPEONFORK to ensure that they arenot needlessly copied into child processes. It can also becombined with mlock and
MADV_DONTDUMP to ensure thatsecrets aren’t saved onto disk.
MADV_WIPEONFORK is superficially similar to pthread_-atfork , but is more robust because it is not thread-specific,and works across all ways to clone a process. pthread_-atfork can be bypassed, intentionally or unintentionally, byusing the clone syscall directly, and by some other inter-faces like posix_spawn . This makes pthread_atfork in-sufficiently robust for use in widely-used libraries.3 .2 Suspend and
MADV_WIPEONSUSPEND
By analogy to
MADV_WIPEONFORK , we have contributed anew
MADV_WIPEONSUSPEND flag to the Linux kernel , whichmarks pages to be wiped when a VM is suspended (a pre-cursor to snapshotting). We expect that PRNGs and crypto-graphic libraries mark their state or guard variables as both MADV_WIPEONFORK and
MADV_WIPEONSUSPEND and use thesame reseeding logic to handle both the fork and VM clonecases.Wiping on suspend, rather than restore, has two benefits.One is performance: in our serverless use-case, restore ison the latency-critical cold-start path, so handling memorywiping at suspend time optimizes restore latency. Secondis security: applications which have high-value secrets theydon’t want to include in the snapshot, or intentionally want toretrieve directly from a key management service or hardwaresecurity module (HSM) on restore, can mark these secrets tobe excluded from the snapshot.
MADV_WIPEONSUSPEND has the benefits of being simple,having very little performance overhead, and being easy touse both in libraries and application code. The performanceoverhead of
MADV_WIPEONFORK and
MADV_WIPEONSUSPEND is small, about 600ns per random generation. In an imple-mentation of NIST’s CTR_DRBG, performing the necessarychecks reduced per-core throughput by just TODO%.
The Windows Server 2012 version of the Hyper-V hypervisorintroduced the VM Generation ID (VmGenId). The guestOS detects that significant changes have occurred, includingcloning and restore, by a change in the VmGenId. Microsoft’sVmGenId implementation [1] exposes a 128-bit unique num-ber, a UUID, in a memory location that is discovered usinga special ACPI device name. While interacting with the low-level implementation is not convenient for applications and li-braries, and they are unlikely to have permissions to do so, theVmGenId is used in the Windows Cryptography API to avoidproblems with cloning. Other hypervisors, including VMWareand QEMU, have since added support for VmGenID.We have contributed support for a similar mechanism tothe Linux kernel, called System Generation ID (SysGenId).The Linux version implements a device driver which exposesa read-only device /dev/SysGenId to userspace, which con-tains a monotonically increasing generation counter. Librariesand applications are expected to open() the device, and thencall read which blocks until the SysGenId changes. Follow-ing an update, read() calls no longer block until the appli-cation acknowledges the new VmGenId by write ing it backto the device. Non-blocking read calls return
EWOULDBLOCK Our contributions here focus on Linux because we use it as the guestOS in AWS Lambda’s microVMs, but are widely applicable to other guestoperating systems, unikernels, and library OSs when their is no new SysGenId available. Alternatively, li-braries can mmap the device to get a single shared page whichcontains the latest VmGenId at offset 0.Linux SysGenId also supports a notification mechanismexposed as two ioctls on the device.
SysGenId_GET_-OUTDATED_WATCHERS immediately returns the number of filehandles that were open during the last SysGenId change buthave not yet acknowledged the new id.
SysGenId_WAIT_-WATCHERS blocks until there are no open file handles on thedevice which haven’t acknowledged the new id. These twointerfaces are intended for serverless and container controlplanes, which want to confirm that all application code hasdetected and reacted to the new SysGenId before sendingan invoke to the newly-restored sandbox. This notificationmechanism, unlike signals, allows SysGenId to be used insidelibraries without requiring changes to application code.The Linux SysGenId implementation also supports a Vm-GenId, as defined by Microsoft. SysGenId is the frontenddriver exposing the 32-bit generation ID, which depends on abackend driver to detect when the system’s identity changes.VmGenId is one such backend. when the VmGenId changes,the SysGenId is increased. Other backends to SysGenId canbe defined, making it more general, and allowing support forcases like userspace checkpoint restore (e.g. CRIU [4]), andLinux containers. In this sense, SysGenId is a generalizationof VmGenId.
The three solutions
MADV_WIPEONSUSPEND , Microsoft’s Vm-GenId, and Linux SysGenId) to handling uniqueness dur-ing VM cloning are compared in Table 1. In
MADV_-WIPEONSUSPEND ’s favor is its compatibility with
MADV_-WIPEONFORK , its additional use for excluding secrets fromsnapshots, and the fact that it can be used in containers andsandboxes that don’t have the ability to open or mmap files.The popularity of container technology, and sandboxing ap-proaches like seccomp-bpf make this property especiallyinteresting. Many sandboxes and containers want to preventprocesses from opening files and devices entirely. open , read and write give a potential attacker a lot more power than just madvise .In SysGenId’s favor is its flexibility, including the ability tomonitor for when processes and libraries have fully caught upto the latest version of a container. On Linux, both solutionshave little memory overhead (involving reading from a specialpage), and can be used by non-root processes. This sameflexibility comes with some risk. By enabling libraries andapplications to do blocking work on the restore path (while thesystem blocks on SYSGENID_WAIT_WATCHERS ), they can re-introduce a form of cold-start latency. Application and libraryauthors must be encouraged to use SysGenId for lightweightpurposes only.In Microsoft’s solution’s favor is its use of a high-quality4ADV VmGenId SysGenIdMechanism Guard Page UUID Inc. IdWorks for fork Yes No NoSecret hiding Yes No NoIn-memory Yes Yes YesNotification No No YesNon-root Yes No YesMin-privilege Yes No NoEntropy No Yes NoContainers No No YesTable 1: Comparison of features of
MADV_WIPEONSUSPEND ,Microsoft’s VmGenId, and Linux SysGenIdUUID VmGenId, which can be used directly as a node iden-tifier for distributed protocols, for logging, and tracing. TheUUID, which Microsoft [1] describe as "a 128-bit, crypto-graphically random integer value identifier" can also be useddirectly as a high-entropy seed for PRNGs, removing the needto separately reseed from the kernel or hardware. It can alsobe used directly for the deterministic construction of IVs (seeNIST SP800-38D, section 8.2.1). Microsoft’s VmGenId hasexisted for longer than either of the Linux solutions, and isbuilt into the Windows Cryptography API, meaning that italready has extensive userspace support.
While
MADV_WIPEONSUSPEND has fairly obvious semantics,SysGenId raises the question of under which circumstancesthe ID should be changed. Microsoft change their VmGenIdon restore, copy, clone, recovery from backup, and (in somecases) on failover. The VmGenId does not change on reboot,pause, resume, live migration, or even on host reboot. Whilemost of these seem like reasonable decisions, it is not clearthat they are a good match for a serverless environment.Consider the case of restoring from a snapshot withoutcloning. The resulting microVM is, for practical purposes, thesame microVM as the one that was snapshotted, as long asthe system can reason that it can never be clones. This meansthat SysGenId is just that—a generation id—and not the id ofthe microVM itself. Figuring out the right ID for a microVMis a separate system-level concern.
One alternative to approaching this problem at the systemlevel is to approach it at the level of cryptographic algorithms.Rogaway and Shrimpton proposed [23, 24] authenticated en-cryption modes that do not depend on a random nonce for theirstrongest security properties. Gueron et al proposed [15] andstandardized [14] AES-GCM-SIV provides a nonce misuse-resistant mode, which keeps it’s confidentiality and authen-tication properties in case of IV re-use (exposing only mes- sage equality) with only small performance overhead. Ris-tenpart [22] proposed changes to the TLS protocol to resistnonce-reuse. Despite these advances, and many others, manyof the cryptographic protocols and modes widely deployedin production remain vulnerable to nonce reuse. We do notexpect this situation to change quickly.Another alternative is to move towards deprecatinguserspace CSPRNGs, and encourage implementors to userandom numbers provided by the kernel or hardware. Dis-cussing the merits of this approach is beyond the scope ofthis paper, but it is clear than making such a change is notpossible without changing the performance of many librariesand applications, and changing many pieces of software toadopt new RNG interfaces. A system-level approach allowsus to approach this change incrementally, which is essentialgiven the size of the installed base and the goal of serverlessplatforms to work with existing application software.A similarly major architectural change is to build applica-tions which rely on cryptographic services provided by hard-ware security modules (HSMs), key management services(such as AWS Key Management Service), trusted executionenvironments (TEEs), and secure enclaves (such as AWS Ni-tro Enclaves). In many cases this would be a good choice,but again isn’t always applicable due to application-specificrequirements, data plane performance requirements, and largebodies of existing code.
Both the
MADV_WIPEONSUSPEND and SysGenId options pro-vide sufficient mechanism for a RNG library to ensure thatit does not generate duplicate random numbers in multiplecloned VMs . In the SysGenId case, this requires that the Sys-GenId update happens before (or atomically with) the restoreoperation. It is up to the VMM and guest kernel to ensure thatthis ordering is not violated.Serverless systems (like AWS Lambda) explicitly trackmanage the requests in-flight in a running VM, and ensurethat VMs cannot perform any actions outside the context ofhandling a request. These systems can therefore fully quiescea VM and ensure that it does not handle any requests untilthe notification process is complete. This is sufficient to en-sure that random numbers are never re-used across multiplerequests, provided that application code does not cache valuesbetween requests.In general, however, none of these mechanisms are suffi-cient to strongly prevent re-use, due to time of creation timeof use (TOCTOU) issues. In any program which does notatomically generate, use, and discard the generated value, aclone operation between generation and use, or between useand discard, can cause re-use. Atomicitiy of such complexoperations is not generally supported. Some external fence TLA+ specifications demonstrating this property are available athttp://redacted.example.com/
In order to understand the steady-state and restore-time per-formance impact of these mechanisms, we ran a number ofbenchmarks on typical x86 (EC2 m5.12xlarge) server. Wefirst considered the impact of reseeding: getting entropy fromthe kernel ( /dev/urandom ) or hardware (
RDRAND or equiva-lent), and passing it into the userspace PRNG (using openssl1.1’s
RAND_seed ).This basic reseeding is, as expected, very fast. Reseeding32 bytes from /dev/urandom (including open , read , and close ) took a mean of 11 µs , with insignificant deviation. Re-seeding 32 bytes from RDRAND or RDSEED took 0 . µs per runwith insignificant deviation (consistent with Intel’s advertisedperformance for RDRAND and RDSEED [3]).Checking the guard page (either a page mapped with MADV_WIPEONSUSPEND or the mmap ed VmGenId page) is,again, very fast. For the simple case of generating a sequentialnonce (just a 128-bit increment operation, in other words),enabling the check reduces throughput by 13x. Doing anykind of meaningful work inside the check, on the other hand,causes its cost to amortize to near-zero. For example, runningOpenSSL’s default md_rand generator with or without theguard page check showed no statistically significant differ-ence. Absolute differences will vary from platform to plat-form, but we expect these relative results to be durable acrossplatforms.
Snapshot, clone and restore are useful and powerful primitivesfor reduced cold start times in serverless compute platforms.Using these primitives introduces several challenges: breakingcommon network protocols, moving in-memory state ontodisk, and losing the ability for short-lived clones to makeunique decisions. We described two new kernel interfaces thatwe proposed for tackling this problem in context of Linuxand AWS Lambda, one of which has been contributed to themainline Linux kernel. This mechanism, a Linux flavor ofVmGenId, provides a flexible way to approach these problems,with little performance overhead, relatively easy adoption, andwide applicability.
References [1] Virtual machine generation id. Technical report, Mi-crosoft, August 2012. URL: http://go.microsoft. com/fwlink/?LinkId=260709 .[2]
Intel® 64 and IA-32 Architectures Software Developer’sManual Combined Volumes: 1, 2A, 2B, 2C, 3A, 3B and3C . Intel Corporation, June 2013.[3]
Intel® Digital Random Number Generator (DRNG) Soft-ware Implementation Guide, revision 2.1 . Intel Corpora-tion, October 2018. URL: .[4] Checkpoint/restore in userspace. Technical report,January 2021. URL: .[5] Alexandru Agache, Marc Brooker, Alexandra Iordache,Anthony Liguori, Rolf Neugebauer, Phil Piwonka, andDiana-Maria Popa. Firecracker: Lightweight virtual-ization for serverless applications. In { USENIX } Symposium on Networked Systems Design and Imple-mentation ( { NSDI } , pages 419–434, February 2020.[6] Bharathan Balaji, Christopher Kakovitch, and Balakrish-nan Narayanaswamy. Microplace: Placing micro virtualmachines with hindsight imitation. Technical report,October 2020.[7] Elaine Barker and John Kelsey. SP 800-90A Rev. 1.Recommendation for Random Number Generation Us-ing Deterministic Random Bit Generators . NationalInstitute of Standards & Technology, 2015.[8] Roy Bryant, Alexey Tumanov, Olga Irzak, Adin Scan-nell, Kaustubh Joshi, Matti Hiltunen, Andres Lagar-Cavilla, and Eyal De Lara. Kaleidoscope: cloud micro-elasticity via vm state coloring. In
Proceedings of thesixth conference on Computer systems , pages 273–286,2011.[9] James Cadden, Thomas Unger, Yara Awad, Han Dong,Orran Krieger, and Jonathan Appavoo. Seuss: Skipredundant paths to make serverless fast. In
Proceed-ings of the Fifteenth European Conference on ComputerSystems , EuroSys ’20, New York, NY, USA, 2020. As-sociation for Computing Machinery. doi:10.1145/3342195.3392698 .[10] John (JD) Douceur. The sybil attack. In
Pro-ceedings of 1st International Workshop on Peer-to-Peer Systems (IPTPS) , January 2002. URL: .[11] Dong Du, Tianyi Yu, Yubin Xia, Binyu Zang, Guan-glu Yan, Chenggang Qin, Qixuan Wu, and Haibo Chen.6atalyzer: Sub-millisecond startup for serverless com-puting with initialization-less booting. In
Proceedingsof the Twenty-Fifth International Conference on Archi-tectural Support for Programming Languages and Oper-ating Systems , ASPLOS ’20, page 467–481, New York,NY, USA, 2020. Association for Computing Machinery. doi:10.1145/3373376.3378512 .[12] Morris J Dworkin.
NIST SP 800-38D. recommendationfor block cipher modes of operation: Galois/countermode (gcm) and gmac . National Institute of Standards& Technology, 2007.[13] James Gosling, Bill Joy, Guy Steele, Gilad Bracha, AlexBuckley, and Daniel Smith.
The Java® Language Spec-ification: Java SE 11 Edition . Oracle, August 2018.[14] Shay Gueron, Adam Langley, and Yehuda Lindell.AES-GCM-SIV: Nonce Misuse-Resistant Authenti-cated Encryption. RFC 8452, April 2019. URL: https://rfc-editor.org/rfc/rfc8452.txt , doi:10.17487/RFC8452 .[15] Shay Gueron and Yehuda Lindell. Gcm-siv: full noncemisuse-resistant authenticated encryption at under onecycle per byte. In Proceedings of the 22nd ACM SIGSACConference on Computer and Communications Security ,pages 109–119, 2015.[16] Jashwant Raj Gunasekaran, Prashanth Thinakaran,Nachiappan Chidambaram, Mahmut T Kandemir, andChita R Das. Fifer: Tackling underutilization in theserverless era. arXiv preprint arXiv:2008.12819 , 2020.[17] Erika Hunhoff, Shazal Irshad, Vijay Thurimella, AliTariq, and Eric Rozner. Proactive serverless function re-source management. arXiv preprint arXiv:2010.04312 ,2020.[18] Erika Hunhoff and Eric Rozner. Poster: Network con-nection optimization for serverless workloads. In { USENIX } Symposium on Networked Systems Designand Implementation ( { NSDI } , February 2020.[19] Leslie Lamport. Paxos made simple. ACM SIGACTNews (Distributed Computing Column) 32, 4 (Whole Number 121, December 2001) , pages 51–58, December2001. URL: .[20] Paul J. Leach, Rich Salz, and Michael H. Mealling. AUniversally Unique IDentifier (UUID) URN Namespace.RFC 4122, July 2005. URL: https://rfc-editor.org/rfc/rfc4122.txt , doi:10.17487/RFC4122 .[21] Jacob R Lorch, Andrew Baumann, Lisa Glendenning,Dutch Meyer, and Andrew Warfield. Tardigrade: Lever-aging lightweight virtual machines to easily and ef-ficiently construct fault-tolerant services. In { USENIX } Symposium on Networked Systems Designand Implementation ( { NSDI } , pages 575–588, 2015.[22] Thomas Ristenpart and Scott Yilek. When good ran-domness goes bad: Virtual machine reset vulnerabilitiesand hedging deployed cryptography. In Network andDistributed Security Symposium , 2010.[23] P Rogaway and T Shrimpton. Deterministicauthenticated-encryption. In
Advances in Cryptology–EUROCRYPT , volume 6, 2007.[24] Phillip Rogaway and Thomas Shrimpton. The sivmode of operation for deterministic authenticated-encryption (key wrap) and misuse-resistant nonce-basedauthenticated-encryption.
Aug , 20:3, 2007.[25] Sergej Schumilo, Cornelius Aschermann, Ali Abbasi,Simon Wörner, and Thorsten Holz. Nyx: Greybox hy-pervisor fuzzing using fast snapshots and affine types.[26] Mohammad Shahrad, Rodrigo Fonseca, Inigo Goiri, Go-har Chaudhry, Paul Batum, Jason Cooke, Eduardo Lau-reano, Colby Tresness, Mark Russinovich, and RicardoBianchini. Serverless in the wild: Characterizing andoptimizing the serverless workload at a large cloudprovider. In , pages 205–218. USENIX Associ-ation, July 2020. URL: