PPersistent Data Retention Models
Tiancong Wang and James TuckNC State University { twang14,jtuck } @ncsu.edu Abstract
Non-Volatile Memory devices may soon be a part of mainmemory, and programming models that give program-mers direct access to persistent memory through loadsand stores are sought to maximize the performance ben-efits of these new devices. Direct access introduces newchallenges. In this work, we identify an important as-pect of programming for persistent memory: the persis-tent data retention model.
A Persistent Data Retention Model describes what hap-pens to persistent data when code that uses it is modi-fied. We identify two models present in prior work butnot described as such, the Reset and Manual Model, andwe propose a new one called the Automatic Model. TheReset model discards all persistent data when a programchanges leading to performance overheads and write am-plification. In contrast, if data is to be retained, the Man-ual Model relies on the programmer to implement codethat upgrades data from one version of the program to thenext. This reduces overheads but places a larger burdenon the programmer.We propose the Automatic Model to assist a program-mer by automating some or all of the conversion. Wedescribe one such automatic approach, Lazily ExtendibleData Structures, that uses language extensions and com-piler support to reduce the effort and complexity asso-ciated with updating persistent data. We evaluate ourPDRMs in the context of the Persistent Memory Devel-opment Kit (PMDK) using kernels and the TPC-C ap-plication. Manual Model shows an overhead of 2.90% to4.10% on average, and LEDS shows overhead of 0.45% to10.27% on average, depending on the workload. LEDS re-duces the number of writes by 26.36% compared to Man-ual Model. Furthermore, LEDS significantly reduces theprogramming complexity by relying on the compiler tomigrate persistent data.
Recently, Non-Volatile Memory (NVM) technologies aregarnering attention because they provide persistent stor-age, byte-addressability, and have reasonably fast accesslatency [15, 23, 20, 17, 18, 3]. These NVM technologiesmay be adopted as part of the main memory hierarchy,and adoption of non-volatile main memory (NVMM) may have a large impact on the way future computer systemsare designed and programmed.In conventional systems, persistent storage is accessedthrough the file system. In contrast, NVMM offers thepotential for programmers to access persistent data di-rectly through loads and stores. Providing direct access (DAX) to programmers improves performance, by cuttingaway unnecessary layers of software, but also introduceschallenges, like support for failure-safety [22, 11, 16, 5]and the need for position independence [9].In the pursuit of direct access, many have proposed thatpersistent data will be created and manipulated directlyby programs using conventional declarations, data struc-tures, and pointers [10, 25, 2, 26]. Data can be placed inpersistent memory on one run of a program and then ac-cessed again on a subsequent run. However, this createsa semantic challenge for programmers with regard to howpersistent data is described and manipulated over time.Since persistent data is retained across runs of a program,does a declaration of a type refer to the persistent datacreated in a previous run of the program or to the or-ganization of the data on the next run of the program?In systems with a conventional memory and storage hier-archy, programmers trust that data types describe whatwill happen when the program runs next, because there isno data already in memory – memory is repopulated fromscratch each run. However, in the context of persistentmemory the declarations describe both – what’s alreadyin persistent memory and what will happen on the nextrun.Consider an analogy with files on conventional systemswithout NVMM. If a file format changes, the software ismodified to support reading the old file type and convert-ing it as needed into a new format. For data stored seri-ally on secondary storage, this is reasonable since it mustbe moved from disk to memory on each invocation of theprogram. But, persistent memory is different in that per-sistent structures never need to be serialized. However,if the type declarations used in the previous executionchange, then the address calculations for the persistentdata accessed in the next execution will be invalid or er-roneous. Current languages and compilers have no au-tomatic way of detecting the discrepancy or helping theprogrammer deal with it. There are lot’s of software-based solutions here. The program-mer could add fields to the data structures, like a magic word, to a r X i v : . [ c s . P L ] S e p he responsibility of understanding what’s alreadyin persistent memory and how to handle changes fallssquarely to the programmer. Few works [2, 5], if any,have considered this problem. In PMDK and other frame-works, the programmer will keep the old declaration, adda new declaration to support the needed changes, andthen copy from the old data structure into the new oneon a subsequent run of the program. This approach leadsto redundantly declared structures and harmful write am-plification due to copied data.To begin addressing this problem, we propose that pro-gramming environments define a Persistent Data Reten-tion Model (PDRM) that explains how persistent data is retained as programs are modified and change over time.Programmers need, at a minimum, a clearly specifiedmodel of behavior.Perhaps the simplest model would be one that discardsall persistent data any time code changes. In a sense,this is equivalent to treating the persistent data the sameway we think of DRAM being cleared on each run of theprogram, except that the data in NVM would be clearedon each modification and re-compilation of the program.We call it the
Reset Model . While simple, it implies thatprogrammers cannot rely on data remaining in persistentmemory after making changes to their code, which defeatsthe purpose of persistent data structures and is inconsis-tent with the goal of supporting direct access to NVMM.It may also causes write amplification, since data will berecalculated and stored into memory again.Alternatively, we can make the programmer fully re-sponsible for distinguishing such data and managing itacross changes to the code. When a program is changed ina way that affects persistent data structures, the program-mer should manually convert the data that is affected bythe layout change into a new format, so we call it the
Manual Model . While programmers take full responsi-bility in this model, some minimal supports are neededfrom the environment to ensure that it is supported. Thisis supported in the PMDK because all persistent objectsare contained in pools, but this is not supported for staticpersistent variables in Mnemosyn because there is no wayto refer to the value held in a static persistent variable be-fore it was modified.Last, we define the
Automatic Model . The AutomaticModel assists programmers in the conversion of data andensures the data continues to be used appropriately af-ter code changes. This requires a mechanism that candistinguish which data can be retained (and how to) andwhich cannot after a program is modified. We describeone technique, Lazy Extendible Data Structures (LEDS),that fits into this classification that uses language andcompiler support to assist the programmer in automati-cally updating the layout of data structure, as we expectthis to be a common need. Furthermore, for the use cases track versions and manually convert between types on any change tothe code. However, this is an onerous and error-prone requirementto put on all persistent data. it supports, we show that it is competitive with an opti-mized
Manual Model based approach while requiring lessprogrammer effort.Other useful PDRMs may exist, in particular hybridmodels that are composed of these three, but we do notconsider them in depth in this work.We make the following contributions: 1) We iden-tify the problem of persistent data retention and de-scribe three different Persistent Data Retention Models,the Reset Model, the Manual Model and the AutomaticModel represented by LEDS. The Reset Model and Man-ual Model are present, to some degree in prior work, butLEDS is new to this work. 2) We examine the implemen-tations of the PDRMs in the context of the PersistentMemory Development Kit (PMDK, formerly known asNVML) [5], and we implemented and evaluated LEDS.3) We evaluate the Manual Model and LEDS with ker-nels written for PMDK and the TPC-C application wherethe Manual Model shows an overhead of 2.9% to 4.10%on average and LEDS shows and overhead of 0.45% to10.27%, on average, depending on the workloads. Wealso find that the number of writes is reduced by 26.36%,on average, when comparing LEDS to Manual Model. ForTPC-C, LEDS reduces the number of writes by 7.2 × overManual Model.The rest of the paper is organized as follows. Section 2gives a comparison of the traditional file-based persistentprogramming with persistent memory programming mod-els for NVMM and also explores the PDRMs in existingpersistent programming extensions. Section 3 introducesthe definition of Reset Model, Manual Model and Auto-matic Model in general persistent programming languagesand points out key differences between Manual Model andAutomatic Model. Section 4 demonstrates implementa-tions of each model built on PMDK libraries. Section 5presents the environment setup for the experiments andexplains the workloads and some terminologies used in ex-periments. Section 6 demonstrates the overhead of Man-ual Model and Automatic Model and analyzes the com-ponents of the overhead and also characterizes the twomodels. Section 7 presents related work and Section 8concludes the paper. Let’s compare traditional file-based support for persistentdata and new persistent memory programming models.As shown in Figure 1(a), with traditional file-based pro-gramming models, there are two copies of the data ex-isting in the program: one in volatile memory for use incomputation and one in a file. The interpretation of thetwo copies is written by the programmer. Here, the struct2learly only describes the copy in memory, not storage.For persistent memory, as shown in Figure 1(b), theprogrammer only needs to define a data structure anddenote that it resides in NVMM. The struct itself nowdescribes storage . Since we are considering persistentprogramming models that provide direct access to pro-grammers, there’s only one copy of the data structureand it’s persistent. struct example {int x;int y;};struct example array [100];int main (){// InterpreterFILE * fp = open (" somefile . txt ", "r");while ( fscanf (fp , "%d %d",& array [i].x, & array [i].y) !=EOF ){i ++;}fclose (fp);// Some operations...// Another interpreter to save data to files} (a) Traditional file-based programming struct example {int x;int y;};persistent struct example array [100];int main (){// Some operations on array...} (b) Persistent programmingFigure 1: A comparison of having a persistent data struc-ture using (a) traditional file-based programming modelsand (b) using persistent programming models.However, it also introduces a problem when a programchanges the struct definition. In the programs in Fig-ure 1, suppose the programmer wants to add a new field int z to the struct example . It’s easy for traditionalmodels in (a). They can modify the volatile data struc-ture freely because the struct does not describe the filecontents. However, with persistent programming modelin (b), the persistent data structure is laid out accordingto the original struct (two integers), but after the changethe program believes the persistent struct has three in-tegers. Thus it will read the wrong data if it continuesusing the old data still held in persistent memory.There are many reasonable software approaches to fixthis problem. We may want to reset the persistent mem- Here we assume some persistent programming model likeMnemosyne. Other persistent programming models like NV-Heapsand PMDK can also achieve this with more complicated interfaces. ory (Reset Model). However, the two integer fields x and y of the struct might still be useful and should notbe discarded. For example, a programmer might onlyadd a field in a well-established data structure. Thusall the other fields in the data structure should still beuseful. Furthermore, if discarded, they may simply be re-computed leading to detrimental write amplification andlowering the lifetime of the device. Instead, the ManualModel and the Automatic Model allow the programmerto retain the persistent data but with different complexityand overhead.Hence, the key issue is this: declarations now may de-scribe persistent data. As a result, modifications to dec-larations for persistent data need additional attention bythe programmer to ensure that the persistent data canevolve as expected when program version evolves. Wewant to minimize programmer effort while retaining highperformance and write endurance. We found no literature that discusses a similar conceptas Persistent Data Retention Models. However, there isrecognition that such support is needed. Table 1 sum-marizes the support in existing persistent programmingmodels. All the libraries we examined could support Re-set Model because they use a file abstraction for persistentdata.For NV-Heaps and PMDK, Manual Model can be im-plemented by manually duplicating the data structureand copying from an old struct layout to a new one.NVM-direct [2] identified the problem of needing to up-grade data structures and even suggests allocating thestruct with more memory than needed to support in-placeupdates. They also define an upgrade() function thatcan perform an in-place upgrade on the struct. However,the function may fail if there’s not enough space for theupgrade and will force a reset.
Our implementations are based on Persistent Memory De-velopment Kit (PMDK, formerly known as NVML) [5].It provides a collection of libraries for various use cases,which is tuned and validated to production quality andthoroughly documented [5]. It targets server-class appli-cations that requires skilled programmers to write pro-grams with low-level interfaces. There are 10 librariesin the PMDK and we will use the interfaces exclusivelyfrom libpmemobj as described in the man page [1]. Themain idea of PMDK is to store persistent regions in fileabstractions, or pools . Pools are directly mapped to aprogram’s address space and programmers can refer topersistent data with persistent pointers called ObjectIDs,or PMEMoid, which are the equivalent of addresses topersistent memory.3 rogrammingModels ResetModel ManualModel AutomaticModel
Mnemosyne [25] Yes Partial;Only fordynamicallocations NoNV-Heaps [10] Yes Yes NoNVM-direct [2] Yes Yes; upgrade() function NoPMDK [5] Yes; Also,can detecta change ofstruct lay-out and re-sets Yes No
Table 1: PDRMs in existing persistent programmingmodels that supports user-defined data types.In PMDK, all persistent data is held withinpools. There are no persistent global variables, as inMnemosyne [25], that are outside of a pool. Each poolhas a root object, which is defined as a struct, to storevariables associated with the pool, some of which may bepointers to data structures in the pool. To modify per-sistent data, a coder would modify one or more structdefinitions contained within the persistent pool. There-fore, in our Automatic Model approach (Section 4), wewill focus only on dealing with struct layout changes.PMDK offers some features to help detect if a poolwill be misinterpreted. Each pool can be tagged witha layout name when it’s created. If the pool is openedwith a layout name that’s different than the one used increation, the library will generate an error and abort theprogram.
As discussed in Section 2, persistent data is tied to theirprogram in persistent programming. In this section, wedescribe three different models to retain persistent datawhen a program evolves from one version to another. Weonly briefly describe Reset Model and Manual Model be-cause they reflect the state-of-the-art.
Reset Model mandates that the persistent data region iscleared/reset for a modified program in order to preventdata misinterpretation. The Reset Model is compellingfor its simplicity, effectiveness, and soundness. It’s detri-mental due to write amplification from recomputing dataand rebuilding data structures. Write amplification ismost severe when the working set of a program is much larger than the amount of data affected by the data struc-ture change.It’s easy for programming models to support ResetModel as long as they can easily clear the persistent re-gions. For example, for all the persistent programminglanguages in Table 1, they use files to hold persistent data.So when a program is modified, the file should be deletedso the persistent data of the program is cleared. Note,this does not happen automatically, even though it is anecessity for some of the systems to function properly.
The
Manual Model requires programmers to transformall the related persistent data to reflect the changes in anewer version of program.A sufficient requirement to provide the Manual Modelis that persistent data is dynamically allocated in a persis-tent heap. Then programmers can allocate new persistentobjects, copy the old ones, and fix-up all of the relevantpointers between persistent objects. The Manual Modelmay not work for persistent data that is statically mappedby the compiler, like in Mnemosyne, unless an interface isprovided for re-linking the data to a new location, whichhas been proposed in other contexts [14] and could beapplied here.The Manual Model can be implemented as a separateconversion program to be executed during re-compilationor as part of the larger program, running only when datalayout changes occur. The potential advantage of ManualModel is that programmers have the freedom to eitherretain or reset data precisely as needed. However, thisapproach places a large burden on the programmer. Theburden of Manual Model makes sense if persistent data iseither rarely used or predominantly read-only.
We wish to make retention of data simpler than in theManual Model, thus we propose
Automatic Model . Ide-ally, Automatic Model would fully or partially automatethe update process from one version of code to the next,requiring no special intervention. We believe a wide vari-ety of techniques may fall into this model. On a pathtoward that goal, we propose Lazily Extendable DataStructures (LEDS).
Using new language support, we only require that pro-grammers provide a high level description of how eachstructure is extended from its previous definition. Theextensions allow the structure to grow to encompass newfields or pointers. The compiler analyzes the descriptionand inserts code that will upgrade each object from theold definition to the new one when it is encountered dur-ing a subsequent program execution.4n example of possible C/C++ language extensionsare shown in Figure 2. The extendible keyword identi-fies the struct as one that may be extended in the future.The
EXTENSION block can be added once to indicate newfields that are desired. Within the block, any number ofnew fields may be added, including a nested
EXTENSION block that holds another set of extensions. The
INIT block defines how to initialize the new fields from exist-ing fields.Each
EXTENSION block represents one change or upgrade to the data structure. In our current design, extensionblocks can only be added and should never be removed orchanged, as this would again lead to a mismatch betweenthe struct and the persistent data in memory. extendible struct A{int val ;};
First version of struct A. extendible struct A{int val ;EXTENSION {double val_dbl ;INIT (A* obj ){// initialize new field using oldval val_dbl = obj -> val ;}}};
Second version of struct A.Figure 2: An example of retention with Automatic Model:struct A adds a floating-point type field to the struct Ain the second version, and copy the integer value.Based on the extension blocks, the compiler insertschecks into the code to test if an upgrade is needed ornot at each access to an object of the same type. Howthis check is done will be clear after we explain the up-grade process.If an upgrade is needed, we do not actually allocate anew object and copy over the old one, as in the Man-ual Model. Instead, for all structs marked with the extendible keyword, we embed an extra unused pointerfield at the end of the struct. If a programmer adds anextension, we allocate it as a new object in the same per-sistent pool and store its address in the pointer. Thismeans we do not make a copy, and hopefully do not sig-nificantly increase the number of writes. It also means wedo not need to fix-up references to the upgraded objectheld elsewhere in memory. However, accesses to the fieldsin the extension have to use an additional load throughthe pointer to access the desired field, which adds over-head.The presence of this pointer field makes it simple to testif an object has already been upgraded. If the pointer is Figure 3: A description of the differences in ManualModel and Automatic Model using LEDS when retain-ing persistent data in a changed program. The exampleshows a linked list with three nodes and searching in theupdated linked list hits the first node. In Manual Model,a new linked list needs to be created. And in AutomaticModel using LEDS, only the node accessed in the newprogram (first node) is being updated.NULL, it has not yet been updated; otherwise, it hasbeen.Our approach supports arbitrarily deep nesting. Anynew extension field is always given an extra pointer fieldto point to the next extension.A comparison between Manual Model and AutomaticModel is shown in Figure 3. Note that Manual Model, ingeneral, would make a copy of the existing data structure.However, in Automatic Model, we extend the existingdata by allocating a linked node.
What makes LEDS lazy?
Another advantage of ourdesign is that we do not need to track down all linkedobjects and convert them all at once, as a manual tech-nique might choose to do. Instead, we only convert theobjects when their changed fields happen to be accessedduring one run of the program. Because the languagecontains the information for how to convert the object,we can convert lazily as we encounter objects with NULLextension fields. Even if multiple nested extensions areneeded, they can simply be chained in sequence.
In this section, we discuss the implementations of threePDRMs in libpmemobj in PMDK [5].
To support the Reset Model, we manually delete all poolsused by a program before the next execution.5 .2 Implementation of Manual Model
PMDK supports the Manual Model. To convert persis-tent data, the coder must read out the persistent datausing the current layout from the pool, transform it intoa new layout by copying it into a new pool or a differentobject in the same pool, just like what we would expectfrom the file-based approach. Figure 4 gives an exampleof adding field to a persistent linked list using the libp-memobj [1] in PMDK.This example reveals that retaining only one field inthe root object, which is a linked data structure, takes anon-trivial amount of effort from a programmer, and ithas proven to be error prone in the context of PMDK.We make some observations from this example. First,some of this code is general enough that it could be putinto libraries. For example, the function root retain is for creating a new root object and retaining fieldsfrom an old one. Most of the code applies generallyto other programs: it needs to allocate a new root ob-ject, read out all the fields from the old root and copyto it. Since there’s only one root object in a pool, weshould use a temporary holder to hold the contents of anew root before actually making it the official root ob-ject. The only part that might be different is how toretain each field (e.g. linkedlist retain ). Thus if afunction is created for each field that needs retention, root retain can automatically call those functions oneach field. The other observation is that the compiler/li-braries can’t fully automate this whole process. It’s be-cause linkedlist retain needs to iterate through every old LL object in the old linked list in order to create anew linked list. The iteration might be different depend-ing on the data structure (e.g. graph versus linked list)and hard to automate. Furthermore, it’s also complicatedto retain data when there are nested structs. Even ifonly one struct changes its layout, all the objects relatedto it should change (e.g. in the example, only structLL changes but it also leads to convert the root object,which contains a field related to struct LL ). Thus inManual Model, we need the programmer to manually it-erate through the old data structure but there could besome library and compiler support to generate some com-mon code to reduce the amount of programming. We donot explore this possibility further.
We also implement LEDS for the PMDK libraries. Wehave not yet implemented a front-end for the language ex-tensions, because we wanted to justify that such an effortwould be worthwhile. Hence, we focus our implementa-tion on accurately measuring the performance and writeamplification of the proposed language extensions. Forour implementation, we manually implemented the codeusing the extensions we described earlier and manuallylower them into C/C++ language implementations. struct old_LL {int val ;TOID ( old_LL ) next ; // TOID : PMEMoid with aspecified type , used with D_RO and D_RW};struct LL {int val ;float val_fl ; // Added field in the newversion .TOID (LL) next ;};struct old_root_t {TOID ( old_LL ) ll_head ;};struct new_root_t {TOID (LL) ll_head ;};// Retain the linked listTOID (LL) linkedlist_retain ( PMEMobjpool * pop ,const TOID ( old_LL )& old_head ){ TOID (LL) head = TX_ZNEW (LL); // TX_ZNEW :Allocate a new object inside a transactionD_RW ( head ) ->val = D_RO ( old_head ) ->val ; //D_RO and D_RW : translates a PMEMoid typeinto an address , and reads / write from itD_RW ( head ) -> val_fl = D_RO ( old_head ) ->val ;TOID ( old_LL ) old_p = D_RO ( old_head ) -> next ;TOID (LL) p = head ; // Retain the next nodeof pTX_FREE ( old_head ); // TX_FREE : Free anobject inside a transactionwhile (! TOID_IS_NULL ( old_p )){D_RW (p) -> next = TX_ZNEW (LL);// Retain the content from old_p to D_RW(p) -> nextD_RW ( D_RW (p) -> next ) ->val = D_RO ( old_p ) ->val ; D_RW ( D_RW (p) -> next ) -> val_fl = D_RO ( old_p) ->val ;// Move on to the next node in thelinked list and free the old nodeTOID ( old_LL ) to_free = old_p ;old_p = D_RO ( old_p ) -> next ;p = D_RO (p) -> next ;TX_FREE ( to_free );}return head ;}// Retain the root objectTOID ( new_root_t ) root_retain ( PMEMobjpool * pop ){ TOID ( old_root_t ) old_root = POBJ_ROOT (pop ,old_root_t ); // POBJ_ROOT : Obtain the rootobject of a pool// Create a temporary holder of the new rootTOID ( new_root_t ) new_root_temp = TX_ZNEW (new_root_t );// Copy the whole linkedlist pointed by thehead fieldD_RW ( new_root_temp ) -> head =linkedlist_retain (pop , D_RO ( old_root ) -> head );// Allocate a new root and copy from thetempTX_FREE ( old_root );TOID ( new_root_t ) new_root = POBJ_ROOT (pop ,new_root_t );TX_MEMCPY ( new_root , new_root_temp ); //TX_MEMCPY : Perform memcpy () -like operationsfrom one PMEMoid to another inside atransactionreturn new_root ;}
Figure 4: An example manual retention of a linked-listwhen adding a floating point field and retaining the valuefrom the integer field, with native PMDK libraries.6 .3.1 Lowering the extendible
Struct. extendible struct LL{int val ;TOID ( struct LL) next ;EXTENSION {float val_fl ;INIT (LL* obj ){val_fl = obj -> val ;}}}; (a) Written by programmers. struct LL{int val ;PMEMoid to_extend ;};struct LL_EXT {double val ;PMEMoid to_extend ;LL_EXT ( const LL* obj ) {val_fl = obj -> val ;to_extend = OID_NULL ;}}; (b) Generated by the compiler.Figure 5: An example of retention with Automatic Modelusing LEDS: adding a floating point value and initializingits value from the integer field of last run.An extendible struct is supported by splitting its defi-nition into multiple structs, one for the original struct andone for each
EXTENSION block. A pointer is placed at theend of each struct marked with the extendible keywordto point to the next extension. By convention, we referto this pointer as to extend , and it will be used to ac-cess the extension indirectly. We are essentially buildinga linked list from the extension fields.Extensions are enforced by the compiler to be strictlynested, to reflect the sequential order of updates to a pro-gram. As shown in Figure 5, the original struct LL is splitinto two structs, LL and LL EXT.
The compiler must generate code that upgrades an objecton the first access to one of its extended fields. Note, wecould attempt to upgrade on any access, but we makethis as lazy as possible to reduce the overhead of benignupdates to code, for example, adding a field that is notused.For each struct marked extendible , we traverse thefull code and find all references to its fields that are in an
EXTENSION block. Just before the access (either a loador store), we insert code to check if the corresponding to extend field used to access it is non-NULL. If non-NULL, the object has already been extended and the codefalls-through to the access. For he other case, we add code to allocate the extension and insert the initialization codespecified in the
INIT initializer. Last, the appropriateindirect access using the to extend pointer is generated.Since LEDS is a lazy approach, a persistent objectmight not be accessed after an update is made to itsstructure definition. In this case, when the programmeraccesses fields in the newest extension, the routine alsoneeds to check all the intermediate to extend fields thatlead to the newest extension in sequence, allocating exten-sions that turn out to be NULL. For any given extension,the extension only needs to be allocated once, whether inthe current run or a later run of the program.The NULL pointer check on to extend is only neededon the first access of one of the fields it directly pointsto. However, in general, deciding if it is the first access isdifficult. Therefore, by default, we always insert a checkbefore any dereference of an extension field.Figure 5(a) shows the same data structure fromFigure 4 rewritten for PMDK using LEDS. Now, linkedlist retain and root retain are no longerneeded.
LEDS has the potential to incur higher overheads than amanual approach. However, based on our observations,many of these overheads can be reduced through opti-mization.
Extra Checks.
Because LEDS lazily updates struc-tures, every run of the program will need to check if ex-tendible objects have been updated before accessing anyof their extended fields. Since the compiler may not beable to determine the first access of each object, many re-dundant checks may be inserted. Furthermore, for nested
EXTENSION blocks, there will be multiple levels of check ifthe field is placed in an innermost
EXTENSION block.The compiler can reduce redundant checks on extendedobjects. For example, a redundant check B can be re-moved if a check A of the same object is placed in a basicblock that dominates B’s basic block. We do not evaluatethis optimization further in this paper.
ObjectID Translation.
In PMDK, persistent objectsare accessed using ObjectIDs. ObjectIDs must be trans-lated to an address before they can be used to accessmemory. This overhead can be reduced through compileror hardware support [9, 26].The compiler can also reduce the cost of translationsby caching them and re-using them. Thus it saves someredundant translations. We evaluate this optimization inSection 6.2.
Unnecessary Copies.
Lastly, the compiler can avoidallocating extensions for objects that only exist in volatile7emory for a short period of time. The programmermight create a temporary local variable to hold a copyof persistent data. When copying a field to such a local,we may not need to perform a deep copy if the fields arenever modified. We can perform a shallow copy instead.Two of our workloads, namely btree and ctree, benefitfrom this optimization. We evaluate this optimization inSection 6.2.
LEDS keeps the original data layout by adding new fieldsas extensions. This breaks an assumption that program-mers and compilers often make: all the data of an objectis stored contiguously in the address space. For example,copying structs is a simple memcpy operation. However,that will not work for our extendible structs because theywill need a deep copy instead. When a compiler detects acopy, it can insert code as necessary to make a deep copy.However, if a programmer manually copies a struct, itcould lead to an error, perhaps adding some difficulty forusing LEDS.The extension field also adds some complexity in termsof freeing the extendible object. When free() is called onthe extendible object, it needs an explicit call to free theextensions as well. The compiler can insert a routine tomake sure the extensions of the object are all freed, in-cluding the extensions of the extensions. However, again,there may be cases this goes undetected due to pointercasting, allowing some extended objects to leak. However,these dangers are not significantly different than those al-ready present in the C language.
The evaluation is performed on a workstation summa-rized in Table 2. We implement the Manual Model andusing interfaces in Persistent Memory Development Kit(PMDK) [5], formerly known as NVM Library (NVML),developed by Intel. We also implement Lazily ExtendableData Structures (LEDS) as an Automatic Model usingPMDK libraries. From now on, we will refer to LEDS asAutomatic Model. We run the benchmarks on a hard diskinstead of real NVM hardware. PMDK supports harddisk with NVM interfaces and simulates the clflush in-structions (used for making sure the data is persistent)with an msync() call. In our experiment, we use the con-figuration of PMEM IS PMEM FORCE=1 provided byPMDK to avoid issuing msync() and unnecessarily slow-ing down the program.
The workloads are shown in Table 3. We select multi-ple implementations of maps (list map, hash map andtree map) from examples/libpmemobj in PMDK as our
Processor Intel Core 2 Duo CPU E8400 @3.00GHzCPU Cache L1D: 32KB, L1I: 32KB, L2: 6MBMemory 4GB DRAMOperating Sys-tem Linux version 2.6.32 (Red Hat Enter-prise Linux Workstation release 6.9)Hard Disk 110GLibrary Persistent Memory Development Kit(PMDK) Version 1.3.1
Table 2: Summary of the environment for experimentsworkloads, and we implement a TPC-C application withPMDK interfaces.The original programs are modified to use a 32-bit key and are referred as P-Original . In order to measure theoverhead of different retention situations, we make somechanges to the layout (
Layout-x ). In order to performretention and preserve the data in the
P-Original , wewrite retentions and updated programs for both the Man-ual Model and the Automatic Model. For the ManualModel, we write a separate program to perform the re-tention (
P-Retain ), so the updated program
P-Manual only needs to change the definition in the
P-Original .With Automatic Model, we integrate the retention in theupdated program
P-Auto using our proposed languagefeatures and their required transformation.
Workloads Descriptions skiplist Skip list data structure with 4 levels.ctree Crit-bit Treebtree B-Tree of Order of 8rbtree Red Black Treehashmap tx Hash map implementations usingtransaction APIs, with 10 bucketsinitiallyTPC-C The implementation is based froman implementations that uses volatilememory. We move the B+ Trees (Or-der of 8) on to the NVM with PMDKinterfaces.
Table 3: Description of the workloads from PMDKIn our experiments, we make two different data layoutchanges:
Layout-Change is that we change the 32-bitkey to 64-bit key and
Layout-Add is that we add a newfield to the node, a char array as the name. This measuresthe impact of different usage scenarios when changing afield: the key field will be used heavily, whereas the newlyadded field might be used very lightly. We expect the realworld usage will be a combination of the two situations.The experiments measure the overhead of retention inboth models: in Manual Model, the overhead of retentionis the time spent on the retention program
P-Retain . So PMDK kernels implements 64-bit key so we change it to 32-bit,in order to retain the keys in the updated program.
8e calculate the overhead as:
Overhead
Manual = T P-Retain T P-Manual × Overhead
Auto = T P-Auto − T P-Manual T P-Manual × P-Retain , P-Manual and
P-Auto for fivetimes and calculate the average execution time beforefeeding into the two equations to calculate overheads.For the PMDK map kernels, we write a main programthat performs all three operations (insertion, deletion andsearch) on the map randomly: we randomly generate anumber of integers and search them in the map. If it’sfound, we remove the key and value pair from the map.Otherwise, we insert a key-value pair for the integer. Thesequence of random numbers is provided as an input toensure the same behavior across all executions. In our ex-periments, we use three different types of inputs to evalu-ate different behaviors,
PMDK-INS , PMDK-DEL and
PMDK-RAND , which is named after the behavior inthe update program with explanations in Table 4. Theseconfigurations are used to measure the impact of differ-ent sequences of operations in the update program, whichmight result in different overheads for the retention mod-els.
Name Abbr. Descriptions
DeletionsOnly PMDK-DEL
P-Original inserts N unique keys and
P-Manual / P-Auto deletes the same Nunique keys.InsertionsOnly PMDK-INS
P-Original inserts N unique keys and
P-Manual / P-Auto inserts another Nunique keys.Combinationof Inser-tions andDeletions PMDK-RAND
P-Original inserts N random keys thatmay be repeated and
P-Manual / P-Auto uses the same input. The updateprogram will have a combination of inser-tions and deletions.
Table 4: Description of different configurations of inputsto the benchmarks in PMDK.
In this section, we present the overhead of the ManualModel and the Automatic Model with two different
Lay-out changes on the workloads mentioned in Table 3. The
Category Term Explanation
Differentlayoutchanges
Layout-Change
Change the field key from 32-bit to64-bit.
Layout-Add
Add a new field of a char array.Differentprogramsin theexperiments
P-Original
The original version of program with32-bit keys.
P-Retain
The program used to perform reten-tion in Manual Model and executedbefore the updated program.
P-Manual
The program with updated layoutdefinition in Manual Model.
P-Auto
The program with updated layoutdefinition and primitives for reten-tion in Automatic Model.
Table 5: Terms used in the experiments.Figure 6: Overall performance of the overhead of ManualModel and Automatic Model on different workloads. (a)
Layout-Change : key changes from 32-bit to 64-bit. (b)
Layout-Add : add a new field to each node. The resultof PMDK kernels is measured with 100000 operations inboth
P-Original and updated program, and the resultof TPC-C is measures with 10 warehouses and 200000random client operations.overheads calculated by equations Eq. 1 and Eq.2 areshown in Figure 6.We analyze the PMDK kernels first. For the ManualModel, both
Layout-Change and
Layout-Add needto duplicate the old data structure because the size of thenode changes. The execution time of
P-Retain is similarin these benchmarks but the overhead is slightly differentdue to different execution times for the
P-Manual , as aresult of a different number of insertions or deletions. Formost workloads, an insertion operation and a deletion op-eration take a similar amount of time, except for skiplistwhere skiplist(INS) runs much longer than skiplist(DEL)due to larger data size. Hence, the overhead of retentionis amortized in that case. For
Layout-Change , ManualModel has 4.10% and 2.90% overhead in PMDK-DEL andPMDK-INS kernels respectively, while
Layout-Add has9.87% overhead in PMDK-RAND kernels. Note, theseoverheads are similar.For Automatic Model, the overhead can vary sig-nificantly across workloads. First,
Layout-Change has a bigger impact on the overhead (7.81%, 10.21%for PMDK-DEL and PMDK-INS) than
Layout-Add (0.45%) because the key is constantly accessed. Hence, itneeds checking, redirection, and translation on the fieldquite often. However, with
Layout-Add , it’s a newlyadded field that is not otherwise used, so it adds littleoverhead. In reality, we expect applications will havea combination of changes akin to
Layout-Change and
Layout-Add so the performance will be somewhere in-between.When comparing the PMDK-DEL and PMDK-INS,Automatic Model can have larger overhead in the lat-ter because there are more keys in PMDK-INS and everyaccess to the key can add overhead. PMDK-INS can alsohave larger overhead because newly allocated nodes carryextra work to build the extensions. We will break downthe overhead of Automatic Model model in Section 6.2.When comparing the Manual Model and the AutomaticModel across the workloads, some general trends can beobserved. The Manual Model has similar overhead toretain data whether it’s adding new fields or changingexisting fields (as long as the size changes) because itneeds to allocate a new data structure and copy from theold one. But, the overhead paid in the Automatic Modelis relative to the number of accesses of the changed fields.If the accesses are rare, the Automatic Model can havemuch smaller overhead than Manual Model.As for the TPC-C application, Manual Model has simi-lar overheads with
Layout-Change and
Layout-Add ,with 10.65% and 10.49% overheads respectively. The Au-tomatic Model attains a better performance in these twoscenarios, with 8.98% in
Layout-Change and 1.85% in
Layout-Add .Furthermore, in TPC-C, the Automatic Model hassmaller overhead than the Manual Model, which showsanother advantage of the Automatic Model. The Auto-matic Model only transforms what is needed instead oftransforming all the data. Our experiment shows thatthe second run of the TPC-C after the update only ac-cesses about 67.4% of the nodes from the previous run, soAutomatic Model only performs 67.4% of the work thatManual Model does. We further analyze the sensitivityon the ratio of working data set over total data set inSection 6.3.
There are multiple components of the overheads in Auto-matic Model, as analyzed in Section 4.4. We will evaluatethree major sources of overhead: the time for allocatingextension object and copying fields, the extra redirectionand translation cost to access extensions, and the over- Figure 7: A breakdown of the overheads in AutomaticModel. The experiment is done with PMDK-DEL work-loads of
Layout-Change , with 100000 insertions in
P-Original and 100000 deletions in
P-Auto .head when copying objects with extensions. To measurethe time that
P-Auto programs spent on each overhead,we implement multiple programs that remove each com-ponent from the
P-Auto program individually and seehow much overhead it can save. Figure 7 shows the frac-tion of each component for each workload.The most significant component is the cost of allocatingthe extension and initializing it. We see that PMDK-DELspends on average 55.63% of the overhead for allocation.The second most significant component is the overheadof translation. On average PMDK-DEL spends 36.44%of the overhead for such translations. We are using na-tive PMDK translation functions and we think with fastertranslation techniques like [26, 9], this overhead can bereduced. It’s also worth noting that even with virtualaddresses stored in the to extend field, accessing a fieldin extension still has an extra level of redirection becauseit needs extra loads to locate the extension object first.The portion of such overhead is illustrated in the Otherspart in the Figure 7 and is non-trivial.The last component of overhead we analyzed is the costof making deep copies of objects with extensions. Onlytwo workloads ctree and btree have this situation. Inbtree, 14.10% of the overhead falls into this category, andwe can save the unnecessary allocations in about 51.61%of the copies.
Now we consider the impact that the size of the data sethas on the overhead. In this experiment, we vary the totalamount of data and the total number of operations per-formed on the data in the same proportion.Figure 8 showsthe overhead of Manual Model and Automatic Model withmultiple data set sizes to retain.In the case of Manual Model, for sizes above 100 keys,the behavior is fairly uniform. skiplist is an exceptionbecause with increasing N each deletion might take longerto perform due to the larger data structure. This makesthe cost of P-Retain relatively less than
P-Original .For the Automatic Model, the results are also fairlyuniform, with a slight worsening for large data sizes. The10igure 8: Sensitivity analysis of the impact of differentdata set on both (a) Manual Model and (b) AutomaticModel. The experiment is done with PMDK-DEL of
Layout-Change , with different number of operationsof 100, 1000, 10000, 100000, resulting in different size ofdata set for retention.overhead comes from the time spent on allocation andcopying new extension objects and also the accesses tothe fields placed in extensions. Thus for larger N , eachdeletion will access a larger number of keys that will in-crease the overhead. The trend is the most significant forskiplist that needs to access at most N keys for each dele-tion. Evenso, for some workloads, like rbtree, ctree, andhashmap, Automatic Model performs better than ManualModel. The Automatic Model only transforms data that is ac-cessed. For programs with a small working set com-pared to their total data set, Automatic Model has anadvantage. We vary the number of deletions performedin the update program
P-Manual and
P-Auto andpresent the overhead of both Manual Model and Auto-matic Model in Figure 9. For small working sets, Au-tomatic Model significantly outperforms Manual Model,even for the 10% case.
NVMM technologies have limited write endurance, hencetechniques that increase writes are undesirable. Reducingthe number of writes is always prudent.In this section, we estimate the number of bytes writtenunder the Manual Model and the Automatic Model whenconverting old data structure. Table 6 shows the numberof additional writes incurred by the Manual Model andthe Automatic Model for PMDK-DEL and TPC-C on the Figure 9: Sensitivity analysis of different ratio of workingdata set in updated program on both (a) Manual Modeland (b) Automatic Model. The experiment is done withPMDK-DEL of
Layout-Change , where
P-Original performs 100000 insertions and update program deletes0.1%, 1%, 10% and 100% of the keys respectively. They-axis is on logarithmic scale.
Layout-Change workload.
Reset Manual AutoTotalBytes(MB) Struct in retentionprogram Eachmig.(B) TotalBytes(MB) Eachmig.(B) TotalBytes(MB)skiplist { uint64 t ,PMEMoid } , TOID[4] 88 8.4 40 3.8 ctree { uint64 t ,PMEMoid } [2] 52 4.9 40 4.3 btree { uint64 t ,PMEMoid } [8], TOID[8] 324 6.7 40 9.5 rbtree uint64 t , PMEMoid,enum, TOID, TOID [2] 76 7.2 40 3.8 hashmap uint64 t , PMEMoid,TOID 40 3.8 40 3.8 TPC-C uint64 t [8],PMEMoid [8] 196 81.0 40 11.1
Table 6: Number of writes for different Persistent DataRetention Models after 100000 insertions in
P-Original .For the Reset Model, the data of original program isdiscarded after 100000 runs. There’s retention of thedata. When the program changes, we need to repeat100000 runs and during each run, we need to repeat thewrites to NVM as the original program. Table 6 showsan estimation of the number of bytes written when theprogram is re-executed.In Manual Model, when we modify the key field from32-bit to 64-bit, we need to create new objects. Also,this adds extra writes for the fields that do not changesince they are copied. Different workloads have differentstructures to hold the key-value pair.Table 6 shows the definition of each struct that needsretention in the second column. The { uint64 t andPMEMoid } pair is the key-value pair, and note that weneed to retain all the structs that have the key-valuestruct as its field. The size of each struct is calculated11n the second column . All the bytes that are writtenduring the P-Retain are calculated in the third column.Meanwhile, with the Automatic Model, we only need toallocate data for the fields placed in the extension, whichin this case is the 64-bit key. We also need the to extend field in the extension in order for further extension, result-ing in 24 bytes total. We also need to write the extensionfield of the object (with the allocated ObjectID), whichis 16 Bytes. Thus in Automatic Model, each retentionneeds to write 40 Bytes of data, and the total numberof bytes to write is calculated in the last column. Fromthe results in column 3 and 5, we can conclude that Au-tomatic Model writes fewer bytes to NVM than ManualModel in these scenarios. For TPC-C, LEDS reduces thenumber of writes by 7.2 × over Manual Model.Automatic Model reduces amplification and increasesthe write endurance of NVM by only updating the datathat is accessed instead of copying the whole data struc-ture. To the best of our knowledge, there is no prior work thatdiscusses a Persistent Data Retention Model or a similarconcept.There are several research projects focusing on pro-gramming interfaces for persistent memory, and herewe discuss relevant works not previously discussed inthe paper. Recent research like [25, 10, 7, 2] seeks alightweight programming model to support programmingwith NVMM. Mnemosyne [25] provides simple interfacesfor the C language and using software transactional mem-ory to provide atomicity. NVHeaps [10] provides persis-tent objects with transactional semantics while prevent-ing possible pointer-related errors. In [7], they systemat-ically explore a programming model suitable for persis-tent programming by defining semantics and identifyingimplementation costs for a wide range of programs. Theydo mention that persistent programming models need toadopt a model for checking if it can restart from the per-sistent data or not.There are more prior works, like [4, 19, 24, 8, 6, 21], thatprovide persistent programming to database systems. Webelieve that while the implementation of our design isbased on PMDK, our approach can be applied in a vari-ety of existing persistent programming libraries that pro-vide dynamic persistent memory allocation and atomicityprimitives.Another technique that bears resemblance to ours isDynamic Software Updating (DSU) [14]. DSU, at itsheart, is also about retaining data from one version ofsoftware to another. But our work has different goals.Their work focuses on upgrading a running program,like a server that cannot be brought down. However, The size of PMEMoid and TOID is 128-bit (16 Bytes), andTOID is a typed PMEMoid type in PMDK
PDRM focuses on giving programmers a means of rea-soning about outcomes when developing programs withpersistent data. Another difference is that DSU requiresprogrammers to provide both versions of the program anda dynamic patch in order for compilers to fix the mem-ory during compilation, whereas PDRM doesn’t requirefull source code of previous versions and only needs thelayout of the data that changes.LEDS bears similarity to structure splitting [13, 12].
We identify an important dimension introduced by per-sistent programs: how to retain persistent data in thepresence of data layout changes. A key challenge is givingthe programmer a way to reason about such changes. Werefer to this as the Persistent Data Retention Model, andwe propose LEDS, an example of the Automatic Model.We show that LEDS offers competitive performance andsignificantly fewer writes than the state-of-the-art for theworkloads studied.
Acknowledgements
Wang and Tuck were supported in part by NSF grantCNS-1717486 and by NC State.
References [1] Man page for the libpmemobj library in persis-tent memory development kit (pmdk) in linux,http://pmem.io/pmdk/manpages/ linux/mas-ter/libpmemobj/libpmemobj.7.html.[2] Nvm direct library, 2015.[3] Hiroyuki Akinaga and Hisashi Shima. Resistive ran-dom access memory (reram) based on metal oxides.In
IEEE, Vol. 98, Issue: 12 , Oct. 2010.[4] Timothy Andrews and Craig Harris. Combining lan-guage and database advances in an object-orienteddevelopment environment. In
Conference Proceed-ings on Object-oriented Programming Systems, Lan-guages and Applications , OOPSLA ’87, pages 430–440, New York, NY, USA, 1987. ACM.[5] ”NVM Library Team at Intel”. Persistent memoryprogramming, http://pmem.io, August 2016.[6] M. P. Atkinson, L. Dayn`es, M. J. Jordan, T. Print-ezis, and S. Spence. An orthogonally persistent java.
SIGMOD Rec. , 25(4):68–75, December 1996.[7] Hans-J. Boehm and Dhruva R. Chakrabarti. Persis-tence programming models for non-volatile memory.12n
Proceedings of the 2016 ACM SIGPLAN Interna-tional Symposium on Memory Management , ISMM2016, pages 55–67, New York, NY, USA, 2016. ACM.[8] Michael J. Carey, David J. DeWitt, Michael J.Franklin, Nancy E. Hall, Mark L. McAuliffe, Jef-frey F. Naughton, Daniel T. Schuh, Marvin H.Solomon, C. K. Tan, Odysseas G. Tsatalos, Seth J.White, and Michael J. Zwilling. Shoring up persis-tent applications. In
Proceedings of the 1994 ACMSIGMOD International Conference on Managementof Data , SIGMOD ’94, pages 383–394, New York,NY, USA, 1994. ACM.[9] Guoyang Chen, Lei Zhang, Richa Budhiraja, XipengShen, and Youfeng Wu. Efficient support of positionindependence on non-volatile memory. In
Proceed-ings of the 50th Annual IEEE/ACM InternationalSymposium on Microarchitecture . ACM, 2017.[10] Joel Coburn, Adrian M. Caulfield, Ameen Akel,Laura M. Grupp, Rajesh K. Gupta, Ranjit Jhala,and Steven Swanson. Nv-heaps: making persis-tent objects fast and safe with next-generation, non-volatile memories. In
International conference onArchitectural support for programming languages andoperating systems , Mar. 2011.[11] Jeremy Condit, Edmund B. Nightingale, Christo-pher Frost, Engin Ipek, Benjamin Lee, Doug Burger,and Derrick Coetzee. Better i/o through byte-addressable, persistent memory. In
ACM Symposiumon Operating Systems Principles , Oct. 2009.[12] Olga Golovanevsky and Ayal Zaks. Struct-reorg:current status and future perspectives. In
Proceed-ings of the GCC Developers Summit , pages 47–56,2007.[13] Mostafa Hagog and Caroline Tice. Cache aware datalayout reorganization optimization in gcc. In
Pro-ceedings of the GCC Developers Summit , pages 69–92, 2005.[14] Michael Hicks and Scott Nettles. Dynamic soft-ware updating.
ACM Trans. Program. Lang. Syst. ,27(6):1049–1096, November 2005.[15] Intel and Micron. Intel and micron produce break-through memory technology, Jul. 2015.[16] Arpit Joshi, Vijay Nagarajan, Marcelo Cintra, andStratis Viglas. Efficient persist barriers for multi-cores. In
International Symposium on Microarchi-tecture , Dec. 2015.[17] T. Kawahara, R. Takemura, K. Miura, J. Hayakawa,S. Ikeda, Y. Lee, R. Sasaki, Y. Goto, K. Ito,T. Meguro, F. Matsukura, H. Takahashi, H. Mat-suoka, and H. Ohno. 2mb spin-transfer torque ram(spram) with bit-by-bit bidirectional current write and parallelizing-direction current read. In
IEEE In-ternational Solid-State Circuits Conference. Digestof Technical Papers , Feb. 2007.[18] Emre Kultursay, Mahmut Kandemir, Anand Siva-subramaniam, and Onur Mutlu. Evaluating stt-ram as an energy-ef?cient main memory alternative.In
IEEE International Symposium on PerformanceAnalysis of Systems and Software , Apr. 2013.[19] Charles Lamb, Gordon Landis, Jack Orenstein, andDan Weinreb. The objectstore database system.
Commun. ACM , 34(10):50–63, October 1991.[20] Benjamin C. Lee. Phase change technology and thefuture of main memory. In
IEEE Micro, Vol. 30,Issue: 1 , Jan. - Feb. 2010.[21] B. Liskov, A. Adya, M. Castro, S. Ghemawat,R. Gruber, U. Maheshwari, A. C. Myers, M. Day,and L. Shrira. Safe and efficient sharing of persis-tent objects in thor. In
Proceedings of the 1996 ACMSIGMOD International Conference on Managementof Data , SIGMOD ’96, pages 318–329, New York,NY, USA, 1996. ACM.[22] Steven Pelley, Peter M. Chen, and Thomas F.Wenisch. Memory persistency. In
International sym-posium on Computer architecuture , Jun. 2014.[23] Raghunath Rajachandrasekar, Sreeram Potluri, Ak-shay Venkatesh, Khaled Hamidouche, Md. Wasiur Rahman, and Dhabaleswar K. (DK) Panda. Mic-check: A distributed check pointing framework forthe intel many integrated cores architecture. In
In-ternational symposium on High-performance paralleland distributed computing , Jun. 2014.[24] Vivek Singhal, Sheetal V. Kakkad, and Paul R. Wil-son. Texas: Good, fast, cheap persistence for c++.In
Addendum to the Proceedings on Object-orientedProgramming Systems, Languages, and Applications(Addendum) , OOPSLA ’92, pages 145–147, NewYork, NY, USA, 1992. ACM.[25] Haris Volos, Andres Jaan Tack, and Michael M.Swift. Mnemosyne: Lightweight persistent memory.In
Proceedings of the Sixteenth International Confer-ence on Architectural Support for Programming Lan-guages and Operating Systems , ASPLOS XVI, pages91–104, New York, NY, USA, 2011. ACM.[26] Tiancong Wang, Sakthikumaran Sambasivam, YanSolihin, and James Tuck. Hardware supported per-sistent object address translation. In