Equational reasoning for non-determinism monad: the case of Spark aggregation
aa r X i v : . [ c s . P L ] J a n Equational Reasoning for Non-determinism Monad:
The Case of Spark Aggregation
SHIN-CHENG MU,
Academia Sinica, TaiwanAs part of the author’s studies on equational reasoning for monadic programs, this report focus on non-determinism monad. We discuss what properties this monad should satisfy, what additional operators andnotations can be introduced to facilitate equational reasoning about non-determinism, and put them to thetest by proving a number of properties in our example problem inspired by the author’s previous work onproving properties of Spark aggregation.
In functional programming, pure programs are those that can be understood as static mappingsfrom inputs to outputs. The main advantage of staying in the pure realm is that properties of pureentities can be proved by equational reasoning. Side effects, in contrast, used to be consideredthe “awkward squad” that are difficult to be reasoned about. Gibbons and Hinze [2011], however,showed that effectful, monadic programs may also be reasoned about in a mathematical manner,using monad laws and properties of effect operators.This report is part of a series of the author’s studies on equational reasoning for monadic pro-grams. In this report we focus on non-determinism monad — in our definition that is a monadhaving two effect operators, one allowing a program to fail, another allowing a non-deterministicchoice between two results. We discuss what properties these operators should satisfy, what addi-tional operators and notations can be introduced to facilitate equational reasoning of this monad,and put them to the test by proving a number of properties in our example problem: Spark aggre-gation.Much of this report is inspired by the author’s joint work with Chen et al. [2017], in which weformalised Spark, a platform for distributed computation, and derived properties under which adistributed Spark aggregation represents a deterministic computation. Therefore, many examplesin this report are about finding out when processing a non-deterministic permutation (simulatingarbitrary distribution of data) produces a deterministic result.
A monad consists of a type constructor M :: ∗ → ∗ and two operators return :: a → M a and “bind” ( = << ) :: ( a → M b ) → M a → M b that satisfy the following monad laws : f = << return x = f x , (1) return = << m = m , (2) f = << ( g = << m ) = ( 𝜆 x → f = << g x ) = << m . (3)Rather than the usual ( >> = ) :: M a → ( a → M b ) → M b , in the laws above we use the reversed bind ( = << ) , which is consistent with the direction of function composition and more readable when weprogram in a style that uses composition. When we use bind with 𝜆 -abstractions, it is more naturalto write m >> = 𝜆 x → f x . In this report we use the former more than the latter, thus the choice ofnotation. We also define m << m = const m = << m . Note that ( >> ) has type M a → M b → M b .More operators we find useful are given in Figure 1. Right-to-left Kleisli composition, denotedby ( < = < ) , composes two monadic operations a → M b and b → M c into an operation a → M c . Author’s address: Shin-Cheng Mu, Institute of Information Science, Academia Sinica, Taiwan, [email protected] Report TR-IIS-19-002, Institute of Information Science, Academia Sinica. Publication date: June 2019.
Shin-Cheng Mu ( < = < ) :: ( b → M c ) → ( a → M b ) → a → M c ( f < = < g ) x = f = << g x ( h $ i ) :: ( a → b ) → M a → M bf h $ i m = ( return · f ) = << m ( h•i ) :: ( b → c ) → ( a → M b ) → ( a → M c ) f h•i g = ( return · f ) < = < g Fig. 1. Some monadic operators we find handy for this paper.
Operators ( h $ i ) and ( h•i ) are monadic counterparts of function application and composition: ( h $ i ) applies a pure function to a monad, while ( h•i ) composes a pure function after a monadic function.We now introduce a collections of properties that allows us to rotate an expression that involvestwo operators and three operands. These properties will be handy when we need to move paren-thesis around in expressions. To begin with, the following properties show that ( h $ i ) and ( h•i ) shareproperties similar to pure function application and composition: ( f h•i g ) x = f h $ i g x , (4) f h $ i ( g h $ i m ) = ( f · g ) h $ i m , (5) f h•i ( g h•i h ) = ( f · g ) h•i h . (6)We also have the following law that allows us to rotate an expression that uses ( h•i ) and (·) : f h•i ( g · h ) = ( f h•i g ) · h . (7)Note that g in (7) must be a function returning a monad. Furthermore, (8) and (9) relate ( = << ) and ( h $ i ) , both operators applying functions to monads, while (10) and (11) relate ( < = < ) and ( h•i ) , bothoperators composing functions on monads: f = << ( g h $ i m ) = ( f · g ) = << m , (8) f h $ i ( g = << m ) = ( f h•i g ) = << m , (9) f < = < ( g h•i h ) = ( f · g ) < = < h , (10) f h•i ( g < = < h ) = ( f h•i g ) < = < h . (11)Having these properties is one of the advantages of writing ( = << ) and ( < = < ) backwards. All the prop-erties above can be proved by expanding definitions, and it is a good warming-up exercise provingsome of them. Some of them are proved in Appendix A.None of these operators and properties are strictly necessary: they can all be reduced to return , ( = << ) , and 𝜆 -abstractions. As is often the case when designing notations, having more operatorsallows ideas to be expressed concisely in a higher level of abstraction, at the expense of havingmore properties to memorise. It is personal preference where the balance should be. Properties (4)through (11) may look like a lot of properties to remember. In practice, we find it usually sufficientto let us be guided by types. For example, when we have f h $ i g x and want to bring f and g together, by their types we can figure out the resulting expression should be ( f h•i g ) x . Non-determinism Monad.
Non-determinism is the only effect we use in this report. We assumetwo operators ∅ and ( ) : the former denotes failure, while m n denotes that the computationmay yield either m or n . As pointed out by Gibbons and Hinze [2011], for proofs and derivations,what matters is not how a monad is implemented but what properties its operators satisfy. Whatlaws ∅ and ( ) should satisfy, however, can be a tricky issue. As discussed by Kiselyov [2015], it Technical Report TR-IIS-19-002, Institute of Information Science, Academia Sinica. Publication date: June 2019. quational Reasoning for Non-determinism Monad 3 eventually comes down to what we use the monad for. It is usually expected that ( a , ( ) , ∅) be amonoid. That is, ( ) is associative, with ∅ as its zero: ( m n ) k = m ( n k ) , ∅ m = m = m ∅ .It is also assumed that monadic bind distributes into ( ) from the end, while ∅ is a right zero for ( = << ) : f = << ( m m ) = ( f = << m ) ( f = << m ) , (12) f = << ∅ = ∅ . (13)For our purpose in this section, we also assume that ( ) is commutative ( m n = n m ) andidempotent ( m m = m ). Implementation of such non-determinism monads have been studied byFischer et al. [2011].Here are some induced laws about how ( h $ i ) interacts with return and non-determinism opera-tors: f h $ i return x = return ( f x ) , (14) f h $ i ∅ = ∅ , (15) f h $ i ( m m ) = ( f h $ i m ) ( f h $ i m ) . (16) As a warm-up example, the function perm non-deterministically computes a permutation of itsinput, using an auxiliary function insert that inserts an element to an arbitrary position in a list: perm :: [ a ] → M [ a ] perm [ ] = return [ ] perm ( x : xs ) = insert x = << perm xs , insert :: a → [ a ] → M [ a ] insert x [ ] = return [ x ] insert x ( y : xs ) = return ( x : y : xs ) (( y : ) h $ i insert x xs ) . For example, possible results of perm [ , , ] include [ , , ] , [ , , ] , [ , , ] , [ , , ] , [ , , ] ,and [ , , ] . Determinism.
The following lemma presents properties under which permuting the input listdoes not change the result of a foldr : Lemma 3.1.
Given (⊙) :: a → b → b. If x ⊙ ( y ⊙ z ) = y ⊙ ( x ⊙ z ) for all x , y :: a and z :: b, we havefoldr (⊙) z h•i perm = return · foldr (⊙) z . Since perm is defined in terms of insert , proof of Lemma 3.1 naturally depends on a lemma abouta related property of insert : Lemma 3.2.
Given (⊙) :: a → b → b, we havefoldr (⊙) z h•i insert x = return · foldr (⊙) z · ( x : ) , provided that x ⊙ ( y ⊙ z ) = y ⊙ ( x ⊙ z ) for all x , y :: a and z :: b. Technical Report TR-IIS-19-002, Institute of Information Science, Academia Sinica. Publication date: June 2019.
Shin-Cheng Mu
Proof.
Prove foldr (⊙) z h $ i insert x xs = return ( foldr (⊙) z ( x : xs )) . Induction on xs . Case xs : = [ ] : foldr (⊙) z h $ i insert x [ ] = { definition of insert } foldr (⊙) z h $ i return [ x ] = { by (14) } return ( foldr (⊙) z [ x ]) . Case xs : = y : xs : foldr (⊙) z h $ i insert x ( y : xs ) = { definition of insert } foldr (⊙) z h $ i ( return ( x : y : xs ) (( y : ) h $ i insert x xs )) = { by (16), (14), and (5) } return ( foldr (⊙) z ( x : y : xs )) (( foldr (⊙) z · ( y : )) h $ i insert x xs ) . Focus on the second branch of ( ) : ( foldr (⊙) z · ( y : )) h $ i insert x xs = { definition of foldr } (( y ⊙) · foldr (⊙) z ) h $ i insert x xs = { by (5) } ( y ⊙) h $ i ( foldr (⊙) z h $ i insert x xs ) = { induction } ( y ⊙) h $ i return ( foldr (⊙) z ( x : xs )) = { by (14) } return ( y ⊙ foldr (⊙) z ( x : xs )) = { definition of foldr } return ( y ⊙ ( x ⊙ foldr (⊙) z xs )) = { since x ⊙ ( y ⊙ z ) = y ⊙ ( x ⊙ z ) } return ( foldr (⊙) z ( x : y : xs )) . Thus we have ( foldr (⊙) z h•i insert x ) ( y : xs ) = { calculation above } return ( foldr (⊙) z ( x : y : xs )) return ( foldr (⊙) z ( x : y : xs )) = { idempotence of ( ) } return ( foldr (⊙) z ( x : y : xs )) . (cid:3) Proof of Lemma 3.1 then follows:
Proof.
Prove that foldr (⊙) z h $ i perm xs = return ( foldr (⊙) z xs ) . Induction on xs . Case xs : = [ ] : foldr (⊙) z h $ i perm [ ] = { definitions of perm } foldr (⊙) z h $ i return [ ] Technical Report TR-IIS-19-002, Institute of Information Science, Academia Sinica. Publication date: June 2019. quational Reasoning for Non-determinism Monad 5 = { by (14) } return ( foldr (⊙) z [ ]) . Case xs : = x : xs : foldr (⊙) z h $ i perm ( x : xs ) = { definition of perm } foldr (⊙) z h $ i ( insert x = << perm xs ) = { by (9) } ( foldr (⊙) z h•i insert x ) = << perm xs = { Lemma 3.2 } ( return · foldr (⊙) z · ( x : )) = << perm xs = { definitions of foldr and ( h $ i ) } (( x ⊙) · foldr (⊙) z ) h $ i perm xs = { by (5) } ( x ⊙) h $ i ( foldr (⊙) z h $ i perm xs ) = { induction } ( x ⊙) h $ i ( return ( foldr (⊙) z xs )) = { by (14) } return ( x ⊙ foldr (⊙) z xs ) = { definition of foldr } return ( foldr (⊙) z ( x : xs )) . (cid:3) Map, Filter, and Permutation.
It is not hard for one to formulate the following relationship be-tween map and perm , which is also based on a related property relating map and insert : Lemma 3.3. perm · map f = map f h•i perm. Lemma 3.4. insert ( f x ) · map f = map f h•i insert x. The lemma is true because map f is a pure computation — in reasoning about monadic programsit is helpful, and sometimes essential, to identify its pure segments, because these are the partsmore properties are applicable. Note that the composition (·) on the lefthand side is turned into ( h•i ) once we move map f leftwards.We prove only Lemma 3.4. Proof.
Prove by induction on xs that map f h $ i insert x xs = insert ( f x ) ( map f xs ) for all xs .We present only the inductive case xs : = y : xs : map f h $ i insert x ( y : xs ) = { definition of insert } map f h $ i ( return ( x : y : xs ) (( y : ) h $ i insert x xs )) = { by (16) and (14) } return ( map f ( x : y : xs )) ( map f h $ i (( y : ) h $ i insert x xs )) . For the second branch we reason: Lemma 3.3 and 3.4 are in fact free theorems of perm and insert [Voigtländer 2009]. They serve as good exercises,nevertheless.Technical Report TR-IIS-19-002, Institute of Information Science, Academia Sinica. Publication date: June 2019.
Shin-Cheng Mu map f h $ i (( y : ) h $ i insert x xs ) = { by (5) } ( map f · ( y : )) h $ i insert x xs = { definition of map } (( f y : ) · map f ) h $ i insert x xs = { by (5) } ( f y : ) h $ i ( map f h $ i insert x xs ) = { induction } ( f y : ) h $ i insert ( f x ) ( map f xs ) . Thus we have map f h $ i insert x ( y : xs ) = { calculation above } return ( f x : f y : map f xs ) (( f y : ) h $ i ( insert ( f x ) ( map f xs ))) = { definitions of insert and map } insert ( f x ) ( map f ( y : xs )) . (cid:3) One may have noticed that the style of proof is familiar: replace return x by [ x ] and ( ) by (++) , the proof is more-or-less what one would do for a list version of insert . This is exactly thepoint: the style of proofs we use to do for pure programs still works for monadic programs, aslong as the monad satisfies the demanded laws, be it a list, a more advanced implementation ofnon-determinism, or a monad having other effects.A similar property relating perm and filter can be formulated. Lemma 3.5. perm · filter p = filter p h•i perm. Its proof is routine and omitted. Finally, in a number of occasions it helps to know that xs is aresult of perm xs . The proof is also routine and omitted. Lemma 3.6.
For all xs we have that perm xs = return xs m for some m. Spark [Zaharia et al. 2012] is a popular platform for scalable distributed data-parallel computationbased on a flexible programming environment with high-level APIs, considered by many as thesuccessor of MapReduce. In a typical Spark program, data is partitioned and stored distributivelyon read-only
Resilient Distributed Datasets (RDDs) — we can think of it as a list of lists, whereeach sub-list is potentially stored on a remote node. On an RDD one can apply operations, called combinators , such as map , reduce , and aggregate . The aggregate combinator, for example, takesuser-defined functions (⊗) and (⊕) : (⊗) accumulates a sub-result for each data partition while (⊕) merges sub-results across different partitions.Programming in Spark, however, can be tricky. Since sub-results are computed across parti-tions concurrently, the order of their applications varies on different executions. Aggregation inSpark is therefore inherently non-deterministic. An example from Chen et al. [2017] showed thatcomputing the integral of 𝑥 from 𝑥 = − 𝑥 =
2, which should be 0, using a function in theSpark machine learning library, yields results ranging from − . . Technical Report TR-IIS-19-002, Institute of Information Science, Academia Sinica. Publication date: June 2019. quational Reasoning for Non-determinism Monad 7
Since a Spark aggregation is typically used to computes a list homomorphism [Bird 1987], we di-gress a little in this section to give a brief review and present some results that we will use. Afunction h :: List a → b is called a list homomorphism if there exists z :: b , k :: a → b , and (⊕) :: b → b → b such that: h [ ] = zh [ x ] = k xh ( xs ++ ys ) = h xs ⊕ h ys . That h is such a list homomorphism is denoted by h = hom (⊕) k z . Note that the properties aboveimplicitly demand that (⊕) be associative with z as its identity element.Lemma 4.1 and 4.2 below are about when a computation defined in terms of foldr is actually alist homomorphism. In Lemma 4.2, img f denotes the image of a function f . Lemma 4.1. h = hom (⊕) ( h · wrap ) z if and only if foldr (⊕) z · map h = h · concat, wherewrap x = [ x ] . Lemma 4.2.
Let (⊕) :: b → b → b be associative on img ( foldr (⊗) z ) with z as its identity, where (⊗) :: a → b → b. We have foldr (⊗) z = hom (⊕) (⊗ z ) z if and only if x ⊗ ( y ⊕ w ) = ( x ⊗ y ) ⊕ wfor all x :: a and y , w ∈ img ( foldr (⊗) z ) . Notice, in Lemma 4.2, that (⊗ z ) = foldr (⊗) z · wrap . Proofs of both lemmas are interestingexercises, albeit being a bit off-topic. They are recorded in Appendix A. Distributed collections of data are represented by
Resilient Distributed Datasets (RDDs) in Spark.Informally, an RDD is a collection of data entries; these data entries are further divided into parti-tions stored on different machines. Abstractly, an
RDD can be seen as a list of lists: type
Partition a = [ a ] , type RDD a = [ Partition a ] , where each Partition may be stored in a different machine.While Spark provides a collection of combinators (functions on
RDD s that are designed to becomposed to form larger programs), in this report we focus on a particular one, aggregate . It canbe seen as a parallel implementation foldr . The combinator processes an
RDD in two levels: eachpartition is first processed locally on one machine by foldr (⊗) z . The sub-results are then com-municated and combined — this second step can be think of as another foldr with (⊕) . Spark programmers like to assume that their programs are deterministic. To exploit concurrency,however, the sub-results from each machine might be processed in arbitrary order and the resultcould be non-deterministic. The following is our characterisation of aggregate , where we use perm to model the fact that sub-results from each machine are processed in unknown order: aggregate :: b → ( a → b → b ) → ( b → b → b ) → RDD a → M baggregate z (⊗) (⊕) = foldr (⊕) z h•i ( perm · map ( foldr (⊗) z )) . It is clear from the types that foldr (⊗) z and foldr (⊕) z are pure computations, and non-determinism is introduced solely by perm . In fact, the actual Spark aggregation (and that modelled in Chen et al. [2017]) are like foldl . For convenience in our proofswe see all list operations the other way round and use foldr . This is not a fundamental difference.Technical Report TR-IIS-19-002, Institute of Information Science, Academia Sinica. Publication date: June 2019.
Shin-Cheng Mu
Deterministic Aggregation.
We are interested in finding out conditions under which aggregate produces deterministic outcomes.
Theorem 4.3.
Given (⊗) :: a → b → b and (⊕) :: b → b → b, where (⊕) is associative andcommutative, we have:aggregate z (⊗) (⊕) = return · foldr (⊕) z · map ( foldr (⊗) z ) . Proof.
We reason: aggregate z (⊗) (⊕) = { definition of aggregate } foldr (⊕) z h•i ( perm · map ( foldr (⊗) z )) = { by (7) } ( foldr (⊕) z h•i perm ) · map ( foldr (⊗) z ) = { Lemma 3.1, since (⊕) is associative and commutative } return · foldr (⊕) z · map ( foldr (⊗) z ) . (cid:3) The following corollary summaries the results and present conditions under which aggregate computes a homomorphism.
Corollary 4.4. aggregate z (⊗) (⊕) = return · hom (⊕) (⊗ z ) z · concat, provided that (⊕) isassociative, commutative, and has z as identity, and that x ⊗ ( y ⊕ w ) = ( x ⊗ y ) ⊕ w for all x :: a andy , w ∈ img ( foldr (⊗) z ) . Proof.
We reason: aggregate z (⊗) (⊕) = { Theorem 4.3 } return · foldr (⊕) z · map ( foldr (⊗) z ) = { foldr (⊗) z = hom (⊕) (⊗ z ) z by Lemma 4.2; Lemma 4.1 } return · hom (⊕) (⊗ z ) z · concat . (cid:3) Determinism Implies Homomorphism.
The final part of the report deals with an opposite ques-tion: what can we infer if we know that aggregate is deterministic? To answer that, however, weneed to assume two more properties: m m = return x ⇒ m = m = return x . (17) return x = return x ⇒ x = x . (18)Property (17) can be seen as the other direction of idempotency of ( ) , while (18) states that return is injective.The following lemma can be understood this way: when aggregate z (⊗) (⊕) , which couldbe non-deterministic, can be performed by a deterministic function, the operator (⊕) should beinsensitive to ordering: Lemma 4.5.
If aggregate z (⊗) (⊕) = return · foldr (⊗) z · concat, and perm xss = return yss mfor some m, we have Technical Report TR-IIS-19-002, Institute of Information Science, Academia Sinica. Publication date: June 2019. quational Reasoning for Non-determinism Monad 9 foldr (⊗) z ( concat xss ) = foldr (⊕) z ( map ( foldr (⊗) z ) xss ) = foldr (⊕) z ( map ( foldr (⊗) z ) yss ) . Proof.
We reason: return · foldr (⊗) z · concat $ xss = { assumption } aggregate z (⊗) (⊕) $ xss = { definition of aggregate , Lemma 3.3, and (6) } ( foldr (⊕) z · map ( foldr (⊗) z )) h $ i perm xss = { assumption: perm xss = return yss m , by (16) and (14) } ( return · foldr (⊕) z · map ( foldr (⊗) z ) $ yss ) (( foldr (⊕) z · map ( foldr (⊗) z )) h $ i m ) . Thus by (17) and (18), foldr (⊗) z · concat $ xss equals foldr (⊕) z · map ( foldr (⊗) z ) $ yss . The formeralso equals foldr (⊕) z · map ( foldr (⊗) z ) $ xss because, by Lemma 3.6, perm xss = return xss m for some m . (cid:3) Based on Lemma 4.5, the following theorem explicitly states that (⊕) should be associative,commutative, and has z as its identity in restricted domain. Theorem 4.6.
If aggregate z (⊗) (⊕) = return · foldr (⊗) z · concat, we have that (⊕) , whenrestricted to values in img ( foldr (⊗) z ) , is associative, commutative, and has z as its identity. Proof.
In the discussion below, let x , y , and w be in img ( foldr (⊗) z ) . That is, there exists xs , ys , and ws such that x = foldr (⊗) z xs , y = foldr (⊗) z ys , and w = foldr (⊗) z ws . Identity . We reason: y = foldr (⊗) z ( concat [ xs ]) = { perm [ xs ] = return [ xs ] ∅ , Lemma 4.5 } foldr (⊕) z ( map ( foldr (⊗) z ) [ xs ]) = y ⊕ z . Thus z is a right identity of (⊕) . Similarly, y = foldr (⊗) z ( concat [ [ ] , xs ]) = { perm [ [ ] , xs ] = return [ [ ] , xs ] m , Lemma 4.5 } foldr (⊕) z ( map ( foldr (⊗) z ) [ [ ] , xs ]) = z ⊕ ( y ⊕ z ) = { z is a right identity of (⊕) } z ⊕ y . Thus z is also a left identity of (⊕) . Commutativity . We reason: x ⊕ y = { z is a right identity } x ⊕ ( y ⊕ z ) = foldr (⊕) z ( map ( foldr (⊗) z ) [ xs , ys ]) Technical Report TR-IIS-19-002, Institute of Information Science, Academia Sinica. Publication date: June 2019. = { perm [ xs , ys ] = return [ ys , xs ] m , Lemma 4.5 } foldr (⊕) z ( map ( foldr (⊗) z ) [ ys , xs ]) = y ⊕ ( x ⊕ z ) = { z is a right identity } y ⊕ x . Associativity . We reason: x ⊕ ( y ⊕ w ) = { z is a right identity } x ⊕ ( y ⊕ ( w ⊕ z )) = foldr (⊕) z ( map ( foldr (⊗) z ) [ xs , ys , ws ]) = { (⊕) commutative } foldr (⊕) z ( map ( foldr (⊗) z ) [ ws , xs , ys ]) = w ⊕ ( x ⊕ ( y ⊕ z )) = { z is a right identity } w ⊕ ( x ⊕ y ) = { (⊕) commutative } ( x ⊕ y ) ⊕ w . (cid:3) Theorem 4.7.
If aggregate z (⊗) (⊕) = return · foldr (⊗) z · concat, we have foldr (⊗) z = hom (⊕) (⊗ z ) z. Proof.
Apparently foldr (⊗) z [ ] = z and foldr (⊗) z [ x ] = x ⊗ z . We are left with provingthe case for concatenation. foldr (⊗) z ( xs ++ ys ) = foldr (⊗) z ( concat [ xs , ys ]) = { Lemma 4.5 } foldr (⊕) z ( map ( foldr (⊗) z ) [ xs , ys ]) = foldr (⊗) z xs ⊕ ( foldr (⊗) z ys ⊕ z ) = { Theorem 4.6, z is identity } foldr (⊗) z xs ⊕ foldr (⊗) z ys . (cid:3) Corollary 4.8.
Given (⊗) :: a → b → b and (⊕) :: b → b → b. aggregate z (⊗) (⊕) = return · foldr (⊗) z · concat if and only if ( img ( foldr (⊗) z ) , (⊕) , z ) forms a commutative monoid,and that foldr (⊗) z = hom (⊕) (⊗ z ) z. Proof.
A conclusion following from Theorem 4.3, Theorem 4.6, and Theorem 4.7. (cid:3)
Acknowledgements.
In around late 2016, Yu-Fang Chen, Chih-Duo Hong, Ondřej Lengál, NishantSinha and Bow-Yaw Wang invited me into their project formalising Spark. It was what inspiredmy interests in reasoning about monads, which led to a number of subsequent work. The initialproofs of properties of aggregate and other combinators were done by Ondřej Lengál, withoutusing monads. Affeldt et al. [2019] modelled a hierarchy of monadic effects in Coq. The formali-sation was applied to verify a number of equational proofs of monadic programs, including someof the proofs in an earlier version of this report. I am solely responsible for any remaining errors,however.
Technical Report TR-IIS-19-002, Institute of Information Science, Academia Sinica. Publication date: June 2019. quational Reasoning for Non-determinism Monad 11
REFERENCES
Reynald Affeldt, David Nowak, and Takafumi Saikawa. 2019. A hierarchy of monadic effects for program verification usingequational reasoning. In
Mathematics of Program Construction , Graham Hutton (Ed.). Springer.Richard S. Bird. 1987. An introduction to the theory of lists. In
Logic of Programming and Calculi of Discrete Design , ManfredBroy (Ed.). Number 36 in NATO ASI Series F. Springer-Verlag, 3–42.Yu-Fang Chen, Chih-Duo Hong, Ondřej Lengál, Shin-Cheng Mu, Nishant Sinha, and Bow-Yaw Wang. 2017. An executablesequential specification for Spark aggregation. In
International Conference on Networked Systems . Springer-Verlag.Sebastian Fischer, Oleg Kiselyov, and Chung-chieh Shan. 2011. Purely functional lazy nondeterministic programming.
Journal of Functional Programming
21, 4-5 (September 2011), 413–465.Jeremy Gibbons and Ralf Hinze. 2011. Just do it: simple monadic equational reasoning. In
International Conference onFunctional Programming , Olivier Danvy (Ed.). ACM Press, 2–14.Oleg Kiselyov. 2015. Laws of MonadPlus. http://okmij.org/ftp/Computation/monads.html
International Conference on Functional Pro-gramming , Andrew Tolmach (Ed.). ACM Press, 173–184.Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J. Franklin,Scott Shenker, and Ion Stoica. 2012. Resilient distributed datasets: a fault-tolerant abstraction for in-memory clustercomputing. In
Networked Systems Design and Implementation , Steven Gribble and Dina Katabi (Eds.). USENIX.
A MISCELLANEOUS PROOFS
Proving (8) . f = << ( g h $ i m ) = ( f · g ) = << m . Proof.
We reason: f = << ( g h $ i m ) = { definition of ( h $ i ) } f = << (( return · g ) = << m ) = { monad law (3) } ( 𝜆 x → f = << return ( g x )) = << m = { monad law (1) } ( 𝜆 x → f ( g x )) = << m = ( f · g ) = << m . (cid:3) Proving (5) . f h $ i ( g h $ i m ) = ( f · g ) h $ i m . Proof.
We reason: f h $ i ( g h $ i m ) = { definition of ( h $ i ) } ( return · f ) = << ( g h $ i m ) = { by (8) } ( return · f · g ) = << m = { definition of ( h $ i ) } ( f · g ) h $ i m . (cid:3) For the next results we prove a lemma: ( f = << ) · ( g = << ) = ((( f = << ) · g ) = << ) . (19) ( f = << ) · ( g = << ) = { 𝜂 intro. } Technical Report TR-IIS-19-002, Institute of Information Science, Academia Sinica. Publication date: June 2019. ( 𝜆 m → f = << ( g = << m )) = { monad law (3) } ( 𝜆 m → ( 𝜆 y → f = << g y ) = << m ) = { 𝜂 reduction } ((( f = << ) · g ) = << ) . Proving (6) . f h•i ( g h•i m ) = ( f · g ) h•i m . Proof.
We reason: f h•i ( g h•i m ) = { definition of ( h•i ) } (( return · f ) = << ) · (( return · g ) = << ) · m = { by (19) } (((( return · f ) = << ) · return · g ) = << ) · m = { monad law (1) } (( return · f · g ) = << ) · m = { definition of ( h•i ) } ( f · g ) h•i m . (cid:3) Proving (10) . f < = < ( g h•i h ) = ( f · g ) < = < h . Proof.
We reason: f < = < ( g h•i h ) = { definitions of ( < = < ) } ( f = << ) · (( return · g ) = << ) · h = { by (19) } ((( f = << ) · return · g ) = << ) · h = { monad law (1) } (( f · g ) = << ) · h = { definition of ( < = < ) } ( f · g ) < = < h . (cid:3) Proving (11) . f h•i ( g < = < h ) = ( f h•i g ) < = < h . Proof.
We reason: f h•i ( g < = < h ) = { definitions of ( < = < ) and ( h•i ) } (( return · f ) = << ) · ( g = << ) · h = { by (19) } (((( return · f ) = << ) · g ) = << ) · h = { definition of ( h•i ) } (( f h•i g ) = << ) · h = { definition of ( < = < ) } ( f h•i g ) < = < h . Technical Report TR-IIS-19-002, Institute of Information Science, Academia Sinica. Publication date: June 2019. quational Reasoning for Non-determinism Monad 13 (cid:3)
Proof of Lemma 4.1.
Proof.
A Ping-pong proof.
Direction (⇒) . Let h = hom (⊕) ( h · wrap ) z , prove foldr (⊕) z ( map h xss ) = h ( concat xss ) by induction on xss . Case xss : = [ ] : foldr (⊕) z ( map h [ ]) = foldr (⊕) z [ ] = z = h ( concat [ ]) . Case xss : = xs : xss : foldr (⊕) z ( map h ( xs : xss )) = h xs ⊕ foldr (⊕) z ( map h xss ) = { induction } h xs ⊕ h ( concat xss ) = { h homomorphism } h ( concat ( xs : xss )) . Direction (⇐) . Assuming foldr (⊕) z ( map h xss ) = h ( concat xss ) , prove that h = hom (⊕) ( h · wrap ) z . Case empty list: h [ ] = h ( concat [ ]) = { assumption } foldr (⊕) z ( map h [ ]) = z . Case singleton list: certainly h [ x ] = h [ x ] . Case concatentation: h ( xs ++ ys ) = h ( concat [ xs , ys ]) = { assumption } foldr (⊕) z ( map h [ xs , ys ]) = h xs ⊕ ( h ys ⊕ z ) = h xs ⊕ h ys . (cid:3) Proof of Lemma 4.2.
Proof.
A Ping-pong proof.
Direction (⇐) . We show that foldr (⊗) z = hom (⊕) (⊗ z ) z , provided that x ⊗ ( y ⊕ w ) = ( x ⊗ y ) ⊕ w .It is immediate that foldr (⊗) z [ ] = z around foldr (⊗) z [ x ] = x ⊗ z . For xs ++ ys , note that foldr (⊗) z ( xs ++ ys ) = foldr (⊗) ( foldr (⊗) z ys ) xs .The aim is thus to prove that Technical Report TR-IIS-19-002, Institute of Information Science, Academia Sinica. Publication date: June 2019. foldr (⊗) ( foldr (⊗) z ys ) xs = ( foldr (⊗) z xs ) ⊕ ( foldr (⊗) z ys ) . We perform an induction on xs . The case when xs : = [ ] trivially holds. For xs : = x : xs , we reason: foldr (⊗) ( foldr (⊗) z ys ) ( x : xs ) = x ⊗ foldr (⊗) ( foldr (⊗) z ys ) xs = { induction } x ⊗ (( foldr (⊗) z xs ) ⊕ ( foldr (⊗) z ys )) = { assumption: x ⊗ ( y ⊕ w ) = ( x ⊗ y ) ⊕ w } ( x ⊗ ( foldr (⊗) z xs )) ⊕ ( foldr (⊗) z ys ) = ( foldr (⊗) z ( x : xs )) ⊕ ( foldr (⊗) z ys ) . Direction (⇒) . Given foldr (⊗) z = hom (⊕) (⊗ z ) z , prove that x ⊗ ( y ⊕ w ) = ( x ⊗ y ) ⊕ w for y and w in the range of foldr (⊗) z .Let y = foldr (⊗) z xs and w = foldr (⊗) z ys for some xs and ys . We reason: x ⊗ ( y ⊕ w ) = x ⊗ ( foldr (⊗) z xs ⊕ foldr (⊗) z ys ) = { since foldr (⊗) z = hom (⊕) (⊗ z ) z } x ⊗ ( foldr (⊗) z ( xs ++ ys )) = foldr (⊗) z ( x : xs ++ ys ) = { since foldr (⊗) z = hom (⊕) (⊗ z ) z } foldr (⊗) z ( x : xs ) ⊕ foldr (⊗) z ys = ( x ⊗ foldr (⊗) z xs ) ⊕ foldr (⊗) z ys = ( x ⊗ y ) ⊕ w . (cid:3)(cid:3)