Armiarma: Ethereum2 Network Monitoring Tool
AArmiarma: Ethereum2 Network Monitoring Tool
Mikel Cortes-Goicoechea
Barcelona Supercomputing Center
Barcelona, [email protected]
Leonardo Bautista-Gomez
Barcelona Supercomputing Center
Barcelona, [email protected]
Abstract —Achieving the equilibrium between scalability, sus-tainability and security has prevailed as the ideal solution forthe decentralized blockchain applications over the last years.Several approaches have been proposed being Ethereum a solidproposal among them. Ethereum is on the path of a majorprotocol improvement called Ethereum 2.0 (Eth2), implementingSharding and introducing the Proof-of-Stake (PoS). As the changeof consensus mechanism is a delicate matter, this improvementwill be achieved through different phases, first of which is theimplementation of the Beacon Chain. The implementation of thelatest has been stated with the recent launch of the Eth2 mainnet. In this work, we introduce an Eth2 network monitor tool,called Armiarma, used to generate a complete analysis of the p2pnetwork of the Eth2 main net. In this paper, we present some ofthe results of what this Eth2 network monitor can achieve.
Index Terms —Blokchain, Ethereum2, Eth2, Beacon Chain,Sharding, Monitoring, Proof of Stake, Smart Contracts, Scaling
I. I
NTRODUCTION
Ethereum [1] is a mature decentralized web infrastructurethat has proved reliability and had led the growth of decen-tralized applications. It features an ecosystem for decentralizedapplications based on a general purpose virtual machine andits dedicated programming language. This ecosystem holdscertain characteristics, which has generated a large communityof developers that have proposed new methods to tackle someof the limitations affecting the Ethereum protocol.The Ethereum Foundation together with the Ethereum com-munity have been working on the development of the secondgeneration of Ethereum, Ethereum 2.0 (Eth2). The approachproposed on the Eth2 protocol tackles both: the scalabilitychallenge by introducing the sharding, and the sustainabilitychallenge by changing the consensus protocol to Proof-of-Stake (PoS). This ambitious project is divided into differentphases because of its complexity . Taking place, with thelaunch of the Eth2 main net, the beginning of the project orphase 0, the Beacon Chain [2].Eth2 aims to create a major network of clients that willprovide the infrastructure for sharding. The different developerteams have been working over the last two years to generateEth2 client implementations, each client targeting a specifictype of user. This variety of clients, together with the lowcomputing-power required by the Eth2 protocol, offers to mostof the users the possibility to join the network.In contrast to the Ethereum1 client, the Eth2 one is dividedin two main parts, the Beacon Node and the Validator. Thebeacon node, or node, performs all kind of interactions with the Eth2 network. It establishes the connections with othernodes, exchanges the messages, downloads the blocks andmaintains the view of the Beacon Chain as the view of theShards that the validator is performing at, building amongthem the entire network. On the other hand, the validator isin charge of the logical part, generating the attestations on thereceived blocks, as well as proposing the blocks when needed.To perform its duties, the validator needs to follow the currentstate of the Chain, and to be in constant communication witha Beacon Node that provides such information.A project with this complexity level completely relies on thehealthiness of the network. To propagate the messages betweenthe nodes of the network, Eth2 relies on the GossipSubp2p protocol [3]. The protocol is based on the exchange ofmessages and metadata between peers distributed in meshes,achieving the message propagation within the targeted time-margins and minimizing wasteful bandwidth consumption.Given the Eth2 network requirements and the interactionbetween the clients, it is important to monitor critical pointsfor possible attacks. In this paper, we introduce a networkmonitoring tool that we call Armiarma, used to generate acomplete analysis of the p2p network of the Eth2 main net.he tool is capable of showing a general overview of thecomposition of the network, and is also capable of doing adetailed analysis of a specific node.The remainder of this paper is organized as follows. Sec-tion II discusses the background and related work. Section IIIexplains the methodology used for our study and evaluation.In Section IV we show and analyze the results obtained byour tool. Finally, Section V concludes this work and presentssome possible future directions.II. S
TATE O F T HE A RT On a large decentralized blockchain that involves so manynodes and clients, the security and integrity of the protocolrelies on the healthiness of the network. Being able to debugfrom inside all the occurring events, offers the possibility toget an in-depth view of the network status and might even helpto prevent performance issues as well as more serious networkreliability problems or even security vulnerabilities. Escapingfrom the complex and tedious work that would mean generatespecific test cases for each of the clients, the community havetried to develop tools to interact easily with the network,without having to operate as a full client.1 a r X i v : . [ c s . CR ] D ec . Rumor One of the those tools is Rumor. Rumor appears as aninteractive shell script tool for debugging and testing theinteraction between the Eth2 clients. Originally started as aEth2 networking tool written in Go, Rumor was designedas an alternative to become the common platform for testingthe different Eth2 nodes in a less effort and time-consumingway. By offering a set of commands that can replicate mostof the Eth2 network protocol interactions, Rumor offers theabstraction from the code implementation that makes, settinga custom node, possible by simply tipping a few commands.The platform is based on an environment close to theBash shell that eases the test building labour, where some ofthe Bash syntax (e.g., variables and control flow statements)are accepted. Along with the scripting capabilities, Rumorincludes a complex system of sub-environments (known inRumor as actors) that allows to generate different nodes withdifferent properties simultaneously. The different nodes dis-tributed along the different actors share the common environ-ment, which perform as link between them. This environmentsettles the possibility for the peers to exchange information,needed to perform a synchronization of actions based onconstantly occurring events.Due to its debugging purposes, most of the commands of theplatform offer the debugging and testing related informationin a log format. This logs, configurable most of the time indifferent levels (error, debug, info, etc..), are the ones thatprovide the insights of the processes that are taking placeon the node. The governance over the node specs and itsconduct makes Rumor the appropriate tool for testing anddebugging the Eth2 clients and their behaviour under differentconditions. The variety of commands that Rumor offers is stillon a developing stage, but it already includes most of Eth2networking specs such: • Generate a ENR identity for the node needed to partici-pate on some of the Eth2 protocols. • The Eth peer discovery protocol dv5.1 . • Generate a peer database or peerstore of the connectedand discovered peers. • Establish a connection with a given peer. • Exchange Status and Metadata of the Beacon State withthe connected peers. • Listen and publish messages on the GossipSub p2pprotocol used in Eth2. • Send and respond to RPCs from and to the peers. • Import blocks and states from the Beacon Chain.The different hosts configurations and behaviour strategiescan be deployed in different ways. If the desired test is nottoo long, it can be launched from the rumor shell by typingthe commands in the desired order. For more repetitive tasksor tests, the platform offers the possibility to read the list ofcommands from a given file (e.g., script.rumor ). For a morecomplex solutions, Rumor also offers the option to add thegiven test-plan into a service, allowing the test-case to behaveas a server.
B. Node Discovery Protocol v5
As a decentralized platform, Eth2 tries to avoid any pointof centralization inside the proposed protocol. The networkingarea, and more specifically the peer discovery, is not anexception to this rule. The implemented GossipSub protocol,despite being a protocol oriented to a message propagationon decentralized applications, does not offer any kind ofpeer discovery service. Leaving that task in charge of theapplication. Eth2 developed its own node discovery protocol,Discovery 5 ( dv5 ), currently on its 5.1 version [4].Dv5 focuses on the Ethereum Node Records (ENR) ex-change along the peers. This node records, including the IPand ports needed to establish a connection, are recorded in aDistributed Hash Table (DHT). This DTH offers the possibilityto sample and search along the generated database, allowing toeasily update the node record of a specific peer if modificationsare detected. As entry point to the dv5 protocol, the nodes canstart searching for others that publicly advertise themselveson specific topics, or as an alternative that improves theperformance, a first connection with boot-nodes is possible.The peer discovery service is essential for the networkin general. New peers joining the network need connectionsfrom where synchronize the chain. Peers that are already onthe network also need from time to time to actualize theirinformation source, ensuring that they are properly followingthe head of the chain. Furthermore, to increase the resilienceof the network to Sybil attacks, dv5 serves a record of nodeson the network that the node could connect at any point.III. M
ETHODOLOGY
The approach proposed on this paper tackles the lack ofinformation about the performance of the GossipSub protocoland the peers on the Eth2 main net. Armiarma is built ontop of Rumor and it offers a simple yet powerful method thatprovides meaningful data about the p2p network and the Eth2clients. The approach is divided into two parts, the Armiarmacrawler and the Armiarma analyzer, as described bellow.
A. Armiarma Crawler
Armiarma Crawler is the data gatherer part of the tool. Thecrawler has been based on the Eth2 client debugging toolRumor, and its performance is linked to its ability to discoverand peer with clients from the Eth2 network. Launching thecrawler has been summarized in a chain of commands thatexecute the initialization from a bunch of files. Having ahead file that initializes all the entire crawling process, therest of the files get called from the main one performing thehost initialization. This process starts with the host set up,by assigning the IP and ports that will be used by the host.Followed by the ENR generation, the Eth2 ENR identity getsgenerated from the host info and by specifying the networkthat we want to belong to, that in this case is the main net[5]. Once we advertise on the ENR of the crawler that weparticipate on the Eth2 main net, a Beacon Node needs tobe faked. Claiming to be at the genesis state prevents thecrawler from being penalized for not offering services from2he beacon chain as blocks synchronization. For successfullyfaking an Eth2 client and being accepted by the rest of thepeers, the custom host has been set up to be at the genesis stateof the Beacon Chain. After setting the host, the GossipSubp2p protocol is initialized, joining the main topics from theEth2 Specs. The Rumor GossipSub implementation has beenmodified [6] to store all the data gathered from the peers weget connected to, getting from each of them:1) Information about the peer: • Peer Id • Node Id • Client Type and Client Version • Pubkey • MultiAddress • IP • Country • City • Latency2) The connection/disconnection events with the timestampof each event.3) A Counter of every message we have received from eachpeer on the five main GossipSub topics of Eth2: • BeaconBlock • BeaconAggregateAndProof • VoluntaryExit • ProposerSlashing • AttesterSlashingOnce the host is fully initialized, the peer discovering pro-tocol dv5 [4] and the connectall
Rumor services are launched.This way, all the peers that we are able to find throughthe dv5 protocol are recorded into a peerstore and attemptedto be connected. From all of those peers that we establisha connection, the crawler would start to get the messagesfrom the GossipSub topics previously joined, generating aswell the metrics previously mentioned. To export the metrics,a new command has been added to Rumor, exporting thepeerstore and the metrics every given time interval. As theArmiarma Implementation is constantly improving and addingnew features or gathering new metrics from the network, itssynergy with Rumor also leads Rumor to be continuouslyimproving its implementation with new features.
B. Armiarma Analyzer
While Armiarma crawler is focused on the interaction withthe p2p network, recording into a json file all the events andthe peer information, Armiarma analyzer gets in charge ofperforming an analysis from the raw information gathered.At the moment, the analysis gets performed in a subsequentmoment after the metrics get exported from the crawler. In acombination between the read peerstore and the metrics fromthe GossipSub implementation, the analyzer parses the peerinformation individually, ending up in a large panda databaseof the crawled peers. During the parsing process, the format ofseveral items gets modified, aiming to ease the data analysis.One of those format modifications are the recorded connec-tions and disconnections events that get counted. From there, the total connection time to each peer is obtained. As explainedin Section III-A, the crawler supports the main five topics fromthe Eth2 p2p network. Because of this, the crawler usuallyreports connections and disconnections in batches of five,mostly at the same exact time. To prevent an over-counting ofthe events, and most importantly, to get an accurate connectiontime per peer, the events of the same type (i.e., connection ordisconnection) and almost the same time (i.e., inside of a time-window of 500ms) are counted only once. This helps to get acloser representation of the actual number of connections andtheir duration.Once the parsing process has been successfully achieved,the analyzer offers several tools for filtering and plotting frompandas. That eases the chart representation of desired metricsfrom the database. The obtained compilation of graphs andtheir insights will be discussed on the following section IV.TABLE I: Crawler Running Period
Event DateInitial Date 2020-12-12 16:25:54Finishing Date 2020-12-14 13:34:26
IV. A
NALYSIS
In order to test the capabilities of the Armiarma tool weperformed an experiment in which we let the crawler run forseveral days collecting as much data as possible. Following thelow processing-power oriented philosophy on the Eth2 project,the crawler has been set on a ARM based Raspberry Pi 4 with4GB of RAM and with a standard 1 Gbit Ethernet connection.The exact running period of the experiment is shown in table I.The discussion of the results obtained from the analysis havebeen divided in the three subsections described bellow.Fig. 1: Comparison between the number of peers connectedand the number of peers in the peerstore.
A. Eth2 Network Analysis
In blockchain protocols as Eth2, the security of the networkrelays on a high number of nodes working together. To havea stable performance, the network has a margin of peers that3an go offline, without necessarily leading to the crash on theprotocol. In this case, Eth2 needs at least 66% of the clientsactively attesting on the generated blocks to finalize.Fig. 2: Number of peers classified by country of origin.The first part of the network analysis consists on having thecrawler to get a list of peers and attempt connecting to them.We can observe on figure 1 that from the amount of peerswe obtained from the dv5 protocol, the crawler wasn’t ableto connect with all of them. That is normal considering thatnot all the peers are connected all the time and most clientshave restrictions in the number of peers they can connect with,therefore rejecting additional connections. Also, this could berelated to a non port forwarding configuration by those peers,preventing us to initiate the peering handshake.Fig. 3: Number of connections established with each peer.Blockchain networks are geographically distributed allaround the world. Having the peers distributed, means that thepeers are less likely to get affected by the same type of regionalproblems or events (e.g., National censorship). On figure 2,we can appreciate that despite having a large concentrationof peers in two countries, the US and Germany, the peersdistribution is mostly homogeneous.Taking a closer look, on Figures 3 and 4, we can observethat not all the peers connected have had the same interaction Fig. 4: Number of disconnections established with each peer.with the crawler. Note that the number of connections and dis-connections registered by the crawler are based on the numberof GossipSub topics supported by the connected peer, in thiscase all the clients supports all the five main Eth2 protocols.The difference on the time we have been connected to thepeers, shown on Figure 5 can be related to the pruning processthat takes place on the GossipSub protocol. This process aimsto keep the predefined range of peer connections. Note that tokeep the computing-power and network bandwidth low, severalclient implementations reduce the maximum number of peersthe node is allowed to be connected to.Fig. 5: Amount of time connected with each peer.In a decentralized network such as Eth2, low latency con-nections are essential for tracking every occurring event as fastas posible. In this case, on the Eth2 Beacon Chain protocol,there is a new Slot event every 12 seconds. Meaning thatevery 12 seconds a randomly chosen validator will have thechance to propose a Beacon Block and therefore receive thetoken reward. On figure 6, we can observe the latency of theconnections that the crawler recorded while being connectedto the peers. We can observe that the majority stays with alatency below 3 seconds, having an outlier with a 31 secondslatency. There are two things we should take into account whenanalyzing the latency: • The crawler is not a fully functional client. Its purposeis to provide insights about the network and therefore, it4ries to stay connected with as many peers as possible.Meanwhile, a client would focus on keeping a low rangeof peers (30-60) that would be chosen based on theirspecs (latency, message sharing, etc.). • The crawling process was done on a raspberry Pi locatedin Spain. Thus, peers located on the other side of theworld would usually report a higher latency, e.g. theoutlier peer with an IP from USA (See Section IV-C).Fig. 6: Connection latency with each peer.
B. Eth2 Clients Interaction
Currently, there are five Eth2 clients available for participat-ing on the network. With Armiarma, we were able to detectwhich client is used by each connected peer, and it is evenpossible to get the version they are using, as shown in Figure 7and Table II. Note that we did not received any connectionfrom any Lodestar client, but got connected to 26 peers from unknown origin. This
Unknown peers could participate on thenetwork as bootnodes, other crawlers, possible attackers or justclients that did not set the user agent field.Fig. 7: Number of peers connected from each client.As we can see in Figure 7, there is a large number of peersfrom Lighthouse. In contrast, on the peerstore we observethat 55.17% of the peers use the TCP port used by Prysmclients ( ), showing that the majority of nodes in the Eth2 network use Prysm. There are several reasons for thisdisparity. First, the figure does not show all the peers thecrawler discovered , but only those it managed to established aconnection with. Lighthouse and Teku clients usually accepthigher number of peers and are more flexible in the numberof peers they keep. Prysm on the other hand, has a morestrict limit on the number of peers they accept. Although theyhave flags to change the maximum number of peers, mostnodes usually run with the default parameters. Finally, issuesrelated to port-forwarding could have also played a role andprevented some connections from Prysm nodes. Nevertheless,the distribution of Eth2 clients is not the focus of this paper,but the capabilities of the network crawler. With Armiarma,we could see how many version of each client are out therein the wild, as shown in Table II. This is useful as we canobserve over a long period of time, how client teams deploynew versions and how fast users update their clients.TABLE II: Number of versions observed of each client.
Lighthouse Teku Nimbus Prysm Lodestar Unknown5 5 2 5 0 1
On the comparison of the behaviour of the different clientswith the crawler, we can observe several differences. First wecompare the average number of connections and disconnec-tions for each one of the clients, as shown in Figure 8 and 9.Lighthouse, Teku, Nimbus and the Unknown clients obtain anrelatively close number of connection events that goes from620 to 840 on average per node, being Teku the client thatobtains the highest number of dialing events. Once again,the low number of events from the Prysm nodes stands out.Despite connecting to the same number of peer as Nimbus,Prysm nodes have dialed x23 times less to our crawler.Fig. 8: Average connections to peers classified by client types.The unexpected number of dialing events seen from Prysmnodes, brings up again the different peering strategies adoptedby the clients while implementing the GossipSub protocol. Although it is possible to change the dialing port of the host in all theclients, the percentage of users that change it to this one specifically is low. chatty ofthe clients. Given that we are measuring who talks first tothe crawler, we suspected that perhaps in this case, networklatency could have an important impact. So we studied theaverage latency for each client. As we can observe in Figure15, both Prysm and Nimbus have the lowest latency rate on average. This fits well with the clients that have the highestaverage number of messages, explaining the phenomena weobserve. Overall, we can still see that, despite having ahigher average latency rate, Lighthouse and Teku are stillable to provide a significant number of messages, showingcontribution to the network.Fig. 15: Average latency with peers classified by client types.Leaving the type of client aside, we also wanted to seeif all peers communicate in relatively similar amounts or ifthere was any node communicating significantly more thanothers. Thus, we plot in Figure 16 the number of beacon Blockmessages received from each peer. As we can see, there arefive peers that communicated over a thousand Beacon blocks,while the large majority of the peers did not communicatemore than ten. More precisely, 75% of Beacon Blocks werereceived from only ten peers, while 91.3% of the peerstransmitted less than ten Beacon Blocks. Keep in mind thatdue to the way the GossipSub protocol works, this does notmean that the large majority of peers do not communicate atall, simply that a few ones are systematically the first ones totransmit new unseen Beacon Blocks to the Armiarma crawler.Fig. 16: Number of Beacon Blocks Received from each Peer.In addition, we analyzed how many messages, peers sendwith respect to the time they keep connected. A node trans-mitting too many messages in a very short connection time7ould possibly signal a deny of service (DoS) attack. While anode with a very long connection but very few or no messagesat all, could signal an attempt of eclipse attack.Fig. 17: Total messages for Connected Time.The distribution is shown in Figure 17. Note that in thisfigure we count Beacon Blocks as well as Proof Aggregationsmessages. The distribution shows a dense cloud of peers thatstayed connected between 5 and 8 hours and transmittedbetween 5 and 1000 messages. Particularly interesting, is thecase of the dot in the top right corner of the figure. This peertransmitted almost 100,000 messages and stayed connectedaround 36 hours. We could call this a super-peer , because itshows an extremely stable connection and it also transmits ahuge amount of Beacon data to its peers in a timely manner.TABLE III: Example of Gathered Information from a Peer
Data Type InformationClient Type Lighthouse/v1.0.15a3b94cb/x86 64linuxCountry United StatesCity New Jersey, North BergenIsp DigitalOcean, LLCLatency(s) 31.509326Connections 10Disconnections 10Connected time(min) 24.600416Beacon Block messages 0Beacon A. And P. messages 0Voluntary Exits messages 0Proposer Slashing messages 0Attester Slashing messages 0
C. Individual Peer Analysis
The large analysis performed on the Eth2 network has beendone from peer related information and it shows a global viewof the network. But Armiarma also has the ability of analyzingpeers individually. Armiarma can extract a large amount ofinformation regarding specific peers. To showcase an examplewe show on the Table III the information extracted from one ofthe peers in the network. For security and privacy reasons, wehave omitted sensible information as peer Id, node Id, publickey and IP related to the peer. This peer showed an unusuallyhigh latency which caught our attention. Thus, Armiarma can help us monitor and keep track of nodes suspected of maliciousbehaviour. This demonstrate that Armiarma could be used insecurity analysis, in addition to the network status study.V. C
ONCLUSION
In this paper we have presented some of the informationthat can be gathered transparently with our network monitoringtool, Armiarma, directly showing a perception of the network,allowing us to get a better overview of the Eth2 networkstate. The presented analysis shows the status of the recentlylaunched Eth2 main net, that, as on the date of writing thispaper, is experiencing a healthy behaviour.Overall, we can see the large amount of information that theArmiarma analyser can produce while checking the internalbehaviour of different clients in the network. While in this casewe mostly focus on different clients, we could also search fordifferences between versions of the same client and thereforetrack long term improvements in the clients development.Both Rumor and Armiarma are tools with a huge potentialthat are still on a development stage. As future work, we wouldlike to keep contributing to the development of Rumor andkeep improving Armiarma as well. In particular, we plan toincrease the customization level of the host, increase the Eth2functionalities, develop a GossipSub protocol modificationcustomized for Armiarma analyzer and increase the range ofmetrics we are gathering.A
CKNOWLEDGMENT
This work has been supported by the Ethereum Founda-tion under Grant FY20-0198. We would like to thank theresearchers of the Ethereum Foundation for their feedback andsuggestions. In particular, we would like to thank @Proto-lambda as the main Rumor maintainer for his help setting upour experiments and his constructive feedback on this study.R