Migration of CMSWEB Cluster at CERN to Kubernetes
Muhammad Imran, Valentin Kuznetsov, Lina Marcella, Katarzyna Maria Dziedziniewicz-Wojcik, Andreas Pfeiffer, Panos Paparrigopoulos
MMigration of CMSWEB Cluster at CERN to Kubernetes
Muhammad Imran, š,š, ā Valentin Kuznetsov, š Lina Marcella, š Katarzyna MariaDziedziniewicz-Wojcik, š Andreas Pfeiļ¬er š and Panos Paparrigopoulos š š CERN,Meyrin, Geneva, Switzerland š National Centre for Physics,Islamabad, Pakistan š Cornell University,New York, USA
E-mail: [email protected], [email protected],[email protected], [email protected],[email protected], [email protected]
The CMS experiment heavily relies on the CMSWEB cluster to host critical services for itsoperational needs. The cluster is deployed on virtual machines (VMs) from the CERN OpenStackcloud and is manually maintained by operators and developers. The release cycle is composedof several steps, from building RPMs, their deployment to perform validation, and integrationtests. To enhance the sustainability of the CMSWEB cluster, CMS decided to migrate its clusterto a containerized solution such as Docker, orchestrated with Kubernetes (k8s). This allows us tosigniļ¬cantly reduce the release upgrade cycle, follow the end-to-end deployment procedure, andreduce operational cost. This paper gives an overview of the current CMSWEB cluster and itsissues. We describe the new architecture of the CMSWEB cluster in Kubernetes. We also providea comparison of VM and Kubernetes deployment approaches and report on lessons learned duringthe migration process. ā Speaker Ā© Copyright owned by the author(s) under the terms of the Creative CommonsAttribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0). https://pos.sissa.it/ a r X i v : . [ c s . D C ] F e b igration of CMSWEB Cluster at CERN to Kubernetes Muhammad Imran
1. Introduction
The Compact Muon Solenoid (CMS) is a general-purpose detector at the Large Hadron Collider(LHC) at CERN, Geneva, Switzerland [1]. The CMS experiment runs hundreds of thousands ofjobs daily on its distributed computing system to simulate, reconstruct and analyse the data takenduring collision runs. A dedicated cluster ("CMSWEB") is used to host essential CMS centralservices which are responsible for the CMS data management, data discovery, and various databookkeeping tasks. The cluster is based on virtual machines (VMs) on the CERN OpenStack cloudinfrastructure. Each service is managed by its own development team. Due to the complexity of theheterogeneous environment, diļ¬erent schedules of development teams, only monthly release cyclescan be aļ¬orded. Each upgrade cycle includes: the build of RPMs from source code, including alldependency chains, the cross-validation of all software components, and the validation of the correctinteractions of all services. This typically requires a lot of communication between developmentteams and operators, as well as coordination of various eļ¬orts in a coherent manner.To enhance the sustainability of CMSWEB, CMS decided to migrate to a containerized solutionbased on Docker and orchestrated with Kubernetes (āk8sā), a de facto industry standard for managingcontainers [6]. Recently, an instance of the testbed CMSWEB cluster was successfully migratedto Kubernetes. With the containerized approach, developers will not have to ask the operatorsto deploy their services, they can deploy new versions of their services in a few seconds. Thissigniļ¬cantly reduces the release upgrade cycle, follows the end-to-end deployment procedures, andreduces operational cost.In this paper, an overview of the current CMSWEB cluster and the issues faced in this clusteris given. The new architecture of the CMSWEB cluster orchestrated by Kubernetes is described.Furthermore, various issues found during the implementation in Kubernetes are discussed. Wealso provide a comparison of VM and Kubernetes deployment approaches, and report on lessonslearned during the migration process.The remainder of this paper is organized as follows. Section 2 describes the current architectureof CMSWEB cluster. Section 3 presents the proposed architecture of CMSWEB in Kubernetes.Section 4 presents the performance analysis. We discuss our plans for the future in Section 5. Weconclude in Section 6.
2. Architecture of CMSWEB
The CMSWEB cluster consists of 21 services maintained by CMS operators. The CMSWEBcluster allows: independent development and evolution of underlying services; simpliļ¬ed integra-tion and regression testing when rolling out new service versions; and building external servicesthat integrate information from several sources in a clean manner.The architecture of the CMSWEB VM cluster is shown in Figure 1. It has two layers ofservices i.e., frontend and backend services. The frontend service is based on Apache whichperforms authentication using certiļ¬cates and redirects requests to the requested backend service,while the backend services perform their relevant tasks. The frontend service has redirect rules toforward the requests to the relevant VM node running the backend service.2 igration of CMSWEB Cluster at CERN to Kubernetes
Muhammad Imran
Apache FrontendsDBS DAS CRAB CouchDB ReqMgr2 Phedex Wmstat DQMBackend ServicesUsersCERN Network
Figure 1:
The CMSWEB VM Cluster Architecture
Apache FrontendsDBS DAS CRAB CouchDB ReqMgr2 Phedex Wmstat DQM
Cluster B
UsersCERN Network nginxCluster Anginx
Figure 2:
The CMSWEB Kubernetes cluster architecture
3. Deployment of CMSWEB cluster to Kubernetes
The proposed architecture of CMSWEB in Kubernetes is shown in Figure 2. It has the twocomponents: frontend cluster (cluster A); and backend cluster (cluster B). The frontend clustercontains the CMSWEB Apache frontend behind the Nginx Kubernetes ingress controller (server).The backend cluster contains all CMSWEB back-end services behind its ingress controller. Thefrontend cluster ingress controller provides transport layer security (TLS) passthrough capabilitiesto pass clientās requests (with certiļ¬cates) to the Apache frontend. The Apache frontend performsCMSWEB authentication and redirects the request to the backend cluster. On the backend cluster,the ingress controller has basic redirect rules to the appropriate services and only allows requests3 igration of CMSWEB Cluster at CERN to Kubernetes
Muhammad Imran
Services R e qu es t s / sec ond Performance Benchmark
Figure 3:
The performance benchmark results in the form of requests/second (y-axis) of some of thecommonly used CMSWEB services (x-axis) for various clusters (i.e. VM production cluster, VM testbedcluster, old Kubernetes cluster and new Kubernetes cluster using a diļ¬erent number of replicas.). The 100tests were performed for each conļ¬guration and results show average values from these tests. from the frontend cluster.To host services in Kubernetes, we need container images of all services, which are thendeployed in Kubernetes. The Docker images are created using RPMs of the current VM cluster,and are kept in a central repository which is available at [2]. Similarly to the docker area in therepository, there is also a kubernetes area for deploying those images in Kubernetes. A central cmsweb directory in the repository contains sub-directories for all CMSWEB namespaces [3].
4. Performance Analysis
We used the hey tool [4] to evaluate the performance of the VM based clusters (both testbed andproduction) and the new testbed clusters in Kubernetes. The initial Kubernetes cluster which usedversion 1.15, showed a severe network degradation caused by a faulty network driver (āļ¬annelā).This issue was ļ¬xed in Kubernetes version 1.17, which uses the calico network driver. In thefollowing comparison, we label the clusters using Kubernetes version 1.15 as āold clusterā, and theones using version 1.17 as ānew clusterā.To test the performance, a conļ¬guration with š =
10 and š = š is thenumber of requests to run and š is the number of workers to run concurrently.The performance results are shown in Figure 3: some services of CMSWEB are shown in thex-axis while the resulting requests/second are shown on the y-axis. A comparative analysis of theVM production cluster, VM testbed cluster, old Kubernetes cluster, and new Kubernetes cluster wasperformed. The old and new Kubernetes clusters were studied with diļ¬erent numbers of replicasof frontend i.e. (4,6,8) to see the impact of replicas on the overall performance. It can be seen thatthe new Kubernetes cluster performs much better as compared to the production, testbed and oldKubernetes cluster for all services benchmarked.4 igration of CMSWEB Cluster at CERN to Kubernetes Muhammad Imran
During this migration process various issues were encountered, which are categorized as eitherinfrastructure or service issues
A major issue of network degradation was identiļ¬ed by us andother groups, which was related to the infrastructure at CERN, Further detail of the performancebenchmark is available in Section 4.
Cluster Creation Issues:
With the new Kubernetes version 1.17, cluster creation failed dueto timeout. This issue was related to storage volumes on servers which were in diļ¬erent availabilityzones and had higher latencies, causing timeouts.
Ceph Mount Issues:
A problem with Ceph mounts showed up after migrating clusters tothe new Kubernetes version. This issue was caused by a version of the cloud client which CERNhas provided for the management and creation of clusters in the CERN network, which was notcompatible with the new version of Kubernetes.
Permission Mount Issues:
Permission issues of the mount point in /etc/grid-security in thenew Kubernetes version were found to be related to the security context of an unconļ¬gured pod. Asecurity context deļ¬nes privilege and access control settings for a Pod or Container
Nginx-ingress controller Issues:
A problem with nginx-ingress controller was encounteredwhen we performed stress tests on the K8s cluster. It was related to the low value of ļ¬le descriptorsin the nginx-ingress controller. During stress tests many requests were failed because of that.These issues were ļ¬xed with the help of CERN IT.
We noticed that CouchDB crashed in the Kubernetes cluster. Handling things likedatabases, availability to other layers of the application, and redundancy for a database can have veryspeciļ¬c requirements. That makes it challenging to run a database in a distributed environment. Itwas therefore decided to keep this service in the VM cluster and all CouchDB requests are redirectedto the VM cluster.
PhEDEx:
The PhEDEx service is one of the legacy application we are required to supportduring the transition to Kubernetes. So, it was decided to not spend time on porting it to newinfrastructure and keep it in the dedicated VMs.
DBS:
Initially, in the Kubernetes cluster, the same accounts as the ones of the VM basedclusters were used, this caused issues. In order to avoid potential overwrite of data in productionDBS DB instances, we were asked by DBS team to use separate accounts for DBS in Kubernetescluster.
The new Kubernetes infrastructure automates the procedure of service deployment as thedevelopers are able to deploy their services directly in the Kubernetes cluster without needing inputfrom the CMSWEB operator. 5 igration of CMSWEB Cluster at CERN to Kubernetes
Muhammad ImranThe current VM cluster lacks autoscaling feature; every service is deployed on the particular VMnodes and the resources are assigned on the VM level instead of the service level. When the servicesare overloaded, they often become unresponsive, then the CMSWEB operator manually interferesand restarts individual services. The auto-scaling feature of Kubernetes, however, automaticallymanages the resources based on the workload.Manifest ļ¬les in Kubernetes require veriļ¬cation. A small typo and wrong indentation leads tofailures.
5. Future work
We plan to work on the following items:
Custom auto-scalers , the default settings of K8s only provides auto-scaling based on CPUand RAM usage of the pods. Since we run several applications within a pod and monitor theirusage via the Prometheus service, we can take advantage of using service metrics for auto-scalersand perform dynamic tuning of services based on those metrics.
Service-mesh deployment , the service-mesh provides plenty of beneļ¬ts to Kubernetes, includ-ing traļ¬c encryption within the cluster, traļ¬c routing between diļ¬erent releases, canary deploymentand rolling release cycles. We would like to bring this functionality to our infrastructure via theIstio [5].
6. Conclusions
In this paper, we give an overview of the current CMSWEB cluster and the issues which wefaced in the cluster. We describe the new architecture of the CMSWEB cluster in Kubernetes andexplain the implementation strategy of the proposed architecture in Kubernetes. Furthermore, wedescribe various issues that we faced during the implementation in Kubernetes. We also provide acomparison of VM and Kubernetes deployment approaches, emphasizing the pros and cons of thenew architecture and report on lessons learned during the migration process. The new cluster ofCMSWEB in Kubernetes enhances sustainability and reduces the operational cost of CMSWEB.
References [1] CMS experiment at CERN. Website: https://home.cern/science/experiments/cms .last accessed: 04.13.2020.[2] Cms repository for docker. Website: https://github.com/dmwm/CMSKubernetes/tree/master/docker . last accessed: 04.04.2020.[3] Cmsweb kubernetes repository. Website: https://github.com/dmwm/CMSKubernetes/tree/master/kubernetes . last accessed: 04.04.2020.[4] Hey tool. Website: https://github.com/rakyll/hey . last accessed: 04.04.2020.[5] Istio middleware. Website: https://istio.io/ . last accessed: 04.16.2020.[6] Kubernetes. Website: https://kubernetes.io/docs/home/https://kubernetes.io/docs/home/