Attacks to Federated Learning: Responsive Web User Interface to Recover Training Data from User Gradients
TThis paper appears in ACM ASIA Conference on Computer and Communications Security (ACM
ASIACCS ) 2020.Please feel free to contact us for questions or remarks.
Attacks to Federated Learning: Responsive Web User Interfaceto Recover Training Data from User Gradients
Hans Albert Lianto ∗ Nanyang Technological [email protected]
Yang Zhao ∗† Nanyang Technological [email protected]
Jun Zhao
Nanyang Technological [email protected]
ABSTRACT
Local differential privacy (LDP) is an emerging privacy standard toprotect individual user data. One scenario where LDP can be appliedis federated learning, where each user sends his/her user gradientsto an aggregator who uses these gradients to perform stochasticgradient descent. In a case where the aggregator is untrusted andLDP is not applied to each user gradient, the aggregator can recoversensitive user data from these gradients. In this paper, we present aninteractive web demo showcasing the power of LDP by visualizingfederated learning with LDP. Moreover, the live demo shows howLDP can prevent untrusted aggregators from recovering sensitivetraining data. A measure called the exp-hamming recovery is alsocreated to show the extent of how much data the aggregator canrecover.
CCS CONCEPTS • Security and privacy → Distributed systems security ; •
Com-puting methodologies → Supervised learning by classification ; Supervised learning by regression . KEYWORDS
Local differential privacy, Machine learning, Federated learning,Linear regression, Stochastic gradient decent.
ACM Reference Format:
Hans Albert Lianto, Yang Zhao, and Jun Zhao. 2020. Attacks to FederatedLearning: Responsive Web User Interface to Recover Training Data fromUser Gradients. In
Proceedings of the 15th ACM Asia Conference on Computerand Communications Security (ASIA CCS ’20), October 5–9, 2020, Taipei,Taiwan.
ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/3320269.3405438
Local differential privacy (LDP) is used to perturb data locally. Evenwhen perturbed data are exposed to adversaries (which could in-clude the data curator or aggregator), LDP guarantees that selected ∗ Both authors contributed equally to the paper. The order of names is alphabetical. † Corresponding authorPermission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).
ASIA CCS ’20, October 5–9, 2020, Taipei, Taiwan © 2020 Copyright held by the owner/author(s).ACM ISBN 978-1-4503-6750-9/20/10.https://doi.org/10.1145/3320269.3405438 user’s sensitive information is not leaked. Hence even the data cu-rator or aggregator is not trusted with true information from eachentry in the dataset. Local differential privacy was implemented byGoogle in its RAPPOR technology in their paper in [4].Local differential privacy ( ϵ -LDP) [4] is defined as follows: Definition 1.1.
A randomized algorithm f satisfies ϵ -local dif-ferential privacy if and only if, for any two tuples t and t ′ in thedomain of f , and for all subsets S of the output range: Pr [ f ( t ) ∈ S ] ≤ e ϵ × Pr [ f ( t ′ ) ∈ S ] , where parameter ϵ is the privacy budget and Pr [·] is probability.Many companies store enterprise data in relational form withRDBMS software. For example, Airbnb and Uber store their datausing mySQL [1]; Netflix and Instagram store their data usingPostgreSQL [2]. Data stored in this relational form come in a set oftuples or records with the same schema. Each tuple or record havethe same number of ‘dimensions’ or rows. LDP-perturbed data canhence also be stored in these relational forms.Methods to perturb this multidimensional data to satisfy ϵ -LDPhave been proposed, including Duchi et al. ’s recent proposal in[3] and Wang’s improved proposal (the Piecewise Mechanism andHybrid Mechanism) in [5]. These methods will be used in the webdemo outlined in subsequent sections of the paper. One practical application for local differential privacy is in ma-chine learning algorithms, particularly in federated or centralizeddistributed learning. As illustrated in Fig. 1, each user is consid-ered a node that trains gradients independently; each node thensends these gradients to the parameter server for aggregation [6].The data that these users make use of to obtain user gradients issensitive information and should not be leaked.
Figure 1: Federated learning flowchart
It can be shown in the subsequent web demo that without localdifferential privacy, a significant portion of the training data couldbe leaked by the untrusted aggregator in any federated learningsetting where each user submits their gradients for training.1 a r X i v : . [ c s . CR ] J u l his paper appears in ACM ASIA Conference on Computer and Communications Security (ACM ASIACCS ) 2020.Please feel free to contact us for questions or remarks.
A web user interface (called ldp-machine-learning) is created tosimulate stochastic gradient descent in federated learning, and forthe aggregator to recover training data from each user. A screenshotof the demo is shown in Fig. 2 and the demo is publicly available inhttps://ldp-machine-learning.herokuapp.com/
Figure 2: Screenshot of ldp-machine-learning demo
In the demo, when all users fill in the training data and the Trainbutton is started, one epoch of stochastic gradient descent takesplace. Unbeknownst to the ‘users’ illustrated in the GUI, the aggre-gator uses the gradients from the user and the current parametersof the model to backtrack and recover the training user data, whichposes a security risk. Features of the UI are outlined in the nextsection.
The main animation in the demo occurs when the Train and Re-cover buttons are clicked. When the Train button is clicked, trainingproceeds and the current cost or training accuracy of the machinelearning model is shown beside each user sending his or her gradi-ents to the aggregator. When training is finished, the final modelcost and accuracy are outputted on screen. When the Recover but-ton is clicked, the untrusted aggregator attempts to recover sensitiveuser training data from the gradients sent by each user. At the end ofrecovery, the average exp-hamming recovery for a user is outputtedon screen. Normally, the training and recovery animation occursat a rate of one user per second; however, changing the TrainingAnimation Speed input from “1 record / second” to “Instant” in theUI input form will remove the animation and immediately displaytraining and recovery results when training finishes.
The demo features federated learning with various machine learn-ing algorithms such as linear regression, logistic regression andsupport vector machine, which can be toggled, as shown in Fig. 3.Moreover, as shown in Fig. 3, the LDP perturbation mechanism toperturb user gradients can be toggled from 4 options: Laplace mech-anism, Duchi et al. ’s mechanism [3], Piecewise mechanism [5] andHybrid mechanism [5]; the privacy budget ϵ of the LDP algorithmcan also be toggled. Figure 3: Toggling between different ML algorithms, LDPalgorithms and privacy budgets
An ‘Add 10 Users’ button and an ‘Add 100 Users’ button were addedto automatically generate new training data. A scroll up buttonfrom the bottom page is added for easier navigation. The buttonsare shown in Fig. 4.
Figure 4: The add users and scroll up buttons
Each user’s training data is generated via the following equation: d = − . x − . x + . x + . x + . , for linear regression. For logistic regression and SVM it is: d = (cid:40) , if − . x − . x + . x + . x + . > , − , otherwise . From these two equations, the ideal weights for the model are [− . , − . , . , . , . ] . The closer the model weights are tothe ideal weights, the better the model is. Initial model weightsand training data are generated with a pseudorandom seed via anextension in the JavaScript Math library. Before the formal definition of the exp-hamming recovery is dis-cussed, it is imperative to identify what it means when it is “moredifficult” for an aggregator to recover user training data. Considera user’s training data to be the vector x = ( x , x , ..., x n ) with n dimensions. Now consider an aggregator’s recovered training datato be the vector x r = ( x r , x r , · , x nr ). The natural convention isthat it is more difficult to recover x if x r is farther from x . Hencea distance metric is needed; naturally, the higher the value of thismetric is, the more difficult it is for an aggregator to recover theuser’s training data. Manhattan distance ( ℓ exp-hamming recovery .The exp-hamming recovery is designed so that the more (respec-tively less) difficult it is for the aggregator for recover training data,the lower (respectively higher) the exp-hamming recovery shouldbe. The exp-hamming recovery is hence formally defined as follows: E = exp (− k (∥ x − x r ∥ ) , where k is a customizable constant (its value is important for accu-rate interpretation), and ∥· ∥ represents the Manhattan distance.This metric naturally makes sense because, if x = x r , there is fullrecovery of user training data, meaning that E =
1. If ∥ x − x r ∥ = ∞ ,meaning that x and x r are infinitely apart, there is no informationgained by the aggregator of what the user training data is like,2his paper appears in ACM ASIA Conference on Computer and Communications Security (ACM ASIACCS ) 2020.Please feel free to contact us for questions or remarks.meaning that E =
0. One can choose a value of k such that E < / e ≈ .
368 should the Manhattan distance ∥ x − x r ∥ > / k . Thiscritical value 0.368 for exp-hamming recovery would hence be agood heuristic to whether the aggregator has enough data to beable to make a good guess at the user’s real training data or not. Inthe demo, the value of k used is 0.5. For privacy budget management in each user, the privacy budgetis allocated equally to each gradient value. Since there are 5 totalvalues to perturb in the demo, if the privacy budget allocated to eachuser is ϵ each of these values are perturbed with privacy budget ϵ / α = . Fig. 5 shows the architecture diagram for the web UI, which showsthat the site’s UI and main logic operate entirely on the front-end.This is to eliminate the latency from API calls to a backend server,which would slow down the site performance. In addition to that,the UML State Diagram for the UI is shown in Fig. 6.
Figure 5: Software architecture diagram forldp-machine-learning
The development library used for the frontend is React, an open-source JavaScript library. Throughout development, a consciouseffort is made to separate the part of the application correspondingto its appearance and to its logic. The appearance is maintainedwithin the project’s ‘components/’ directory, while the site logic lies mainly in the ‘utils/’ directory. The version control system forthe project is git and project code is stored on GitHub. Moreover,the demo is deployed and served on Heroku, a cloud platform-as-a-service.
Figure 6: UML State Machine Diagram fromldp-machine-learning
In this paper, a web GUI is made to illustrate the above results andthe power of locally differentially private mechanisms to perturbdata. The demo is publicly accessible so potential researchers in thefield of differential privacy are able to understand local differentialprivacy in the context of applying it in machine learning algorithms.The GUI is easily extensible to other machine learning algorithmsand LDP perturbation mechanisms in future work. Moreover, train-ing hyperparameters such as batch size and learning rates can beadded to the demo for better training results that nearly matchexperimental setups with real datasets.
ACKNOWLEDGMENTS
This research was supported by 1) Nanyang Technological Univer-sity (NTU) Startup Grant, 2) Alibaba-NTU Singapore Joint ResearchInstitute (JRI), 3) Singapore Ministry of Education Academic Re-search Fund Tier 1 RG128/18, Tier 1 RG115/19, Tier 1 RT07/19,Tier 1 RT01/19, and Tier 2 MOE2019-T2-1-176, 4) NTU-WASP JointProject, 5) Singapore National Research Foundation (NRF) under itsStrategic Capability Research Centres Funding Initiative: StrategicCentre for Research in Privacy-Preserving Technologies & Systems(SCRIPTS), 6) Energy Research Institute @NTU (ERIAN), 7) Sin-gapore NRF National Satellite of Excellence, Design Science andTechnology for Secure Critical Infrastructure NSoE DeST-SCI2019-0012, 8) AI Singapore (AISG) 100 Experiments (100E) programme,and 9) NTU Project for Large Vertical Take-Off & Landing (VTOL)Research Platform.
REFERENCES [1] Accessed in 2020. Why developers like MySQL. https://stackshare.io/mysql.[2] Accessed in 2020. Why developers like PostgreSQL. https://stackshare.io/postgresql.[3] John C Duchi, Michael I Jordan, and Martin J Wainwright. 2018. Minimax optimalprocedures for locally private estimation.
J. Amer. Statist. Assoc.
Proceedings of the 2014 ACMSIGSAC Conference on Computer and Communications Security . 1054–1067.[5] Ning Wang, Xiaokui Xiao, Yin Yang, Jun Zhao, Siu Cheung Hui, Hyejin Shin,Junbum Shin, and Ge Yu. 2019. Collecting and analyzing multidimensional datawith local differential privacy. In . IEEE, 638–649.[6] Ligeng Zhu, Zhijian Liu, and Song Han. 2019. Deep leakage from gradients. In
Advances in Neural Information Processing Systems (NeurIPS) . 14747–14756.. 14747–14756.