An experimental platform for gathering user behavioural data via browser APIs
AAn experimental platform for gathering userbehavioural data via browser APIs
Zhaoyi FanOctober 17, 2019
Abstract
Websites are capable of learning a wide range of information aboutthe platform on which a browser is executing. One major source of suchinformation is the set of standardised Application Programming Interfaces(APIs) provided within the browser, which can be accessed by JavaScriptdownloaded by a website; this information can then either be used bythe JavaScript or sent back to the originating site. As has been widelydiscussed, much of this information can threaten user privacy. The mainpurpose of this paper is to document a publicly available platform designedto enable further investigation of one class of such threats, namely thosebased on analysing user behavioural data. The platform has two maincomponents: a Chrome extension that gathers user keystroke and mousedata via browser APIs, and server software that collects and stores thisdata for subsequent experimentation.
The JavaScript language is widely used by websites to enable dynamicbehaviour. In the context of a web browser, JavaScript code is downloadedalong with the web page. This JavaScript is executed automatically bythe browser, and can access a variety of information via a set of browser-provided Application Program Interfaces (APIs).In general, APIs are constructs made available in programming lan-guagues to allow developers to create complex functionality more easily.Browser-supported APIs, which can be accessed by website-originatedJavaScript, provide website developers with more efficient ways to accom-plish their goals and provide browser users with a better experience.In a modern browser, Client-side JavaScript can access a range of APIs[2]. In general, browser APIs allow developers to create web pages thatincorporate information from the user’s environment [3]. In HTML5, awide variety of information can be accessed via browser APIs, such asaudio, application cache, canvas, fullscreen, geolocation, local storage,notifications, video and web database [1]. However, browser APIs canalso be used by websites to learn potentially privacy-sensitive informationabout the browser, the platform on which the browser is executing, and a r X i v : . [ c s . CR ] O c t ven the user of the browser. For example, platforms can be tracked acrossmultiple web visits using browser fingerprinting techniques [6], it may bepossible to learn secret user data such as PINs [7] and the user location[4], and it may even be possible to learn about the human user.The possibility of behavioural monitoring using browser APIs moti-vates the work described in this paper, namely the development of a plat-form to enable the collection of behavioural data. The platform has beendesigned to support experiments which aim to determine the degree towhich individual users can be identified via browser APIs. When a webuser is browsing a website, the standard Document Object Model (DOM)enables executing JavaScript to access information about keyboard andmouse events. These keyboard and mouse events can potentially be usedto identify individual users using techniques developed for biometric iden-tification and authentication [5]. The platform we describe in the re-mainder of this paper has been designed to support experiments aimedat understanding how effective such identification can be. However, theplatform is not restricted to this application, and is being made availableas a potential tool for further research on browser security and privacy.The platform involves a Chrome extension that has been implementedto collect keystroke and mouse data from browser users. In addition,server functionality has been developed to receive and store this data.The remainder of the paper is structured as follows. Section 2 intro-duces the browser APIs used by the extension. Section 3 then addressesthe implementation of the extension. Section 4 briefly describes the formof the data extracted by the extension. Section 5 introduces an approachto extracting usable data from the collected raw data. Section 6 describespossible ways in which the current platform might be extended, and Sec-tion 7 contains concluding remarks.The experimental platform is made freely available for use by otherresearchers. It can be found at [ https://github.com/fanzhaoyi/DataCollector ]. Please send any comments (including bug reports) to: [ [email protected] ] We first introduce the browser API calls used by the experimental plat-form to gather information about user behaviour. Specifically we describeAPIs which provide information about user interactions with keyboardsand pointing devices (e.g. a mouse).The most commonly discussed browser API is probably the DocumentObject Model API which can create, remove and change HTML and CSS.The Document interface provides a representation of a web page loadedin the browser, and serves as an entry point into the web page contentin the form of the DOM tree. The Event interface, one of the interfacesprovided by DOM, provides the keyboard event and mouse event APIs.The KeyboardEvent API provides information regarding user inter-actions with the keyboard. Each event is triggered by an action on a ey, and can have an event type of keydown , keypress or keyup . Table 1summarises the API calls for KeyboardEvent. Table 1: API calls for keyboard events
API call Properties
KeyboardEvent.code
Returns a string with the codevalue of the key represented bythe event.
KeyboardEvent.key
Returns the key value of the keyrepresented by the event. keydown
Triggered when the key is de-pressed. keyup
Triggered when the key is re-leased.
The MouseEvent API provides information on events that occur whena user interacts with a pointing device, such as a mouse. It can giveinformation regarding events such as moving a mouse, clicking a button,etc. Table 2 gives the API calls for MouseEvent.
Chrome extensions are programs that can be used to customize the brows-ing experience. They enable users to tailor Chrome functionality andbehaviour to individual needs or preferences. They are built on webtechnologies such as HTML, JavaScript, and CSS. By using the browser-supported APIs, an extension can collect user data and send it to a remoteserver.Extensions are made up of a range of possible components, which caninclude background scripts, content scripts, a manifest file and optionspages. We next briefly review these component types.The manifest file provides information about the extension. Typicallythe file manifest.json contains general information about the extension(name, version, etc.), its permissions (download, urls, etc.) and settingsfor option pages.Background scripts can work with or without a background HTMLfile. A background page is loaded when it is needed and unloaded when itbecomes idle. It can communicate with content scripts and popup pagesby sending messages and listening for an event.By using the DOM, content scripts can read details of web pages visitedby the browser, make changes to them and pass information to other scriptpages. Working in an isolated environment, a content script is able tochange the JavaScript environment of a loaded page without conflictingwith the page or other scripts. It can be regarded as a part of the loadedpage; however, a content script is not permitted to access any variables orfunctions created by the web page. The
Cross-Origin XMLHttpRequest policy prevents a content script from directly sending a
XMLHttpRequest
API call Properties click
Triggered when the mouse button is pressedand released on a single element. mousemove
Triggered when the pointer is moving while itis over an element. mousedown
Triggered when the mouse button is pressedon an element. mouseup
Triggered when the mouse button is releasedon an element. wheel
Triggered when the mouse wheel is crollingover an element. pageX
Returns the X coordinate of the mouse pointerrelative to the whole document. pageY
Returns the Y coordinate of the mouse pointerrelative to the whole document. screenX
Returns the X coordinate of the mouse pointerin global(screen) coordinates. screenY
Returns the Y coordinate of the mouse pointerin global(screen) coordinates. clientX
Returns the X coordinate of the mouse pointerin the applications client area. It changeswhen the page is scrolled. clientY
Returns the Y coordinate of the mouse pointerin the applications client area. It changeswhen the page is scrolled. to a remote server. Instead, message passing can be used to pass messagesfrom a content script to the background script.
A Chrome extension was developed to collect user keystroke and mousemovement data. In addition, a server script was written to receive andstore the data. We next describe their development and operation. Thestructure of the experimental platform is shown in Figure 1.
All software development was performed on a Windows 10 system, withdetails as follows. • Programming tool : Sublime Text (Version 3.1.1, Build 3176). • Programming languages:
HTML and JavaScript (client-side),ASP (server-side), Python (data processing). • Google Chrome version:
The extension is made of a manifest.json file, a popup HTML page, apopup.js script file, a background.js script file and a content.js script file.The popup page is shown when a user clicks the extension. The popuppage allows the user to register and then log in. After the user has loggedin, the content script starts monitoring visited webpages. Keyboard eventAPIs and mouse event APIs are used to monitor the user’s keyboard andmouse activities. Since content scripts cannot directly communicate witha remote server, each time the keyboard or mouse event is triggered, thesendMessage method is called to pass the message from the content scriptto the background script. When the background script receives the data,an XMLHttpRequest is used to interact with the server.As shown in Figure 1, at the client the content script extracts keystrokeand mouse data using browser APIs and sends the data to the backgroundscript via message passing. The popup page communicates with the back-ground script and informs the user about the login status. The permis-sions are set in manifest.json . The extension can access any URL and usecookies.Content scripts can read the DOM of a web page. The content scriptuses six event listeners: mousemove, mousedown, mouseup, wheel, key-down and keyup . Every time such an event is triggered, a timestamp iscreated and the corresponding data is captured. The data shown belowis then sent to the background script. chrome.runtime.sendMessage({type: actiontype,data: actiondata},function(response){}) ction type has two possible values: keystroke and mouse. For asingle keystroke, the data is regarded as an object with eight elements:keystroke timestamps, key value, and the usage of functional keys (Ctrlkey, Alt key, Shift key and CapsLock key), as shown in Figure 3. Whena key is depressed, an object of type keystroke with eight elements iscreated. When the key is released, the uptime is updated and the data issent to the background script.For mouse actions the data consists of one of eight mouse action types(mouse move, left button down, left button up, right button down, rightbutton up, wheel rolls, wheel down and wheel up), the coordinates of themouse (X and Y coordinates) and the timestamp, as shown in Figure 4.The background script receives the request message from content scriptand popup script: chrome.runtime.onMessage.addListener(function(request, sender, sendResponse){if (request.type == action type ){SendToServer({data: request.data,type: request.type,})}}) The main function of the background script is communicating with theother scripts and the server. When the background script receives a mes-sage from another script, an
XMLHttpRequest is established to send thedata to remote server: xmlhttp.open("GET", URL+data, TRUE)xmlhttp.send()
The popup page allows the user to register and login. When the user logsin, a message is passed to the server via the background page.As the extension can record every keystroke and mouse movement,some personal data might be gathered, including passwords for other web-sites and other sensitive data. Thus if a user has any concerns that thedata they are typing is possibly sensitive, the user is recommended to logout of the extension by clicking the logout button; the user can log inagain when the sensitive interactions have been completed.
At the server, ASP files are used to interact with the client and database.The data is stored in a SQL Server 2008 database. To prevent SQLinjection attacks, a regular expression check is used when data is received.To connect with the SQL server database,
ADODB.CONNECTION isused in the ASP file:
Set Conn=server.CreateObject("ADODB.CONNECTION")StrConn=" Provider=SQLOLEDB;Data Source= domain;User ID=userID;Password= pwd;Initial Catalog= databaseName"Conn.open StrConn
When a user registers on the popup page, their registration informationis sent to the server. To distinguish data sent from different devices, theserver creates a cookie for each user: response.cookies("uname") = username
When a user logs out, the server deletes the cookie. The server main-tains two databases for each user, one to store keystroke data and theother to store mouse use data.
When the user is logged in and the browser web page is active, the exten-sion monitors every keyboard and mouse event performed by the user.
Each keystroke is regarded as a single object containing the followingelements: key code, key value, key down time, key up time, Ctrl, Alt,Shift, CapsLock. Figure 3 gives an example of the data generated if a userpresses the key D.
Keydown and keyup are timestamps, code representsthe physical key position on the keyboard, while key is the key value thatthe user has typed. alt , ctrl , shift and caps indicate use of these fourfunction keys. It should be noted that, to type an uppercase letter, a usercan either use the Caps Lock key or hold down the Shift key. A keyboard event only has one type of action: press. However, a mouseevent can be triggered by a number of different types of action, such asmouse movement, left button click, right button click and wheel scrolling.Thus each mouse event is regarded as a single object containing the fol-lowing elements: type of event action, X coordinate of the mouse, Ycoordinate of the mouse, and a timestamp, as shown in Figure 4.
Depending on the use to be made of the collected data, it is first necessaryto process it to extract the features to be used. As examples, we nextdescribe possible approaches to such processing, used in experiments thatwere performed with the aid of the platform.
Figure 5 gives an example of raw keystroke data as stored in the serverdatabase. This data was processed in the following way to enable it tobe used for possible user identification based on how letter bigrams aretyped. The raw data is first divided into a sequence of sets using thevalues of the keydown timestamp.The sequence of keystrokes is divided into discrete sets using two rules: • if the time interval between two consecutive keystrokes is greaterthan one second, or • if the key depressed is the space key, then a new set is started. Keystrokes corresponding to the space keyand to function keys are removed from the data set. Using these rules,the data shown in Figure 5 will yield the four sets (words) This Is TheText.The sequence of letters in each set was then used to derive detailedtiming information for each pair of consecutive letters (or bigraph). Foreach such bigraph four timestamps were extracted, namely the key downand key up times for the two letters. The biograph timing informationwas then used to try to identify the user. Of course, this is only oneapproach to extracting usable data from the raw data, and is providedhere for illustrative purposes only. For the purposes of user identification two types of data were extractedfrom pairs of consecutive mouse events: Euclidean distance between thelocations and elapsed time between the events. This was then used tocompute the mouse movement speeds between events. The movementspeed between a range of pairs of types of mouse event were then used toattempt to identify users. Possible future work
The experimental platform is currently capable of gathering keyboardevent and mouse event data. However, there is other behaviour-related(and hence privacy-sensitive) data that could be gathered by the platform.For example, motion sensor data for a mobile device is accessible viamotion sensor APIs. Extensions to the platform are envisaged whichwould gather such data.
This document describes the functioning and use of an experimental plat-form designed to gather user-related data from a browser. The platformhas been developed to support experiments aimed at understanding thedegree to which individual users can be identified using browser-gatheredbehavioural data. It is hoped that the platform will be of value to otherresearchers performing similar experiments.Section 2 introduces the browser APIs used by the experimental plat-form. Section 3 provides details of the implementation. Section 4 de-scribes the form of the data extracted by the extension. Section 5 illus-trates possible approaches to extracting usable data from the collecteddata. Section 6 describes possible further development of the platform.
References [1] Chrome JavaScript Web APIs. https://developer.chrome.com/apps/api_other . Accessed: 28-February-2018.[2] Client-side Web APIs. https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Client-side_web_APIs/Introduction . Ac-cessed: 28-February-2018.[3] W3C JavaScript Web APIs. . Accessed: 28-February-2018.[4] Nasser Mohammed Al-Fannah. One leak will sink a ship: WebrtcIP address leaks. In
International Carnahan Conference on SecurityTechnology, ICCST 2017, Madrid, Spain, October 23-26, 2017 , pages1–5. IEEE, 2017.[5] Kyle O. Bailey, James S. Okolica, and Gilbert L. Peterson. User iden-tification and authentication using multi-modal behavioral biometrics.
Computers & Security , 43:77–89, 2014.[6] Peter Eckersley. How unique is your web browser? In Mikhail J. Atal-lah and Nicholas J. Hopper, editors,
Privacy Enhancing Technologies,10th International Symposium, PETS 2010, Berlin, Germany, July21-23, 2010. Proceedings , volume 6205 of
Lecture Notes in ComputerScience , pages 1–18. Springer, 2010.[7] Laurent Simon and Ross J. Anderson. PIN skimmer: inferringpins through the camera and microphone. In William Enck, Adri-enne Porter Felt, and N. Asokan, editors,
SPSM’13, Proceedings of he 2013 ACM Workshop on Security and Privacy in Smartphonesand Mobile Devices, Co-located with CCS 2013, November 8, 2013,Berlin, Germany , pages 67–78. ACM, 2013., pages 67–78. ACM, 2013.