A Gamification of Japanese Dependency Parsing
AA Gamification of Japanese Dependency Parsing
Masayuki AsaharaAbstract
Gamification approaches have been usedas a way for creating language resourcesfor NLP. It is also used for presentingand teaching the algorithms in NLP andlinguistic phenomena. This paper ar-gues about a design of gamification forJapanese syntactic dependendency parsingfor the latter objective. The user interfacedesign is based on a transition-based shiftreduce dependency parsing which needsonly two actions of SHIFT (not attach)and REDUCE (attach) in Japanese depen-dency structure. We assign the two ac-tions for two-way directional control ona gamepad or other devices. We also de-sign the target sentences from psycholin-guistics researches.
We start this research with a naive question: howto make the syntactic dependency annotation moresimple and joyful for non-specialist by associatingthe game devices. Though gold standard anno-tation might be developped by the specialist, wewould like to collect the distribution of the judgesby non-specialists in order to know which sen-tences are hard to parse for human.We propose a novel game application to test hu-man dependency parsing process. The game isnamed “shWiiFit Reduce Dependency Parsing” ofwhich movieclip can be viewed by clicking Fig-ure 1 or browsing the URL http://goo.gl/cWncIi . The idea is that a player stands onthe balance board, reads a sentence to parse andselects one of two possible actions (“SHIFT” or“REDUCE”) by moving one’s weight to the leftor right depending on whether a pair of focusedphrases are in syntactic dependency relation ornot. In this article, we focus on Japanese syntacticdependency parsing. Reasons are: 1) Japanesesyntactic dependency relations are simpler toparse due to the strictly head final characteris-tics. 2) State-of-the-art parsers for Japanese aremostly transition-based (Kudo and Matsumoto,2002; Iwatate et al., 2008). 3) There is a linear-order, transition-based, and above all, an O ( N ) algorithm (Sassano, 2004) which is a simpli-fied variation of (Nivre and Scholz, 2004)’s shiftreduce-like algorithm.Section 2 shows the basic design of game userinterface. Section 3 presents design of sample sen-tences based on psycholinguistic researches. Sec-tion 4 reports experiments and the analysis of theresults. Section 5 states conclusions and our futurework. http://goo.gl/cWncIi Figure 1: shWiiFit Reduce Dependency Parsing
We describe a fitness game application called“shWiiFit Reduce Dependency Parsing”. Thescreen is shown in Figure 1. The game used Sas-sano’s shift reduce-like Algorithm (Algorithm 1)which parses a length- N sentence at most N ac-tions. “SHIFT” and “REDUCE” are the actions a r X i v : . [ c s . C L ] J a n lgorithm 1 Shift reduce-like Japanese depen-dency parsing% Initialization (cid:104)
S, Q, A (cid:105) = (cid:104) nil, W, φ (cid:105) repeatif S == nil then % “default shift” (cid:104) nil, q | Q, A (cid:105) ⇒ (cid:104) q, Q, A (cid:105) else if | Q | == 1 then % “default reduce” (cid:104) s | S, q, A (cid:105) ⇒ (cid:104)
S, q, A ∪ ( s, q ) (cid:105) else % Judge on (cid:104) s | S, q | Q, A (cid:105) if s and q has a dependency relation then % “REDUCE” (cid:104) s | S, q | Q, A (cid:105) ⇒ (cid:104)
S, q | Q, A ∪ ( s, q ) (cid:105) else % “SHIFT” (cid:104) s | S, q | Q, A (cid:105) ⇒ (cid:104) q | s | S, Q, A (cid:105) end ifend ifuntil S == nil and | Q | == 1 return A which require human (or machine) judgements ondependency relations.We introduce a tuple of stack S , queue Q , andthe set of dependency relations A , initializing nil,an input phrase sequence, and empty respectively.A parsing game proceeds by state transitions ofthe tuple as the following actions. When S is nil(and there are multiple phrases in Q ), the gameautomatically executes the action “default shift”.The first element q of Q is moved onto S . When S is not nil and Q has only single phrase q left,the game automatically executes action “defaultreduce”. As the top element s of S is by defaulta dependent of q , the pair (cid:104) s, q (cid:105) is added to A and s is removed from S . When the top element s of S and the first element q of Q are both defined, theplayer judges whether q is the head of s and in-puts an answer. If the answer is yes, then the gameexecutes the action “REDUCE” and the pair (cid:104) s, q (cid:105) is added to A and s is removed from S . Other-wise, the game executes the action “SHIFT” andthe first element q is moved on the top of S . Thegame ends when S is nil and there is only the lastphrase of an input sentence in Q .A player stands on a balance board and watchesa screen as shown in Figure 1 during a game.A sentence to parse is displayed in the top of the screen. Below it, dependency relations that aplayer builts during a game is shown accordingly.At the start of a game, the face icon in the bot-tom is positioned center. No phrase is in stack S shown in the left hand side of the face icon, whileall 6 phrases are in queue Q shown in the righthand side of the face icon. During the game, aplayer judges whether the first (leftmost) phrase q of queue Q is the head of the top (rightmost)phrase s of stack S . If so, the player weighs tothe right so as for the face icon to move towardsthe REDUCE wall. Otherwise, the player weighsto the left so as for the face icon to move towardsthe SHIFT wall. The game shows 820-860 mil-liseconds animations of screen transitions just af-ter executing each action, to move the icon auto-matically centered back. At the end of the game,the screen displays “OK” is the player wins (i.e.parses correct), “NG” otherwise. The game doesnot indicate which dependency relation was wrongto the player.By jumping on the board, the player can havea go at the next sentence. The software keepstracks of the response time for which each actionhas taken. Although we use a balance board asan input device, the software is implemented sothat joysticks, game pad (including any censors ofNintendo Wii remote), and cursor keys on the key-board can be replaced.Note, whereas the direction of the arrow in theglobal standard is from the head to the dependent,the one of in Japanese standard is from the de-pendent to head. Japanese base phrase (bunsetsu)is strictly head-final dependency, which is alwayspresented by the left arrow in the global standardand by the right arrow in the Japanese standard.There is no ambiguity to present not arrows butplain lines to indicate the dependency edges. The setting of parsing difficulty is one of issuesto make the game more joyful. We focus on gar-den path sentences containing embedded struc-tures presented in Tokimoto’s paper (Tokimoto,2004) for the game. Figure 2 shows 3 types ofsentences used in our experiments.Each sentence takes 6 phrases of the form,
N P
NOM N P
ACC V P AST N P
DAT X V P AST Where the 5th phrase X is variant. De-spite case markers of the 1st N P
NOM , the 2nd ttp://goo.gl/EShaJP Figure 2: Example Sentences
N P
ACC and the 4th N P
DAT are identical, nom-inatives and accusatives of the 3rd and the 6th V P AST vary depending on the 5th phrase.We classify garden path sentences by the 5thphrase into 3 types. A sentence is called Con-trol (CTRL) if the 5th phrase is
N P
NOM witha nominal case marker “ga”. Similarly, a sen-tence is called Early Boundary (EB) if the 5thphrase is N P
ACC with an accusative case marker“wo”. Otherwise, a sentence is called Late Bound-ary (LB).In Figure 2, we notice how different depen-dency relations become among the garden pathsentences. These sentences have different depen-dency structures due to semantic dependency con-straints of the two verbs (3rd and 6th phrases).In the CTRL sentences, the 1st phrase N P
NOM is a subject (nominative) of the 3rd phrase V P AST and the 5th phrase N P
NOM is a subject of the 6thphrase V P AST . Due to the projective and head-final constraints, the 2nd N P
ACC has to be a di-rect object (accusative) of the 3rd phrase V P AST .In the EB sentences, the 4th phrase N P
DAT isa notional subject of the 3rd phrase V P AST and the 1st phrase N P
NOM is a subject of the 6thphrase V P AST , since a verb has only one subjectphrase. Due to the projective and head-final con-straints, the 2nd phrase N P
ACC has to be an di-rect object of the 3rd phrase V P AST , and the 5thphrase N P
ACC has to be an direct object of the6th phrase V P AST .In the LB sentences, the 4th phrase N P
DAT isa notional object of the 3rd phrase V P AST . Dueto single accusative constraint, the 2nd N P
ACC has to be a direct object of the 6th phrase V P AST .Due to the projective and head-final constraints,the 1st phrase N P
NOM has to be a subject of the6th phrase V P AST .(Tokimoto, 2004) performed psycholinguisticexperiments on the garden path sentences as inFigure 2 using self paced reading method andquestion answering method. In his experiments,the phrases of the sentence are incrementally pre-sented. The reanalysis cost of the sentences forhuman judges has been reported as CTRL < EB < LB. He also found that the reanalysis cost var-ied among individuals.able 1: Sentence accuracy and sentence responsetime (Human judges)Filler CTRL EB LBacc. (%) ave. 72 63 82 45stdev. 19.7 38.9 21.7 32.1s.r.r.t. ave. -0.12 0.05 0.12 0.81stdev. 0.12 0.71 0.51 0.40
We propose that our game application can be aninteresting alternative to some what boring cor-pus annotation. We use our shWiiFit Reduce De-pendency Parsing to evaluate parsing difficulty ofgarden path sentences containing embedded struc-tures for 12 graduate students aged 22-27.Each experimental subject attempts a parsinggame of 40 sentences in one main session. 10sentences are the 3 types of garden path sentencesoriginally used in Tokimoto’s experiment and theremaining 30 sentences are filler sentences. Thevocabulary of the sentences is carefully controlledby the word frequency of 10-year newspaper cor-pus and word familiarity rating of NTT database.Sentences are presented in the following order: 1)5 fillers, 2) 15 fillers + + + http://goo.gl/nIe6k3 . http://goo.gl/nIe6k3 Figure 3: The experimental environmentTable 1 summarizes the sentence accuracy(“acc”) and sentence residual response time(“s.r.r.t.”) of the human judgments. “acc. ave.”is the average of sentence accuracy for Fillers,CTRLs, EBs, and LBs. “s.r.r.t. ave.” is an inter-nally studentized residual of the time for the cor-rectly parsed sessions where linear regression ex-cluded (a)
N P
NOM phrase tends to bea nominative of the last 6th V P AST phrase. Thebias helps experimental subjects to parse correctlyin EB, and hinders in CTRL.In terms of sentence residual response time, LBis longer time to be parsed correctly than the oth-ers. There is no statistically significant differencebetween CTRL and EB.Note that Table 1 shows just a general tendency,one experimental subject parsed correctly in allEBs and LBs but make mistakes in CTRLs. Itmeans that the reanalysis cost and bias of humanvaried among individuals.Our game application can replace the balanceboard with any kinds of input devices. An effectof using the balance board is that the experimentalsubjects need to keep their weight centered. Theffect reduces their working memory for parsing.In preliminary experiments with 4 subjects, thedifference of accuracies between CTRL and LBwith the balance board is larger than ones withsimple game pads. We describe a game application to evaluate hu-man dependency parsing process. Using NintendoWii Balance Board as an input device, we appliedthe game application to evaluate difficulty to parsesentences for humans. As far as we know, thiswork is the first comprehensive attempt to evalu-ate how humans accurately parse sentences in thelinear order shift reduce manners by game appli-cation.
Acknowledgements
We really appreciate Prof. Djam´e Seddah to en-courage us to publish the English version of thearticle. The Japanese version is https://doi.org/10.5715/jnlp.18.351
References [Iwatate et al.2008] Masakazu Iwatate, Masayuki Asa-hara, and Yuji Matsumoto. 2008. Japanese Depen-dency Parsing Using a Tournament Model. In
Proc.of the 22nd International Conference on Computa-tional Linguistics(COLING-2008) , pages 361–368.[Kudo and Matsumoto2002] Taku Kudo and Yuji Mat-sumoto. 2002. Japanese Dependency Analyisis us-ing Cascaded Chunking. In
Proc. of the 6th Confer-ence on Natural Language Learning(CoNLL-2002) ,pages 1–7.[Nivre and Scholz2004] Joakim Nivre and MarioScholz. 2004. Deterministic DependencyParsing of English Text. In
Proc. of the 20thInternational Conference on ComputationalLinguistics(COLING-2004) , pages 64–70.[Sassano2004] Manabu Sassano. 2004. Linear-TimeDependency Analysis for Japanese. In