Metrics for Bengali Text Entry Research
Metrics for Bengali Text Entry Research
Sayan Sarcar, Ahmed Sabbir Arif, Ali Mazalek
Abstract
With the intention of bringing uniformity to Bengali text entry research, here we present a new approach for calculating the most popular English text entry evaluation metrics for Bengali. To demonstrate our approach, we conducted a user study where we evaluated four popular Bengali text entry techniques.
Author Keywords
Bangla/Bengali text entry; performance metrics.
Introduction
Bengali text entry on handheld devices is becoming increasingly popular among native speakers [8], as it allows them to communicate with each other without the assistance of a second language. There are numerous Bengali text entry techniques available for handheld devices. There are virtual keyboards that either map different keyboard layouts onto Q
WERTY (I N S CRIPT keyboards, i.e. Bangla Jatiyo, Probhat, and Bijoy [2]) or develop novel keyboard layouts (i.e. Swarachakra [6], iLiPi-B [9], and Bhattacharya [4]). There are phonetic keyboards that use phonetic mapping schemes to support Bengali text entry with Roman characters (i.e. Akkhor, Avro, Google Bengali Transliteration [5], Microsoft Bengali Transliteration, and Onkur). There are techniques that utilize the standard 12-key mobile keypad (i.e. Panini [7] and Nokia Bengali). There are also several techniques that attempt to assign dedicated keys for all Bengali characters. In the Bengali language there are approximately eleven vowels, thirty-six consonants, ten inflexions, four signs,
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author. Copyright is held by the owner/author(s). CHI'15, Apr 18-23, 2015, Seoul, Republic of Korea Workshop on Text entry on the edge
Sayan Sarcar
Center for Human Computer Interaction Kochi University of Technology Kami, Kochi, Japan [email protected]
Ahmed Sabbir Arif
Synaesthetic Media Laboratory Ryerson University Toronto, Ontario, Canada [email protected]
Ali Mazalek
Synaesthetic Media Laboratory Ryerson University Toronto, Ontario, Canada [email protected] and ten numeric characters. Besides, similar to other Indic languages such as Hindi, Bengali supports combinations between consonants (conjunct) and consonants and dependent vowels (glyph). Although, visually the word কা(cid:552) appears to be consisted of two characters কা and (cid:552) , in reality the former is a glyph ( ক + ◌া ) and the latter is a conjunct ( ন + ◌্ + ড ). Therefore, to enter this word with a character-based technique, one would need to input the following five characters in sequence ক + ◌া + ন + ◌্ + ড . Similarly, it would require five backspaces to delete it. As Bengali has more characters than English, I N S CRIPT techniques assign multiple characters to a single key, and then require the use of special keys (i.e. Shift) or key combinations (i.e. Shift + Ctrl) to disambiguate between them. Yet there is no commonly accepted standard for this. Similarly, Phonetic keyboards use different phonetic mapping schemes and character sequence to create conjuncts and glyphs. For example, based on the input sequence used by the scheme, প + ◌্ + র could yield either (cid:437) or পর্ . Besides, some keyboards provide dedicated keys for the most frequent conjuncts (i.e. ক্ষ , (cid:503) , etc.). Therefore, the process of Bengali text entry, editing, and error correction is fundamentally different with different techniques. Researchers usually use the most popular performance metrics for English text entry research, such as words per minute (WPM), keystrokes per character (KSPC), and minimum string distance error rate (MSDER), to evaluate Bengali text entry techniques. As these metrics were not designed for the Bengali language and are not equipped to address the diversity of Bengali text entry techniques, researchers and practitioners are forced to use different conventions to calculate them. This makes it difficult to evaluate and compare different Bengali text entry techniques, as these metrics usually yield different results for the same scenario based on how characters were defined and the sequence in which the conjuncts and the glyphs were generated. For instance, based on whether a conjunct was counted as a single or multiple character(s), these metrics will yield different results for the same input/output. To address this, here we propose a new method for calculating these metrics for Bengali. Proposed Methodology
Here we discuss the new approach for calculating the most common text entry performance metrics, namely WPM, KSPC, ER, and MSDER, for Bengali text entry. We use the following notations [1, 10]. Input Stream (IS) is the text containing all keystrokes produced by the user while entering the presented text. For example, if the user performs the keystrokes b+a+i with a transliteration system to input the Bengali word বই , then the input stream will also contain b+a+i. |IS| is the length of the input stream. Output Stream (OS) is the text that contains all “constituent” characters in a phrase. In other words, it considers the outputted characters instead of the exact sequence of keystrokes performed. Thus, for the above example, the output stream will contain ব + ই . Also, as it disjoins all conjuncts and glyphs into basic characters, even if the user inputs the conjunct ক্ষ ( ক + ◌্ + ষ ) directly, such as by pressing a dedicated key for it, the output stream will contain ক + ◌্ + ষ . Presented Text (P) is what participants had to enter, OS P the presented text’s output stream, and |OS P | is the length of OS P . Transcribed Text (T) is the final text entered by the participant, OS T is the transcribed text’s output stream, and |OS T | is the length of OS T . Incorrect Not Fixed (INF OS ) is the total number of uncorrected characters in the transcribed text’s OS. Minimum String Distance (MSD OS ) is the minimum number of operations needed to transform the transcribed text’s output stream to the presented text’s output stream, where the operations are insertion, deletion, or substitution of a basic character. If a technique allows the insertion, deletion, or substitution of a conjunct or a glyph, then the number of operations required is one over the number of characters in the conjunct or glyph. For instance, if it allows the user to delete a conjunct consisted of four basic characters using one backspace, then the number of operations required is ¼ = 0.25. Words per Minute for Bengali (WPM bn ) WPM BN measures the time it takes to produce a certain number of Bengali words. (cid:1849)(cid:1842)(cid:1839) (cid:3029)(cid:3041) (cid:3404) |(cid:3016)(cid:3020) (cid:3269) |(cid:2879)(cid:2869)(cid:3020) (cid:3400) 60 (cid:3400) (cid:2869)(cid:2873).(cid:2869)(cid:2869) (1) In Equation 1, S is time in seconds measured from the first keystroke to the last, including backspaces and other edit and modifier keys. The constant 60 is the number of seconds per minute, and 1/5.11 accounts for the average length of a word in characters including spaces, numbers, and other printable characters in Bengali [3]. As S is measured from the entry of the very first character to the last, the entry of the first character is never timed, which is the motivation for the –1 in the numerator. Keystrokes per Character for Bengali (KSPC bn ) KSPC bn is the ratio of the length of the input stream to the length of the transcribed text’s output stream. (cid:1837)(cid:1845)(cid:1842)(cid:1829) (cid:3029)(cid:3041) (cid:3404) |(cid:3010)(cid:3020)||(cid:3016)(cid:3020) (cid:3269) | (2) Error Rate for Bengali (ER bn , %) ER bn is the ratio of the total number of incorrect characters in the transcribed text’s output stream to the length of the transcribed text’s output stream. (cid:1831)(cid:1844) (cid:3029)(cid:3041) (cid:3404) (cid:3010)(cid:3015)(cid:3007) (cid:3264)(cid:3268) |(cid:3016)(cid:3020) (cid:3269) | (cid:3400) 100 (3) MSD Error Rate for Bengali (MSDER bn , %) MSDER bn calculates the smallest number of operations to transform the transcribed text’s output stream to match the presented text’s output stream, and then calculates the ratio of that number to the larger of the length of the presented and transcribed text’s output stream. (cid:1839)(cid:1845)(cid:1830)(cid:1831)(cid:1844) (cid:3029)(cid:3041) (cid:3404) (cid:3014)(cid:3020)(cid:3005) (cid:3264)(cid:3268) (cid:3014)(cid:3028)(cid:3051)(cid:4666)|(cid:3016)(cid:3020) (cid:3265) |,|(cid:3016)(cid:3020) (cid:3269) |(cid:4667) (cid:3400) 100 (4) Other Performance Metrics
We could also calculate other popular text entry performance metrics, such as Erroneous Keystrokes [1] and Total Error Rate [10], using this approach.
Current Methods
Researchers use various methods to calculate these metrics. Yet, as none of them deal with conjuncts and glyphs in a meaningful way and do not differentiate between the input and the output, they usually fail to show an accurate picture. One popular approach is to consider conjuncts and/or glyphs as individual characters instead of combinations. As a result, if it takes about the same time to enter two different phrases with two different techniques, one containing substantially more conjuncts than the other, this approach incorrectly assumes that both as equally fast—although, the conjunct-heavy phrase required substantially more character entry than the other. Similarly, this method assumes that the difference between বৃক্ষ and (cid:439)ক্ষ is one character, although with most techniques replacing বৃ ( ব + ◌ৃ ) with (cid:439) ( ব + ◌্ + র ) require correcting two unwanted characters ( ◌্ + র ). A User Study
We conducted a user study to evaluate four popular Bengali text entry techniques: Swarachakra, Panini,
Sample Calculation
Here we calculate the proposed metrics for an example scenario where the examined technique has a dedicated key for the conjunct ক্ষ ( ক + ◌্ + ষ ) Presented Text (P): ক্ষিণেকর অিতিথ
Output Stream (OS P ): ক + ◌্ + ষ + ণ + ি◌ + ক + ে◌ + র + + অ + ত + ি◌ + থ + ি◌ |OS P | = 14 Transcribed Text (T): ক্ষিণেকর অতিথ Output Stream (OS T ): ক + ◌্ + ষ + ণ + ি◌ + ক + ে◌ + র + + অ + ত + থ + ি◌ |OS T | = 13 Input Stream (IS): ক্ষ + Shift + ণ + ি◌ + ক + ে◌ + র + + অ + ত + থ + ি◌ |IS| = 12 INF OS = 1 MSD OS = 1 Let us assume, S = 20 seconds (cid:1849)(cid:1842)(cid:1839) (cid:3029)(cid:3041) (cid:3404) (cid:2869)(cid:2871)(cid:2879)(cid:2869)(cid:2870)(cid:2868) (cid:3400) 60 (cid:3400) (cid:2869)(cid:2873).(cid:2869)(cid:2869) (cid:3404) 7.05 (cid:1837)(cid:1845)(cid:1842)(cid:1829) (cid:3029)(cid:3041) (cid:3404) (cid:2869)(cid:2870)(cid:2869)(cid:2871) (cid:3404) 0.92 (cid:1831)(cid:1844) (cid:3029)(cid:3041) (cid:3404) (cid:2869)(cid:2869)(cid:2871) (cid:3400) 100 (cid:3404) 7.69% (cid:1839)(cid:1845)(cid:1830)(cid:1831)(cid:1844) (cid:3029)(cid:3041) (cid:3404) (cid:2869)(cid:2869)(cid:2872) (cid:3400) 100 (cid:3404) 7.14% InScript, and Bhattacharya (see Figure 1), with our methodology. The study involved inputting thirty short Bengali phrases (randomly collected from different books) with each technique on a touchscreen-based smartphone (Samsung Galaxy S3). Fifteen subjects voluntarily participated in the study. Their age ranged from 22 to 31 years, average 26 (SD = 4). Six of them were female and they all had prior experiences with touchscreen-based Bengali text entry. They were asked to enter the phrases as fast and accurate as possible. In summary, the design was: 15 participants × 4 techniques (counterbalanced) × 30 short Bengali phrases = 1,800 phrases in total.
Techniques WPM bn KSPC bn ER bn MSDER bn Swarachakra 10.07 6.23 10.05% 1.22% Panini 15.02 7.86 14.99% 1.35% Indic/I N S CRIPT
Table 1.
Text entry performance for four popular Bengali text entry techniques.
Results (see Table 1) show that Panini and Indic were substantially faster than the other techniques, but were also unreliable. However, text entry with Indic required about 35% fewer keystrokes than Panini. Swarachakra yielded a mediocre performance, both in terms of speed and accuracy, while Bhattacharya’s performance was the poorest. Our approach allowed us to measure and compare these techniques’ performance, although they are substantially different from one another.
Conclusion
With the intention of bringing uniformity in Bengali text entry research, we presented and demonstrated a new approach for calculating the most popular English text entry evaluation metrics for Bengali.
Future Work
As most Indic languages (i.e. Hindi, Punjabi, Marathi, Gujarati, Bhojpuri, Oriya, Sindhi, Sinhala, Nepali, Assamese, etc.) use a similar writing system, we speculate that this approach could be used with these languages as well. We would like to investigate this in the future.
References [1]
Arif, A. S. and Stuerzlinger, W. Analysis of text entry performance metrics.
TIC-STH '09 . IEEE, 100-105. [2]
Bengali Input Methods. http://en.wikipedia.org/wiki/Bengali_input_methods [3]
Bharati, A., Rao, P., Sangal, R. and Bendre, S. M. Basic statistical analysis of corpus and cross comparison among corpora. Technical Report ('00), IIIT. [4]
Bhattacharya S., and Laha S. Bengali text input interface design for mobile devices.
UAIS 12 , 4 ('13), 441-451. [5] [6]
Joshi, A., Dalvi, G., Joshi, M., Rashinkar, P., and Sarangdhar, A. Design and evaluation of Devanagari virtual keyboards for touch screen mobile phones.
MobileHCI '11 . ACM, 323–332. [7]
Panini Keypad. http://paninikeypad.com/index1.php [8]
Prince, M. E. H., Hossain, G., Dewan, A. A., and Debnath, P. An audible Bangla text-entry method in mobile phones with intelligent keypad.
ICCIT '09 . IEEE, 279-284. [9]
Samanta, D., Sarcar, S., and Ghosh, S. An approach to design virtual keyboards for text composition in Indian languages.
International Journal of Human Computer Interaction 29 , 8 (2013), 516-540. [10]
Soukoreff, R. W. and MacKenzie, I. S. Metrics for text entry research: an evaluation of MSD and KSPC, and a new unified error metric.
CHI '03 . ACM, 113-120.
Figure 1.