Dynamic Thresholding Mechanisms for IR-Based Filtering in Efficient Source Code Plagiarism Detection
DDynamic Thresholding Mechanisms for IR-BasedFiltering in Efficient Source Code PlagiarismDetection
Oscar Karnalim
Faculty of Information TechnologyMaranatha Christian University
Bandung, [email protected]
Lisan Sulistiani
Faculty of Information TechnologyMaranatha Christian University
Bandung, [email protected]
Abstract —To solve time inefficiency issue, only potential pairsare compared in string-matching-based source code plagiarismdetection; wherein potentiality is defined through a fast-yet-order-insensitive similarity measurement (adapted from Infor-mation Retrieval) and only pairs which similarity degrees arehigher or equal to a particular threshold is selected. Definingsuch threshold is not a trivial task considering the thresholdshould lead to high efficiency improvement and low effective-ness reduction (if it is unavoidable). This paper proposes twothresholding mechanisms—namely range-based and pair-count-based mechanism—that dynamically tune the threshold based onthe distribution of resulted similarity degrees. According to ourevaluation, both mechanisms are more practical to be used thanmanual threshold assignment since they are more proportionalto efficiency improvement and effectiveness reduction.
Keywords —source code plagiarism detection; time efficiency;information retrieval; software engineering; computer scienceeducation
I. INTRODUCTIONPlagiarizing another student’s work and claiming it as theirsfor completing course assignment is an illegal behavior forstudents [1]. In programming courses, detecting the exis-tence of such behavior is not trivial considering programmingassignments are usually given weekly [2] and source code(i.e., submitted assignment form) is easy to be replicated& modified [3]. As a result, several automated plagiarismdetection approaches have been developed [4].One of the frequently-used approaches for detecting sourcecode plagiarism is based on string-matching algorithm [3](where given source codes are converted to token sequencesand treated as strings with each token refers to one character).Despite its effectiveness, such approach takes a considerableamount of time when applied on academic environment; givensource codes should be compared to each other in combi-natoric manner and each comparison typically takes eitherquadratic or cubic time complexity.A work in [5] addresses aforementioned time inefficiencyissue by utilizing Information Retrieval (IR) as an initial filter.Instead of comparing each possible combination pair witha string-matching algorithm, it only compares several pairs which IR-based similarity degree passes a particular threshold.In such manner, the processing time will be significantlyreduced considering not all pairs are compared with string-matching algorithm and IR-based similarity algorithm usuallyworks in linear time complexity.To date, the threshold used for IR-based filtering is definedwithout considering two facts: most IR-based similarity de-grees are clustered at certain points and the distribution ofthose degrees varies per source code plagiarism dataset. Asa result, the tendency to incorrectly assign the threshold willbe considerably high. Such incorrect assignment could eitherremove numerous potential plagiarism pairs (when assignedthreshold is overly-high) or lead to limited time efficiencyimprovement (when assigned threshold is overly-low).To mitigate the tendency of incorrect threshold assignment,this paper proposes two dynamic thresholding mechanismsthat define the threshold based on the distribution of re-sulted IR-based similarity degrees. Range-based thresholdingmechanism converts raw threshold (i.e., a manually-definedthreshold) to a proportion toward the distribution of resultedIR-based similarity degrees. Whereas, pair-count-based thresh-olding mechanism converts the raw threshold to a proportiontoward the number of compared pairs; wherein such proportionwill be used to exclude pairs with the lowest similaritydegrees. II. RELATED WORKSSource code plagiarism detection can be classified to threecategories: attribute-based, structure-based, and hybrid ap-proach [3], [6]. Attribute-Based Approach (ABA) relies onshared source code characteristics (e.g., token occurrences [7])to suspect plagiarism. Numerous similarity measurements havebeen used in this approach where most of them are adaptedfrom other domains, such as Fuzzy Logic [7] and InformationRetrieval [8]–[11]. Structure-Based Approach (SBA) relieson shared source code structure to suspect plagiarism. Mostof its similarity measurements are based on string-matchingalgorithms [3], [12]–[15] or graph-matching algorithms [16],[17]. Hybrid approach combines ABA and SBA to suspect a r X i v : . [ c s . S E ] O c t lagiarism. Such combination aims to get either better effec-tiveness [18]–[20] or efficiency [5].Effectiveness-oriented hybrid approach benefits from ABA’sand SBA’s similarity result characteristics. For example, awork in [18] shows ABA’s and SBA’s similarity results at oncefor suspecting plagiarism. They argue that slight modificationis better handled by ABA while the complex one is handledby SBA. Other two examples are works proposed in [19]and [20]. These works treat SBA’s similarity degree (resultedfrom string-matching algorithm) as an attribute for ABA—thatutilizes learning algorithm [19] or clustering algorithm [20].In contrast, efficiency-oriented hybrid approach benefitsfrom ABA’s fast processing time to perform efficient com-parison using SBA; SBA is commonly slow due to its hightime complexity. A work in [5] mitigates the number ofSBA-compared source code pairs by performing ABA-basedfiltering beforehand. A source code pair is only measuredusing SBA (that utilizes string-matching algorithm in theircase) iff its ABA-compared similarity degree (resulted fromInformation Retrieval measurement) passes a particular thresh-old. According to their evaluation, such combination couldenhance the efficiency of source code plagiarism detectionwith no extreme reduction on effectiveness.Considering aforementioned filtering (which will be referredas IR-based filtering at the rest of this paper) is conducted priorto string-matching algorithm [5], defined threshold for such fil-tering is crucial to determine the effectiveness and efficiency. Itcould lead to ineffectiveness when defined threshold is higherthan most pairs’ IR-based similarity degrees; no pairs willbe passed to string-matching algorithm (which is responsibleto determine plagiarism cases). On another extreme point,it could also lead to time inefficiency since most pairs willbe compared using string-matching algorithm (which takes aconsiderable amount of processing time) in addition to IR-based similarity measurement.III. METHODOLOGYThis paper proposes two mechanisms that mitigate the ten-dency of incorrectly assigning threshold for IR-based filtering.Those mechanisms—which are referred as range-based andpair-count-based mechanism—dynamically tune raw thresh-old based on the distribution of resulted IR-based similaritydegrees. They are expected to be more practical to be usedthan manual threshold assignment; they are more proportionalto efficiency improvement and effectiveness reduction. It isimportant to note that effectiveness reduction is unavoidablein most cases since IR-based measurements are typically lessstricter than the string-matching one.Range-based thresholding mechanism considers raw thresh-old (i.e., the manually-defined one) as a proportion towardthe distribution of resulted IR-based similarity degrees. It iscalculated using (1) where in refers to raw threshold and sim max & sim min refer to maximum & minimum IR-basedsimilarity degrees on given dataset respectively. For instance, if50% is passed as the raw threshold toward a plagiarism datasetwhich maximum & minimum IR-based similarity degree are 80% & 30% respectively, range-based thresholding mechanismwill exclude all pairs which IR-based similarity degree is lowerthan 55%—that is resulted from 30% + (50% * (80% - 30%)). RM ( in ) = sim min + ( in ∗ ( sim max − sim min )) (1)In contrast, pair-count-based thresholding mechanism con-siders raw threshold as a proportion toward the number ofcompared pairs. Such proportion will then be used to excludepairs with the lowest IR-based similarity degrees. The numberof excluded pairs will be calculated as in (2) where total pairs refers to the number of compared pairs. If raw threshold is50% and the number of compared pairs is 30, pair-count-basedthresholding mechanism will exclude 15 pairs (50% of 30)with the lowest similarity degrees. P CM ( in ) = in ∗ total pairs (2)For our case study, both thresholding mechanisms will beapplied on an efficiency-oriented hybrid source code plagia-rism detection, which works in fourfold (see Fig. 1). At first,given source codes (where each code represents a student’swork) are converted to token sequences using ANTLR [21]with comment tokens excluded. Second, comparison pairs areformed by pairing each token sequence with other sequences incombinatoric manner. For example, if there are three sequencesnamed A ; B ; and C , their comparison pairs will be (A,B) , (A,C) , and (B,C) . Third, for each comparison pair, IR-basedsimilarity degree is measured using vector space model andcosine similarity [22]. If such degree passes defined threshold(generated by either range-based or pair-count-based mecha-nism), that pair will be passed to the th phase. Otherwise,the pair will be excluded from consideration. Fourth, string-matching-based similarity degree of passed comparison pairswill be measured using either Running-Karp-Rabin Greedy-String-Tiling (RKRGST) [23] or Local Alignment (LA) [24];where both algorithms have been modified to handle sourcecode tokens instead of characters. The results of this phasewill be used as a guideline to suspect source code plagiarism.IV. EVALUATIONThree evaluation metrics will be considered in this study:the number of excluded pairs, reduced asymptotic number ofprocesses (i.e., predicted reduced processing time), and thedissonance degree of excluded pairs (i.e., tendency to excludepotentially-suspected comparison pairs). The former two arerelated to time efficiency while the last one is related toeffectiveness.Each evaluation metric will compare three thresholdingmechanisms: Range-based thresholding Mechanism (RM),Pair-Count-based thresholding Mechanism (PCM), and Staticthresholding Mechanism (SM). RM and PCM are our proposedmechanisms. SM, on the other hand, is a manual thresholdingmechanism that has been used in [5]. It directly uses rawthreshold as similarity degree threshold in IR-based filtering.For comparison purpose, SM will utilize the same efficiency-oriented hybrid source code plagiarism detection as in RMand PCM. All thresholding mechanisms will be converted ig. 1. Efficiency-oriented hybrid source code plagiarism detection used inour study. It contains four consecutive phases. to 11 evaluation scenarios each, varying in terms of initialthreshold (starting from 0% to 100% with 10% increasebetween scenarios).Evaluation dataset is taken from [25] by clustering allplagiarized codes per plagiarist, resulting 9 sub-datasets. Suchdataset was initially formed by asking 9 lecturer assistantsto plagiarize 7 Java source codes (covering Introductory Pro-gramming topics) using 6 plagiarism attack categories (definedin [26] with comment & whitespace modification as the lowestlevel and logic change as the highest one). The statistics ofall sub-datasets can be seen in Table I. For each sub-dataset,source code pairs are generated by comparing the codes toeach other. TABLE IT HE S TATISTICS OF E VALUATION S UB -D ATASETS
ID Min TokenSize Max TokenSize Avg TokenSize Number ofPairs
P1 251 1268 638.35 861P2 273 1499 683.92 741P3 269 1130 614.78 820P4 256 1321 671.21 496P5 271 1240 651.62 780P6 369 1245 689.31 861P7 303 1183 646.30 780P8 288 1195 660.91 528P9 323 1234 644.20 780
A. The Number of Excluded Pairs
The Number of Excluded Pairs (NEP) is defined by countinghow many comparison pairs are out of consideration since theirIR-based similarity degrees are lower than defined threshold.It is related to time efficiency since processing time is affected by the number of comparison pairs passed to string-matching-based similarity measurement.Fig. 2 shows that SM’s scenarios are not proportional toNEP. It only excludes comparison pairs on limited thresh-olds (which are 70% to 100% in our case). Such findingis natural considering IR-based similarity degrees are notequally distributed from 0% to 100%. In most occasions, thosedegrees are clustered at the end of similarity range (more than50%). Further, even in similar scenario, SM excludes variousnumber of comparison pairs among sub-datasets. As depictedon Fig. 3, it only generates considerably similar result on 100%threshold where most pairs are excluded.
Fig. 2. The number of excluded pairs that is averaged among 9 sub-datasetswherein such number for each sub-dataset is normalized toward total pairs.Fig. 3. The number of excluded pairs per sub-dataset from 70% to 100%threshold; each number is normalized toward total pairs.
PCM’s and RM’s scenarios, on the other hand, are moreproportional to NEP. Since both PCM and NEP utilize similarmetric unit (i.e., the number of pairs), they are completelyproportional to each other. As seen in Fig. 2, PCM formsa straight-diagonal graph line as NEP is improved gradually.RM, in contrast, does not form a straight-diagonal graph lineas PCM; they exclude more pairs at middle thresholds (whichare 20% to 70%) than the extreme ones since minimum andmaximum similarity degrees (used for defining range-basedthreshold) are usually outliers, that are located far higher orlower than most pairs’ similarity degrees. . Reduced Asymptotic Number of Processes
Reduced Asymptotic Number of Processes (RANP) iscalculated by subtracting Asymptotic Number of Processes(ANP) prior and upon the implementation of IR-based fil-tering. It is related to time efficiency considering ANP isadapted from asymptotic time complexity, a measurement unitfor predicting processing time.ANP for each scenario (per thresholding mechanism) isgenerated based on (3) where P is a list of comparison pairsand t is a defined threshold for IR-based filtering. It sums upANP for all comparison pairs where each of them consistsANP for IR-based filtering—see (4)—and string-matching-based similarity measurement—see (5). The former one is thetotal token size of a comparison pair (where p a and p b are thetoken sequences) with an assumption that IR-based filteringtakes at most linear complexity. Whereas, the latter one takescubic time complexity of the largest size between p a and p b . Itis based on RKRGST’s time complexity since such algorithmis popular to be used in source code plagiarism detection. Itis important to note that T S is only calculated if resulted IR-based similarity degree is higher or equal to defined threshold. AN P ( P, t ) = n (cid:88) i =1 ( T IR ( P i ) + T S ( P i , t )) (3) T IR ( p ) = | p a | + | p b | (4) T S ( p, t ) = ( max ( | p a | , | p b | )) if sim IR ( p ) > = t otherwise (5)IR-based filtering is applied prior string-matching-basedcomparison measurement. Hence, it will lead to more pro-cesses than conventional approach (i.e., a detection with string-matching measurement only) when few (or no) comparisonpairs are excluded. As depicted in Fig. 4, some scenarios yieldRANP below than 0% since they exclude few comparisonpairs. Fig. 4. Reduced asymptotic number of processes. Value for each thresholdis averaged among 9 sub-datasets.
RANP result (see Fig. 4) shares similar pattern as NEPresult (see Fig. 2): PCM is the most proportional one towardgiven metric, followed by RM and SM. Such resemblanceoccurs due to the fact that both metrics (i.e., RANP and NEP) are related to each other: more excluded pairs leads to fewerinvolved processes. However, their results are not exactlysimilar since token size for each involved token sequence (usedfor calculating ANP) varies.
C. The Dissonance Degree of Excluded Pairs
Considering SBA is probably more effective than ABAin most occasions [27], [28], it is important not to excludecomparison pairs with high SBA-measured similarity de-grees (i.e., high string-matching-measured similarity degrees)through ABA-based filtering (i.e., IR-based filtering). The Dis-sonance degree of Excluded Pairs (DEP) measures how manycomparison pairs with high string-matching-based similaritydegrees are excluded as a result of IR-based filtering. HigherDEP refers to lower effectiveness since it means numerouspairs with high string-matching-based similarity degrees areexcluded.DEP is calculated in threefold. First of all, comparisonpairs are ranked in descending order toward two perspectives:IR-measured and string-matching-measured similarity degrees.Second, per perspective, comparison pairs are assigned witha rank each where the st rank is assigned to a pair withthe highest similarity degree. Third, DEP is defined basedon (6) where E refers to n excluded pairs and Rank IR & Rank
SIM are the ranks of given pair from IR-measured &string-matching-measured similarity degree perspective. Usingsuch equation, the highest and lowest DEPs occurs whenhalf comparison pairs are excluded. The highest one occurswhen all of them are top-half string-matching-measured rankswhile the lowest one occurs when all of them are the bottom-half ranks. For comparison purpose, resulted DEP will benormalized based on its possible lowest score.
DEP ( E ) = n (cid:88) i =1 Rank IR ( e i ) − n (cid:88) i =1 Rank
SIM ( e i ) (6)When RKRGST is used in string-matching-based similaritymeasurement, SM only yields a non-zero DEP when giventhreshold is higher or equal to 70% (see Fig. 5). In contrast,RM and PCM yield DEP in more gradual manner. Their DEPis increased progressively as given threshold gets closer to50%.When compared to each other, RM is more effective thanPCM; it generates lower DEP on most scenarios (see Fig. 5where RM’s graph line is drawn below PCM’s). In average,RM generates 3.58% less DEP.All thresholding mechanisms are considerably effective forexcluding irrelevant pairs when RKRGST is used in string-matching-based similarity measurement. Their highest DEP(at 50% threshold) is extremely low when compared to DEPfor worst case scenario; it only takes about 17% of worstcase scenario’s. Further observation shows that most excludedpairs have considerably similar rank when perceived from IR-measured and string-matching-measured similarity degree per-spectives; similarity degrees resulted from IR-based similaritymeasurement (i.e., cosine similarity) strongly correlates with ig. 5. The dissonance degree of excluded pairs for RKRGST. Value foreach threshold is averaged among 9 sub-datasets. RKRGST’s. According to Pearson correlation, both of themshare 0.79 of 1 correlation degree.Fig. 6 shows that DEP resulted from using LA as string-matching-based similarity measurement shares similar pat-tern as using RKRGST. Such finding is natural since, whenmeasured using Pearson correlation, LA’s similarity degreesstrongly correlate with RKRGST’s (which is 0.78 of 1 corre-lation degree). Despite high resemblance, LA favors RM thanPCM more. They generate 5.55% DEP difference in LA whileonly 3.58% difference resulted in RKRGST. Another finding isthat, in general, LA is less effective than RKRGST when usedas a part of efficiency-oriented hybrid source code plagiarismdetection. It generates 1.62%, 4.45%, and 6.42% more DEPon SM, RM, and PCM respectively.
Fig. 6. The dissonance degree of excluded pairs for LA. Value for eachthreshold is averaged among 9 sub-datasets.
V. CONCLUSION AND FUTURE WORKIn this paper, two dynamic thresholding mechanisms for IR-based filtering (i.e., range-based and pair-count-based thresh-olding mechanism) are proposed. According to our evaluation,such mechanisms are more practical to be used since they aremore proportional than static thresholding mechanism in termsof excluding comparison pairs, reducing number of processes,and mitigating dissonance degree (i.e., a tendency to removepairs with high string-matching-measured similarity degrees). When compared to each other, pair-count-based thresholdingmechanism is more proportional to the number of excludedpairs and reduced asymptotic number of processes. However,range-based thresholding mechanism is more effective sinceit generates less dissonance degree. The implementation ofrange-based thresholding mechanism has been implementedin our other work (which details can be seen in [29]).For future works, we plan to evaluate whether the ef-fectiveness and efficiency of proposed thresholding mecha-nisms are consistent among various algorithms for IR-basedfiltering. Further, we also plan to check whether frequently-used features on source code plagiarism detection (such asmethod linearization [30], [31]) enhance the effectiveness andefficiency of proposed thresholding mechanisms.R
EFERENCES[1] S. Simon, J. Sheard, M. Morgan, A. Petersen, A. Settle, and J. Sinclair,“Informing students about academic integrity in programming,” in
Proceedings of the 20th Australasian Computing Education Conferenceon - ACE ’18 . New York, New York, USA: ACM Press, 2018, pp.113–122.[2] C. Kustanto and I. Liem, “Automatic Source Code Plagiarism Detec-tion,” in . IEEE, 2009, pp. 481–486.[3] O. Karnalim, “A Low-Level Structure-based Approach for DetectingSource Code Plagiarism,”
IAENG International Journal of ComputerScience , vol. 44, no. 4, 2017.[4] T. Lancaster and F. Culwin, “A Comparison of Source Code PlagiarismDetection Engines,”
Computer Science Education , vol. 14, no. 2, pp.101–112, jun 2004.[5] S. Burrows, S. M. M. Tahaghoghi, and J. Zobel, “Efficient plagiarismdetection for large code repositories,”
Software: Practice and Experi-ence , vol. 37, no. 2, pp. 151–175, feb 2007.[6] Z. A. Al-Khanjari, J. A. Fiaidhi, R. A. Al-Hinai, and N. S. Kutti,“PlagDetect: a Java programming plagiarism detection tool,”
ACMInroads , vol. 1, no. 4, p. 66, dec 2010.[7] G. Acampora and G. Cosma, “A Fuzzy-based approach to programminglanguage independent source-code plagiarism detection,” in . IEEE, aug2015, pp. 1–8.[8] U. Inoue and S. Wada, “Detecting plagiarisms in elementary program-ming courses,” in . IEEE, may 2012, pp. 2308–2312.[9] T. Ohmann and I. Rahal, “Efficient clustering-based source code plagia-rism detection using PIY,”
Knowledge and Information Systems , vol. 43,no. 2, pp. 445–472, may 2015.[10] D. Ganguly, G. J. F. Jones, A. Ram´ırez-de-la Cruz, G. Ram´ırez-de-laRosa, and E. Villatoro-Tello, “Retrieving and classifying instances ofsource code plagiarism,”
Information Retrieval Journal , pp. 1–23, sep2017.[11] G. Cosma and M. Joy, “An Approach to Source-Code PlagiarismDetection and Investigation Using Latent Semantic Analysis,”
IEEETransactions on Computers , vol. 61, no. 3, pp. 379–394, mar 2012.[12] L. Prechelt, G. Malpohl, and M. Philippsen, “Finding Plagiarisms amonga Set of Programs with JPlag,”
Journal of Universal Computer Science ,vol. 8, no. 11, pp. 1016–1038, 2002.[13] O. Karnalim, “Detecting source code plagiarism on introductory pro-gramming course assignments using a bytecode approach,” in
The 10thInternational Conference on Information & Communication Technologyand Systems (ICTS) . Surabaya: IEEE, 2016, pp. 63–68.[14] Z. Duric and D. Gasevic, “A Source Code Similarity System forPlagiarism Detection,”
The Computer Journal , vol. 56, no. 1, pp. 70–86,jan 2013.[15] F. S. Rabbani and O. Karnalim, “Detecting Source Code Plagiarismon .NET Programming Languages using Low-level Representation andAdaptive Local Alignment,”
Journal of Information and OrganizationalSciences , vol. 41, no. 1, pp. 105–123, jun 2017.16] D. Fu, Y. Xu, H. Yu, and B. Yang, “WASTK: A Weighted AbstractSyntax Tree Kernel Method for Source Code Plagiarism Detection,”
Scientific Programming , vol. 2017, pp. 1–8, feb 2017.[17] C. Liu, C. Chen, J. Han, and P. S. Yu, “GPLAG: detection of softwareplagiarism by program dependence graph analysis,” in
Proceedingsof the 12th ACM SIGKDD international conference on Knowledgediscovery and data mining - KDD ’06 . New York, New York, USA:ACM Press, 2006, p. 872.[18] M. El Bachir Menai and N. S. Al-Hassoun, “Similarity detection inJava programming assignments,” in . IEEE, aug 2010, pp. 356–361.[19] S. Engels, V. Lakshmanan, M. Craig, S. Engels, V. Lakshmanan, andM. Craig, “Plagiarism detection using feature-based neural networks,”
ACM SIGCSE Bulletin , vol. 39, no. 1, p. 34, mar 2007.[20] J. Y. Poon, K. Sugiyama, Y. F. Tan, and M.-Y. Kan, “Instructor-centricsource code plagiarism detection and plagiarism corpus,” in
Proceedingsof the 17th ACM annual conference on Innovation and technology incomputer science education - ITiCSE ’12 . New York, New York, USA:ACM Press, 2012, p. 122.[21] T. Parr,
The definitive ANTLR 4 reference . Pragmatic Bookshelf, 2013.[22] W. B. Croft, D. Metzler, and T. Strohman,
Search engines : informationretrieval in practice . Addison-Wesley, 2010.[23] M. J. Wise, “Neweyes: A System for Comparing Biological SequencesUsing the Running Karp-Rabin Greedy String-Tiling Algorithm,” in
International Conference on Intelligent Systems for Molecular Biology .AAAI, 1995.[24] M. S. Smith, Temple F; Waterman, “Identification of common molecularsubsequences,”
Journal of Molecular Biology , vol. 147, no. 1, pp. 195–197, mar 1981. [25] O. Karnalim and S. Budi, “The Effectiveness of Low-Level Structure-based Approach Toward Source Code Plagiarism Level Taxonomy,” in
The 6th International Conference on Information and CommunicationTechnology (ICoICT) . Bandung: IEEE, 2018.[26] S. K. Faidhi, J. A. W; Robinson, “An empirical approach for detectingprogram similarity and plagiarism within a university programmingenvironment,”
Computers & Education , vol. 11, no. 1, pp. 11–19, jan1987.[27] K. L. Verco and M. J. Wise, “Software for detecting suspected plagia-rism,” in
Proceedings of the first Australasian conference on Computerscience education - ACSE ’96 . New York, New York, USA: ACMPress, 1996, pp. 81–88.[28] O. Karnalim and L. Sulistiani, “Which Source Code Plagiarism Detec-tion Approach is More Humane?” in
The 9th International Conferenceon Awareness Science and Technology . Fukuoka: IEEE, 2018.[29] L. Sulistiani and O. Karnalim, “ES-Plag: Efficient and sensitive sourcecode plagiarism detection tool for academic environment,”
ComputerApplications in Engineering Education , sep 2018. [Online]. Available:http://doi.wiley.com/10.1002/cae.22066[30] O. Karnalim, “An Abstract Method Linearization for Detecting SourceCode Plagiarism in Object-Oriented Environment,” in
The 8th Inter-national Conference on Software Engineering and Service Science(ICSESS) . Beijing: IEEE, 2017.[31] O. Karnalim, “IR-based technique for linearizing abstract method invo-cation in plagiarism-suspected source code pair,”