Research Methods in Computer Science: The Challenges and Issues
aa r X i v : . [ c s . G L ] M a r Research Methods in Computer Science: TheChallenges and Issues
Hossein Hassani ∗†‡
Abstract
Research methods are essential parts in conducting any research project.Although they have been theorized and summarized based on best prac-tices, every field of science requires an adaptation of the overall approachesto perform research activities. In addition, any specific research needs aparticular adjustment to the generalized approach and specializing themto suit the project in hand. However, unlike most well-established sciencedisciplines, computing research is not supported by well-defined, glob-ally accepted methods. This is because of its infancy and ambiguity inits definition, on one hand, and its extensive coverage and overlap withother fields, on the other hand. This article discusses the research meth-ods in science and engineering in general and in computing in particular.It shows that despite several special parameters that make research incomputing rather unique, it still follows the same steps that any otherscientific research would do. The article also shows the particularitiesthat researchers need to consider when they conduct research in this field.
Dictionaries define research as “the systematic investigation into and study ofmaterials and sources in order to establish facts and reach new conclusions.(Fowler et al., 2011)” Scholars, for example, DePoy and Gitlin (2015) suggestthat the definition should be elaborated and provided in the context of the re-search area and the related branch of science. Therefore, after arguing on theiropinion that the broad definition of the research would not be of practical benefitin research, they provide their definition for their field of interest, which in thiscase is humanities, as “multiple, systematic strategies to generate knowledgeabout human behavior, human experience, and human environments in whichthinking and action processes of researcher are clearly specified so that theyare logical, understandable, confirmable, and useful” (DePoy and Gitlin, 2015). ∗ [email protected] † [email protected] ‡ Hossein Hassani is a lecturer at the University of Kurdistan Hewlˆer and a visiting lecturerat the Sarajevo School of Science and Engineering
Generalizing the results of or findings from some experiments in an area in away that could be applied or used beyond the specific area under investigationis the main goal of research (Hammersley and Traianou, 2012; De Vaus, 2002).From this perspective, the main goal of research remains the same, no matterof in what branch of a science it is conducted. This is also regardless of sciencecategorization. That is, for example, natural science, social science, appliedscience, behavioral science, and humanities share the same main goal in thereresearch. However, the way that scientists are looking at a phenomenon andraise questions about it significantly differs depending on the branch and cate-gory of the science that they are active in. The approaches that they take tosolve the related problems and to answer the questions also differ considerably.As a result, several paradigms about research have formed. Grbich (2013) de-fines paradigms as “worldviews of beliefs, values, and methods for collecting andinterpreting data. (Grbich, 2013, p. 5)” Denicolo and Becker (2012) define it as2a basic set of beliefs, views, values and assumptions that guide action and in-clude the researcher’s epistemological, ontlogoical and methodological premises(Denicolo and Becker, 2012)” , in which “epistemology” refers to the theory ofthe formation of knowledge and “ontolgoy” is the study of the things and thenature of being .Although similar enough to draw a general perspective, the categorizationof research paradigms varies among the scholars (see (Denicolo and Becker,2012; Grbich, 2013; Punch, 2000)). For example, Denicolo and Becker (2012)categorize the paradigms of research as positivism , post-positivism , con-structivism , and critical theory (Denicolo and Becker, 2012), while Grbich(2013) presents it as realism/postpositivism , critical theory , interpre-tivism/constructionism , postmodernism and poststructuralism , and mixed/multiple methods .Table 1 shows a brief description of different research paradigms. The def-initions are according to Denicolo and Becker (2012); other resource such asMackenzie and Knipe (2006) also provide these categorization with some moredetails.Despite the differences that can be seen in different research paradigms,in many situation a combination of approaches that theses paradigms suggestwould serve the research design much better than a single one. Therefore, amix/multi-methods paradigm which is obtained from an amalgamation of dif-ferent paradigms and their related methods has received more attention in recentstudies (Johnson et al., 2007). In fact, Ramesh et al. (2004) show that researchin computing have been conducted according to broad range of paradigms andapproaches.However, adapting a field specific approach is a necessary step in everybranch of science. To that extent, computer science as a fairly new discipline,suffers from “lack of identity”, although it combines the experience of its mainroots, namely, mathematics and engineering, (Demeyer, 2011). In contrast,there are others who believe that “computer science is a well-established disci-pline” that it has all it needs to be considered as any other sciences with a longhistory (Ramesh et al., 2004). In either case, the mentioned main roots affectthe formation of computing research paradigm. That is, a positivism/realism A more broader definition of epistemology has been given in (Wray, 2002, pp. 237-290). Although the concept is mainly discussed from social sciences perspective, it providesa thorough insight into the concept from several viewpoints and by different writers throughnine chapters. Ontology is a concept of philosophy. The word Ontology in the context of computer andinformations sciences has been specialized. This specialization has been provided in (Gruber,2015), which refers the definition of the word by Gruber (1993). This specialization has beenupdated in 2009 (Gruber, 2009). In addition, a more detailed discussion on the subject in thecontext of computing can be found in (Uschold and Gruninger, 1996). Nomothetic means generalizing ideas by finding common base and to extract abstractsfrom different related or correlated phenomenon. Idiographic approach means to study things from individuals view or the groups of peopleperspective. This is of very importance in social sciences. However, unless a computer scientistis not involved in an interdisciplinary research that has some social or life sciences concern,more details do not seem to be relevant. • Nomothetic – abstraction – law generation – generalization (univer-salization) – investigating and ex-plaining relationships(causal or correlation)between phenomena – finding and manipulat-ing related variables – evaluation of hypothe-sis • Deductive • Reductionist • Objectivist • Data collection – mainly quantitative • Dominance – natural science – life science • Idiographic (mainly) – specializing – unique understanding – individualization – defining research ques-tions – finding answers to theresearch questions • Phenomenological • Interpretivist • Subjectivist • Data collection – mainly qualitative • Dominance – social science – humanitiesTable 1: Current research paradigmswhich is the main paradigm of natural and life sciences, is applied to computingas well (Denicolo and Becker, 2012). Despite uncountable books, articles, and discussions about research method and methodology , finding a straightforward definition seems not to be easy.4n fact, the issue is with interpretation of methodology rather than method.Some scholars, such as Clough and Nutbrown (2012), for example, have triedto discuss the terms in more detail. However, they do not seem to be giv-ing a clear-cut and concise differentiation between the two terms. Similarly,McGregor and Murnane (2010) have provided some more explanation, men-tioning that “the word methodology comprises two nouns: method and ology,which means a branch of knowledge”. This is also similar to what can be found in(Online Etymology Dictionary, 2015). Differently, (Merriam-Webster Dictionary,2015) sets origin of methodology to “New Latin methodologia, from Latinmethodus + -logia -logy” and dates it back to 1800. Yet another article by(Lehaney and Vinten, 1994) focuses on the usage of the term in the specificcontext and tries to enlighten the readers about the confusions around this par-ticular usage. But despite all efforts, it seems that “methodology” continuesto remain as a confused term in the research community. Particularly, it hasbeen taken for granted in many ways and has interchangeably been used along-side “method” in many resources. However, in this article we try to clearlydifferentiate these two in a way that follows.We consider “method”, in the context of research, as an approach, proce-dure, and guidelines that are used in conducting a research. A method mightrequire different tools, instruments, equipments, and such. Whereas we consider“methodology” as a scientific approach that investigates, compares, contrasts,and explains the different ways that a research could be conducted alongsidedifferent methods that could be used in these processes. That is, methodologydiscusses the alternative approaches and methods to tackle the research problem.It discusses the advantages/disadvantages, properness/improperness, feasibility,practicality, ethical issues, and such parameters for the approaches to do theresearch. Throughout its discussion, the methodology, as a main ingredient ofthe research, clarifies why a particular approach has been taken to address the“research question(s)” and how this approach would be implemented.Based on what is mentioned, the research methodology should reflect onthe nature of the research and help the researcher to tackle the research area,properly. For this purpose, the researcher should find out, through theoreti-cal/factual discussions, the research category and the paradigm, which bettershow the characteristics of the research and serve the research to be conductedmore properly. For this, Baban (2009) categorizes research based on three mainthemes, which have been summarized as below:1. The application of the research study • Pure research
It aims in discovering new knowledge without ex-pecting an instant affect on the current situation of the field. • Applied research
It aims in solving a specific problem, which iscurrently the concern of the field.2. The objectives in undertaking the research5
Descriptive research It aims in explaining the situation and char-acteristic of a specific problem in order to benefit from it in otherresearch. • Exploratory research It aims in finding proper information in thearea within which researcher cannot find previous information in or-der to build a profound hypothesis. • Correlational research It aims in discovering the correlations amongdifferent variables of problem area in order to recognize the impactsof a phenomenon. • Explanatory research It aims in explaining the reasons behind thecharacteristics of a phenomenon (answering to the why) or how thecharacteristics of a phenomenon forming it. • Analytical research - It can be considered as an extension to thedescriptive research because it does not stay at the description level,and moves beyond that to discover the reasons behind a problem orthe behavior of a phenomenon.3. The type of information sought • Positivism - see Table 1. • Phenomenological - see Table 1.To set a proper paradigm and to suggest well-suited methods that couldbest serve the research purpose are paramount to the research. (Baban, 2009,pp. 28-29 ) discusses both quantitative and qualitative approach based on certainassumptions that researchers may make. These assumptions are ontologicalassumptions, epistemological assumptions, axiological assumptions, rhetoricalassumptions, and methodological assumptions. Others such as Saunders et al.(2007) also give similar perspectives in this regard. Although it seems that theseare different opinions, but the conclusions are similar, and the main differencesremain in the way that the ideas are presented.
Dodig-Crnkovic (2002), quoting Dijkstra, mentions that computer science de-partments, have, under external pressures, underemphasized the “science” as-pects of the knowledge area in favor of “computer” that is a tool not a science;this implies that, for example, the surgeons call surgery a “knife science”, or hav-ing “car engineering”, “train engineering”, or rather “car engine engineering”!This approach focuses on computers as a tool rather than appreciation of the-oretical aspects and abstract elements of this science, such as mathematics andlogic, one the one hand, and undermines its engineering elements as a necessarypart, particularly, in software engineering, on the other hand (Dodig-Crnkovic,2002). Similarly, long while ago in 1970s, Newell and Simon (1976) stated that6Computer science is an empirical discipline. We would have called it an exper-imental science, but like astronomy, economics, and geology, some of its uniqueforms of observation and experience do not fit a narrow stereotype of the exper-imental method (Newell and Simon, 1976, p. 14).” Dodig-Crnkovic (2002) alsorefers to the three fundamental recurring concepts of computing, which are (a)conceptual and formal models, (b) levels of abstraction, and (c) efficiency.So what is computing? How we can define it? To answer this questions isnot as easy as it seems to be at the first glance (Snyder et al., 1994). We haveobserved this confusion among applicants who are interested in computing, butthey do not know which sector of computing should they choose, because thedifference is not made clear for them. However, with regard to undergradu-ate study in computing there is a consensus among the majority of academicson differences and commonalities among different sectors of computing (forexample, see (ACM/IEEE-CS Joint Task Force on Computing Curricula, 2013,2014; Pyster et al., 2009; Topi et al., 2010)).According to ACM/IEEE-CS Joint Task Force on Computing Curricula (2013),“omputing is a broad field that connects to and draws from many disciplines,including mathematics, electrical engineering, psychology, statistics, fine arts,linguistics, and physical and life sciences.” Speaking about its past and cur-rent situation, Denning (2013) state: “computing began as science, morphedinto engineering for 30 years while it developed technology, and then entereda science renaissance about 20 years ago. Although computing had subfieldsthat demonstrated the ideals of science, computing as a whole has only recentlybegun to embrace those ideals. Some new subfields such as network science,network social science, design science, and Web science, are still struggling toestablish their credibility as sciences.” (Denning, 2013, p. 32 )We discussed the research methods and methodology in the context of sci-ence and humanities. But how this applies to computing? Milner (1986), in afascinating inaugural lecture to the opening of the Laboratory for Foundationsof Computer Science at the University of Edinburgh, has provided answers toa question that seems to be still valid to many people, after some 30 yearspassed since the lecture was given. That is, “Is Computing an ExperimentalScience?”. In fact, there is yet a more rudimentary question: Is Computing aScience? or even: Is Computer Science a Science? It must have been taken forgranted that the answers to these questions are rather a simple “yes!”, however,this is not the case (Dodig-Crnkovic, 2002), and unless we are able to includecomputing and computer science as valid members of science, we are not able toapply scientific approaches to their research. This, reminds me of a colleague’ssaying, an academic in humanity, who was showing his full surprise about howcould people in computing, particularly in software engineering, receive a PhDand call themselves a “Doctor of Philosophy” in a subject that, in his opinion,neither was a science nor engineering!Although the phrase “Computing is a science.” seems to be an axiom, forsome scholars it is not. National Academy of Science elaborated on this matterin 1992. It refers to both scientific and engineering aspects of computing andrelates the first to the mathematical and engineering models, based on theory7nd abstraction, whereas relates the second to the practical application, basedon abstraction and design (Hartmanis and Lin, 1992). They also compare theobject of study in computer science with other branches of science. For example,if the object of research in physics is atom, or in biology is a cell, then “focuson information, the ways of presenting information, and on the machines andsystems that perform these tasks” are the objects of study in computer science(Hartmanis and Lin, 1992).Regardless of significant advancement in computing, and no matter if com-puters are becoming a central player in almost every aspect of life in the newmillennium, some scientists still believe that “computing science is an imma-ture discipline.” Johnson (2006) argues on the issues of the advancement incomputing technology and academic research. From his perspective, althoughusing hermeneutics in requirement analysis, and mathematical models to specifyand verify complex systems, for example, has been beneficial to the computingresearch, “lack of any agreed research framework reflects the strength and vital-ity of computing science”. As a result, (Johnson, 2006) encourages computingresearchers contemplate on the various aspects of the research methods theyadopt and critically incorporate them into their own research. He says:“Too often, MSc and PhD theses slavishly follow empirical or formal proof tech-niques without questioning the suitability of those approaches. For example, thehermeneutic tradition has delivered results that ignore the constraints of timeand money on commercial system development. Formal methods research hasproduced results that abstract so far away from the problem domain that theycannot be applied or validated. The tragedy is that unless we begin to recog-nise these failures then we will continue to borrow flawed research methods fromother disciplines (Johnson, 2006).”As a result, research in computing might be of theoretical or experimentalnature or a combination of them; it appreciates different paradigmatic views,and utilizes best suited tools and approaches from both quantitative and qual-itative methods. One could argue that every other science would do the same,so what is the difference? Well, this article does not intend to make a researchin computing a different “thing”, rather it argues for the similarity of researchin computing and other branches of science, while it recognizing its especialcharacteristics that makes it unique as any other branch of science.
An analogy between research an onion might make the concept more under-standable. Some sources have referred to this a “research onion” (see (Saunders et al.,2007, p. 102)). Although this multitude layers that are overlapping each othermight seem complicated, they are very helpful in the discussion and the designof the research methodology. Particularly, in computer science, to understandwhether the research in hand is a theoretical research (sometime it is called ba-8ic research (Kendal, 2015; Saunders et al., 2007)) or experimental one, is a keyquestion which significantly affects the methodology of the research. Now, goingback to the questions that (Milner, 1986) asked (see 4), he clearly showed thathe preferred to have a convergence between theory and experiment. He providesand analogy by giving examples of physicists an chemists who improved and re-fined their theories through taking experiments and suggests the same approachto be taken for computer science, hence he deduces that computer science is asexperimental as it is theoretical (Milner, 1986).Snyder et al. (1994) define Experimental Computer Science (ECS) as “thebuilding of, or the experimentation with or on, nontrivial hardware or softwaresystems.” In this view, computer science and engineering (CS&E) should beconsidered as a whole if one wants to discuss them in the context of experimen-tal research. Johnson (2000), who was awarded the 2010 Knuth Prize, assumesthat “science is the search for the fundamental principles that govern the worldaround us and explain the phenomena we see”, and then suggests that “theTheoretical Computer Science (TCS) is the “science” underlying the field ofcomputing”. He then concludes that as the computation is basically a discretelogical process, the formal and mathematical nature of TCS is especially ap-propriate for a science of computing. He also adds that theory is a significantingredient to not only computing but also for its interdisciplinary characteristics(Johnson, 2000).Professional bodies also have defined computing and its branches. Belowwe refer to two quotes. The first one emphasizes the importance of algo-rithms and their application. This view shows how both theoretical and prac-tical aspects of computing work together and in fact, in some research casesare inseparable. “An important part of computing is the ability to select al-gorithms appropriate to particular purposes and to apply them, recognizingthe possibility that no suitable algorithm may exist. This facility relies onunderstanding the range of algorithms that address an important set of well-defined problems, recognizing their strengths and weaknesses, and their suitabil-ity in particular contexts. Efficiency is a pervasive theme throughout this area.(ACM/IEEE-CS Joint Task Force on Computing Curricula, 2013, p. 55 )”The second quote shows how computing and computer science need to focuson abstraction, which is in turn a reason for looking in some computer scienceresearch as Nomothetic from paradigmatic point of view. “Abstraction is a fun-damental concept in computer science. A principal approach to computing is toabstract the real world, create a model that can be simulated on a machine. Theroots of computer science can be traced to this approach, modeling things suchas trajectories of artillery shells and the modeling cryptographic protocols, bothof which pushed the development of early computing systems in the early andmid-1940s. (ACM/IEEE-CS Joint Task Force on Computing Curricula, 2013,p. 70 )”Finally, there is on aspect of research that is growing steadily among differ-ent fields and branches of science and humanities, which is the interdisciplinarycharacteristics of recent studies. Snyder et al. (2004) define interdisciplinary re-search as: “[p. 26]nap2004interdiscipInterdisciplinary research (IDR) is a mode9f research by teams or individuals that integrates information, data, techniques,tools, perspectives, concepts, and/or theories from two or more disciplines orbodies of specialized knowledge to advance fundamental understanding or tosolve problems whose solutions are beyond the scope of a single discipline or areaof research practice.” They also suggest how to evaluate a proposal for its disci-plinary coverage. This is how they stated: “[p. 169]nap2004interdiscipEvaluatea proposal to its cell-biology research program by using researchers in cell biologyand including a substantial number in chemistry, physics, computer science, thesocial sciences, and the humanities as appropriate; this practice would help toensure disciplinary breadth and reduce bias.” In fact, many contemporary prob-lems cannot be solved through one aspect of knowledge (Snyder et al., 2004).Computing plays a great role in this aspect of research. As the result, we can ex-pect more and more interdisciplinary, cross-disciplinary, and multi-disciplinaryresearch that is one way or another has utilized computing, or rather has beenintertwined with computing.
Wilson (1952) believes that “many scientists owe their greatness not to their skillin solving problems but to their wisdom in chossing them.” He also states that“the most rewarding work is usually to explore a hithherto untouched field”,which “are not easy to find today. Wilson (1952) also states that ” Althoughfrom the day that these have been said, the research field has dramaticallychanged, but his own “wisdom” is more appreciated when one reads his bookin the context of computing, a science which was in its infancy at the time.The following quotation from his book is still invaluable, particularly, when onelooks at different research questions in computing and one wants to choose apath for the research in this area. “A research worker in pure science who doesnot have at all times more problems he would like to solve than he has timeand means to investigate them probably is in the wrong business. He may bean excellent experimenter and may have all the qualities required for success inapplied science, but he lacks qualities of mind important for pure science. Thisis not at all to imply that applied science is easier, less demanding, on in anyway inferior to pure science; it requires its own special abilities, but they aresomewhat different (Wilson, 1952, p. 2 ).”In the following sections, some sample research topics for both TCS and ECSare listed. The topics have been selected from a collection obtained throughusing different search engines. For each topic a brief description is providedthat shows their main focus area.
Theoretical Computer Science (TCS), as it was mentioned earlier in this section,encompasses the formal rules, mainly based on mathematics and logic, which areunderlying computing science as whole. Therefore, most of the research casesin this field form hypotheses that lead to generalization of findings in order to10orm a theorem or a formal model, or suggest improvements to the previousformal models and algorithms, or other kinds of theorization that expands thescientific background of this field of computing.For example, below are a list of articles which are related to TCS research: • A fast string searching algorithm - This research is about improving asearching algorithm, providing a theoretical analysis of the suggested im-provements (Boyer and Moore, 1977). • The smallest automation recognizing the subwords of a text - Automata,finite automaton, and deterministic finite automaton (DFA) have beenessential parts of theoretical computer science for a long while. This re-search provides an algorithm to build a smallest partial DFA for a certainproblem (Blumer et al., 1985). • A faster algorithm for testing polynomial representability of functions overfinite integer rings - This research is also an improvement to an alreadydevised algorithm in polynomial representability. Reading the article thatdescribes this research, one, at the first glance, would say that is a researchin mathematics (Guha and Dukkipati, 2015). However, when it is readcarefully, the algorithms that have been provided explain why this researchhas happened in the computer science area. • Categorial dependency grammars - Formal grammars are another essentialpart of TCS that are of different usage in programming languages andother formal language processing in computing. This paper provides an“abstract theoretical version of sub-commutative” categrial dependencygrammars (Dekhtyar et al., 2015). • Nearly private information retrieval - This is a research concerning theprivacy of data that suggest improvement to the previous approachesfor keeping the data retrieval safe and secure (Chakrabarti and Shubina,2007).The above list and the brief explanations show the theoretical theme that isflowing in these type of research. This, as can be seen below, is different fromwhat ECS targets.
Experimental Computer Science (ECS) is the body of best practices, methods,procedures, and techniques that assist the practitioners of computing in movingthe computer science from its theoretical base towards an applied science. Al-though the computing seems to be an experimental area, research showed thatup to 1995 this was not the case, at least by assessing the published results(Tichy et al., 1995). However, it does not mean that the theoretical researchwas a dominant area. In fact, according to Tichy et al. (1995), about 70% ofpublished papers by ACM (Association for Computing Machinery) was rather11esign and modeling. A quick search using different search engines are stillshowing that this situation has continued to some extent, which is confirmedby Wainer et al. (2009) in as well. Hence a call for a cultural change in com-puter science towards showing more appreciation for experimental approacheswas still there in 2006 (Feitelson, 2006).Despite this situation, we can find experimental research of high qualitynowadays. Below some samples are mentioned. However, it is still to soon tosay that the research in computer science is a well-established discipline. • Verification and change-impact analysis of access-control policies - Thisresearch investigates the data access-control policies through using a soft-ware, which is called Margrave. The aim is to measure how changes inthe policies would affect the performance (Fisler et al., 2005). • A two-tier test approach to developing location-aware mobile learning sys-tems for natural science courses - This research conducts experiments toassess the effectiveness of mobile learning system on elementary schoolstudents (Chu et al., 2010).Having considered the two main areas of computer science research, it is alsoseen that sometime researchers talk about “emperical” computer science. Bythe same analogy that emperical research has been distinguished from experi-mental research in other sciences, these two are also distinguished in computingand computer science. Nevertheless, emperical research has been given some es-pecial attention in Software Engineering area of computing (Perry et al., 2000;Wohlin et al., 2003; Easterbrook et al., 2008).
Research is one of the pillars of advancement in science and technology. It isa methodical approach for finding answers to the problems through investiga-tion and experimentations by which researchers evaluate a hypothesis, provideanswers to the research questions, or suggest solutions to certain problems. Al-though the main goal of research is the same for all branches of science andhumanities, the characteristics of each branch requires a specific adaptation ofthe methods which are applicable for the research. To choose a proper methodand to design the way that the research should be carried out, researchers shoulddiscuss and assess these methods in the context of the research. This processand its outcome, together, is called research methodology.Computing, in general, and computer science, in particular are relativelynew sciences. Although the convergence of different branches of science is a The author has experienced the confusion and misunderstanding about research in com-puting in both industry and academy. It still seems difficult to convince the students tocompletely differentiate between a software development, for example, with an experimentalresearch in computing. The research methods, in general, seem to be much appreciated andunderstood by students of other fields of science (social or natural) and engineering ratherthan computing. production projects and applied/experimental/theoretical researchprojects . References
ACM/IEEE-CS Joint Task Force on Computing Curricula (2013, December).Computer Science Curricula 2013. Technical report, ACM Press and IEEEComputer Society Press.ACM/IEEE-CS Joint Task Force on Computing Curricula (2014). SoftwareEngineering 2014. Technical report, ACM Press and IEEE Computer SocietyPress.Baban, S. M. (2009).
Research: The Journey from Pondering to Publishing .Canoe Press.Blumer, A., J. Blumer, D. Haussler, A. Ehrenfeucht, M.-T. Chen, and J. Seiferas(1985). The smallest automation recognizing the subwords of a text.
Theo-retical Computer Science 40 , 31–55.Boyer, R. S. and J. S. Moore (1977). A fast string searching algorithm.
Com-munications of the ACM 20 (10), 762–772.13hakrabarti, A. and A. Shubina (2007). Nearly private information retrieval. In
Mathematical Foundations of Computer Science 2007 , pp. 383–393. Springer.Chu, H.-C., G.-J. Hwang, C.-C. Tsai, and J. C. Tseng (2010). A two-tier testapproach to developing location-aware mobile learning systems for naturalscience courses.
Computers & Education 55 (4), 1618–1627.Clough, P. and C. Nutbrown (2012).
A student’s guide to methodology (3 ed.).Sage.De Vaus, D. (2002).
Analyzing social science data: 50 key problems in dataanalysis . Sage.Dekhtyar, M., A. Dikovsky, and B. Karlov (2015). Categorial dependency gram-mars.
Theoretical Computer Science 579 , 33–63.Demeyer, S. (2011). Research methods in computer science. In .Denicolo, P. and L. Becker (2012).
Developing research proposals . Sage.Denning, P. J. (2013). The science in computer science.
Communications of theACM 56 (5), 35–38.DePoy, E. and L. N. Gitlin (2015).
Introduction to research: Understanding andapplying multiple strategies . Elsevier Health Sciences.Dodig-Crnkovic, G. (2002). Scientific methods in computer science. In
Proceed-ings of the Conference for the Promotion of Research in IT at New Universi-ties and at University Colleges in Sweden, Sk¨ovde, Suecia , pp. 126–130.Easterbrook, S., J. Singer, M.-A. Storey, and D. Damian (2008). Selectingempirical methods for software engineering research. In
Guide to advancedempirical software engineering , pp. 285–311. Springer.Feitelson, D. G. (2006). Experimental computer science: The need for a culturalchange. .Fisler, K., S. Krishnamurthi, L. A. Meyerovich, and M. C. Tschantz (2005). Ver-ification and change-impact analysis of access-control policies. In
Proceedingsof the 27th international conference on Software engineering , pp. 196–205.ACM.Fowler, H. W., F. G. Fowler, and D. Crystal (2011).
The Concise Oxford Dic-tionary: The Classic First Edition . Oxford University Press.Grbich, C. (2013).
Qualitative data analysis: An introduction (2 ed.). Sage.Gruber, T. (2009). Ontology.
Encyclopedia of database systems , 1963–1965.Gruber, T. (2015, Oct). What is an Otology?14ruber, T. R. (1993). A translation approach to portable ontology specifica-tions.
Knowledge acquisition 5 (2), 199–220.Guha, A. and A. Dukkipati (2015). A faster algorithm for testing polynomialrepresentability of functions over finite integer rings.
Theoretical ComputerScience 579 , 88–99.Hammersley, M. and A. Traianou (2012).
Ethics in qualitative research: Con-troversies and contexts . Sage.Hartmanis, J. and H. Lin (Eds.) (1992).
Computing the Future: A BroaderAgenda for Computer Science and Engineering . National Academies Press.Johnson, C. (2006). What is research in computing science.Johnson, D. (2000). Challenges for theoretical computer science: Draft.Johnson, R. B., A. J. Onwuegbuzie, and L. A. Turner (2007). Toward a definitionof mixed methods research.
Journal of mixed methods research 1 (2), 112–133.Kendal, S. (2015).
How to Write a Research Paper . bookboon.com.Lehaney, B. and G. Vinten (1994). Methodology: An Analysis of Its Meaningand Use.
Work Study 43 (3), 5–8.Mackenzie, N. and S. Knipe (2006). Research dilemmas: Paradigms, methodsand methodology.
Issues in educational research 16 (2), 193–205.McGregor, S. L. and J. A. Murnane (2010). Paradigm, methodology andmethod: Intellectual integrity in consumer scholarship.
International Journalof Consumer Studies 34 (4), 419–427.Merriam-Webster Dictionary (2015, Sep). Merriam-Webster Online Dictionary.Milner, R. (1986). Is computing an experimental Science?Newell, A. and H. A. Simon (1976). Computer science as empirical inquiry:Symbols and search.
Communications of the ACM 19 (3), 113–126.Online Etymology Dictionary (2015, Sep). Online Etymology Dictionary.Perry, D. E., A. A. Porter, and L. G. Votta (2000). Empirical studies of softwareengineering: a roadmap. In
Proceedings of the conference on The future ofSoftware engineering , pp. 345–355. ACM.Punch, K. (2000).
Developing effective research proposals . Sage.Pyster, A. et al. (2009). Graduate Software Engineering 2009 (GSwE2009) Cur-riculum Guidelines for Graduate Degree Programs in Software Engineering.Technical report. 15amesh, V., R. L. Glass, and I. Vessey (2004). Research in computer science:an empirical study.
Journal of systems and software 70 (1), 165–176.Saunders, M., P. Lewis, and A. Thornhill (2007).
Research methods for businessstudents (4 ed.). Pearson Education UK.Snyder, L. et al. (1994).
Academic careers for experimental computer scientistsand engineers . National Academies Press.Snyder, L. et al. (2004).
Facilitating Interdisciplinary Research . NationalAcademies Press.Tichy, W. F., P. Lukowicz, L. Prechelt, and E. A. Heinz (1995). Experimentalevaluation in computer science: A quantitative study.
Journal of Systemsand Software 28 (1), 9–18.Topi, H., K. M. Kaiser, J. C. Sipior, J. S. Valacich, J. Nunamaker Jr,G. de Vreede, and R. Wright (2010). Curriculum Guidelines for Undergrad-uate Degree Programs in Information Systems. Technical report.Uschold, M. and M. Gruninger (1996, 6). Ontologies: principles, methods andapplications.
The Knowledge Engineering Review 11 , 93–136.Wainer, J., C. G. N. Barsottini, D. Lacerda, and L. R. M. de Marco (2009).Empirical evaluation in Computer Science research published by ACM.
In-formation and Software Technology 51 (6), 1081–1085.Wilson, E. B. (1952).
An introduction to scientific research . Courier Corpora-tion. Has been reprinted several tiems and in different formats.Wohlin, C., M. H¨ost, and K. Henningsson (2003). Empirical research meth-ods in software engineering. In
Empirical methods and studies in softwareengineering , pp. 7–23. Springer.Wray, K. B. (Ed.) (2002).