Maram Hasanain

Ethics in engineering education: A literature review

Engineering Ethics is an important topic to be developed in engineering education curriculum. Despite its importance, ethics is not much investigated in engineering education as compared to other disciplines, in particular medicine or biology education. In this paper, a comprehensive review of engineering ethics is provided. The review covers three main topics: 1) Attributes of ethical engineers, 2) Content, logistics and pedagogy of engineering ethics, and 3) Assessment of engineering ethics. A particular focus is given to the Defining Issues Test (DIT) and the Engineering and Science Issues Test (ESIT) that is considered a promising instrument to assess moral judgment development of science and engineering students. Final remarks will conclude the paper.

EveTAR: A New Test Collection for Event Detection in Arabic Tweets

Research on event detection in Twitter is often obstructed by the lack of publicly-available evaluation mechanisms such as test collections; this problem is more severe when considering the scarcity of them in languages other than English. In this paper, we present EveTAR, the first publicly-available test collection for event detection in Arabic tweets. The collection includes a crawl of 590M Arabic tweets posted in a month period and covers 66 significant events (in 8 different categories) for which more than 134k relevance judgments were gathered using crowdsourcing with high average inter-annotator agreement (Kappa value of 0.6). We demonstrate the usability of the collection by evaluating 3 state-of-the-art event detection algorithms. The collection is also designed to support other retrieval tasks, as we show in our experiments with ad-hoc search systems.

On the Evaluation of Tweet Timeline Generation Task

Tweet Timeline Generation (TTG) task aims to generate a timeline of relevant but novel tweets that summarizes the development of a given topic. A typical TTG system first retrieves tweets then detects novel tweets among them to form a timeline. In this paper, we examine the dependency of TTG on retrieval quality, and its effect on having biased evaluation. Our study showed a considerable dependency, however, ranking systems is not highly affected if a common retrieval run is used.

Improving Tweet Timeline Generation by Predicting Optimal Retrieval Depth

Tweet Timeline Generation (TTG) systems provide users with informative and concise summaries of topics, as they developed over time, in a retrospective manner. In order to produce a tweet timeline that constitutes a summary of a given topic, a TTG system typically retrieves a list of potentially-relevant tweets over which the timeline is eventually generated. In such design, dependency of the performance of the timeline generation step on that of the retrieval step is inevitable.

Query performance prediction for microblog search: a preliminary study

Microblogging has recently become an integral part of the daily life of millions of people around the world. With a continuous flood of posts, microblogging services (e.g., Twitter) have to effectively handle millions of user queries that aim to search and follow recent developments of news or events. While predicting the quality of retrieved documents against search queries was extensively studied in domains such as the Web and news, the different nature of data and search task in microblogs triggers the need for re-visiting the problem in that context. In this work, we re-examined several state-of-the-art query performance predictors in the domain of microblog ad-hoc search using the two most-commonly used tweets collections with three different retrieval models that are used in microblog search. Our experiments showed that a temporal predictor was generally the best to fit the prediction task in the context of microblog search, indicating the importance of the temporal aspect in this task. The results also highlighted the need to either re-design some of the existing predictors or propose new ones to function effectively with different retrieval models that are used in our tested domain. Finally, our experiments on combining multiple predictors resulted in achieving considerable improvements in prediction quality over individual predictors, which confirmed the results reported in the literature but in different domains.

EveTAR : building a large-scale multi-task test collection over Arabic tweets

This article introduces a new language-independent approach for creating a large-scale high-quality test collection of tweets that supports multiple information retrieval (IR) tasks without running a shared-task campaign. The adopted approach (demonstrated over Arabic tweets) designs the collection around significant (i.e., popular) events, which enables the development of topics that represent frequent information needs of Twitter users for which rich content exists. That inherently facilitates the support of multiple tasks that generally revolve around events, namely event detection, ad-hoc search, timeline generation, and real-time summarization. The key highlights of the approach include diversifying the judgment pool via interactive search and multiple manually-crafted queries per topic, collecting high-quality annotations via crowd-workers for relevancy and in-house annotators for novelty, filtering out low-agreement topics and inaccessible tweets, and providing multiple subsets of the collection for better availability. Applying our methodology on Arabic tweets resulted in EveTAR, the first freely-available tweet test collection for multiple IR tasks. EveTAR includes a crawl of 355M Arabic tweets and covers 50 significant events for which about 62K tweets were judged with substantial average inter-annotator agreement (Kappa value of 0.71). We demonstrate the usability of EveTAR by evaluating existing algorithms in the respective tasks. Results indicate that the new collection can support reliable ranking of IR systems that is comparable to similar TREC collections, while providing strong baseline results for future studies over Arabic tweets.

Query performance prediction for microblog search

Abstract Query performance prediction (QPP) is the task of estimating the effectiveness of a retrieval system given a search query in the absence of any feedback from the searcher. The task has been proven to be very challenging, and thus it attracted a lot of research attention in domains like news and Web retrieval. However, search in microblogs poses new challenges for the task due to the more prevalent temporality in microblogs and the different types of information needs in such domain. In this work, we aim at studying QPP for microblog search. We conducted large-scale experiments, testing 37 state-of-the-art predictors using several types of retrieval models usually used in microblog search. Moreover, we propose a set of predictors that exhibit statistically-significant improvements over the state-of-the-art predictors with the maximum percentage of improvement reaching 55% over all studied retrieval settings. Further experimental explorations show that using expanded queries in predicting the performance of query expansion models gives much better prediction quality than using the original queries, and that the examined predictors were generally much more effective over temporal queries compared to non-temporal ones; both phenomena have never been studied in the context of microblog search before. As microblog search is considered a major step in several retrieval tasks in the domain (such as timeline generation, summarization, and question answering), improving QPP for microblog search has a high potential to help improve the effectiveness of those closely-related tasks.

Automatic Ranking of Information Retrieval Systems

Typical information retrieval system evaluation requires expensive manually-collected relevance judgments of documents, which are used to rank retrieval systems. Due to the high cost associated with collecting relevance judgments and the ever-growing scale of data to be searched in practice, ranking of retrieval systems using manual judgments is becoming less feasible. Methods to automatically rank systems in absence of judgments have been proposed to tackle this challenge. However, current techniques are still far from reaching the ranking achieved using manual judgments. I propose to advance research on automatic system ranking using supervised and unsupervised techniques.

When Rank Order Isn't Enough: New Statistical-Significance-Aware Correlation Measures

Because it is expensive to construct test collections for Cranfield-based evaluation of information retrieval systems, a variety of lower-cost methods have been proposed. The reliability of these methods is often validated by measuring rank correlation (e.g., Kendalls tau) between known system rankings on the full test collection vs. observed system rankings on the lower-cost one. However, existing rank correlation measures do not consider the statistical significance of score differences between systems in the observed rankings. To address this, we propose two statistical-significance-aware rank correlation measures, one of which is a head-weighted version of the other. We first show empirical differences between our proposed measures and existing ones. We then compare the measures while benchmarking four system evaluation methods: pooling, crowdsourcing, evaluation with incomplete judgments, and automatic system ranking. We show that use of our measures can lead to different experimental conclusions regarding reliability of alternative low-cost evaluation methods.

ACM SIGIR Student Liaison Program

ACM SIGIR has recently created the Student Liaison Program, a means to connect and stay connected with the student body of the information retrieval (IR) community. This report provides more information about the program, introduces the founding ACM SIGIR student liaisons, and explains past, ongoing, and future activities. We seek suggestions and recommendations on the current plans as well as the new ideas that fit into our mission.