Agus Trisnajaya Kwee

Sentence-Level Novelty Detection in English and Malay

Novelty detection (ND) is a process for identifying information from an incoming stream of documents. Although there are many studies of ND on English language documents, however, to the best of our knowledge, none has been reported on Malay documents. This issue is important because there are many documents with a mixture of both English and Malay languages. This paper examines multilingual sentence-level ND in English and Malay documents using TREC 2003 and TREC 2004 Novelty Track data. We describe the text processing for multilingual ND, which consists of language translation, stop words removal, automatic stemming, and novel sentence detection. We compare the results for sentence-level ND on English and Malay documents and find that the results are fairly similar. Therefore, after preprocessing is performed on Malay documents, our ND algorithm appears to be robust in detecting novel sentences, and can possibly be extended to other alphabet-based languages.

Detecting novel business blogs

Nowadays, blogs have become a popular way for people and organizations to publish news and information to the public, as well as advertise their products. Certain information contained in the blogs represent business opportunities and thus can be utilized by speculators. However, many blogs may often contain similar information and the sheer volume of available information really challenges the ability of organizations to act quickly in todays business environment. Thus, a lot of work has been done to develop a novelty detection system, which is able to single out novel information out of a set of massive text documents. This paper explores the feasibility and performance of detecting novel business blogs, which has not been studied before. The results show that our novelty detection system can detect novelty in our dataset of business blogs with very high accuracy, which indicate that our novelty detection algorithm is successful in detecting novel business blogs.

Multilingual sentence categorization and novelty mining

A challenge for sentence categorization and novelty mining is to detect not only when text is relevant to the users information need, but also when it contains something new which the user has not seen before. It involves two tasks that need to be solved. The first is identifying relevant sentences (categorization) and the second is identifying new information from those relevant sentences (novelty mining). Many previous studies of relevant sentence retrieval and novelty mining have been conducted on the English language, but few papers have addressed the problem of multilingual sentence categorization and novelty mining. This is an important issue in global business environments, where mining knowledge from text in a single language is not sufficient. In this paper, we perform the first task by categorizing Malay and Chinese sentences, then comparing their performances with that of English. Thereafter, we conduct novelty mining to identify the sentences with new information. Experimental results on TREC 2004 Novelty Track data show similar categorization performance on Malay and English sentences, which greatly outperform Chinese. In the second task, it is observed that we can achieve similar novelty mining results for all three languages, which indicates that our algorithm is suitable for novelty mining of multilingual sentences. In addition, after benchmarking our results with novelty mining without categorization, it is learnt that categorization is necessary for the successful performance of novelty mining.

Database optimization for novelty detection

Research in the area of optimizing databases in any Database Management System (DBMS) has been evolving constantly. Today, programming languages are being integrated into database systems to help professional programmers develop software quickly to meet deadlines. Therefore, the design of a database must cater to both the needs of customers and the efficiency of database processes. In this paper, a database application, novelty detection, is used to detect new documents for readers who do not want redundant documents to be read again. This application needs a database to store history and current documents. The objective of this research is to optimize the database tables for up to 10 million records. The experiments are done on both sentence level and document level. In both levels, the investigation of data optimization and the use of proper indexing are conducted. In MYSQL, the MYSQL B-Tree index is used to speed up data selection. In addition, the use of EXPLAIN enables us to properly index the correct data column and to avoid redundant indexing. Optimizing data types are also investigated to ensure no extra work is done by MYSQL in selecting data. A technique known as batching is also introduced to speed up results insertion after novelty detection has been done. Overall, the combined optimization improved the speed by up to 90%. Therefore, we have successfully optimized the database for novelty detection, and the techniques have been integrated into a real-time novelty detection application.

Experiments in term weighting for novelty mining

Abstract Obtaining new information in a short time is becoming crucial in today’s economy. A lot of information both offline or online is easily acquired, exacerbating the problem of information overload. Novelty mining detects documents/sentences that contain novel or new information and presents those results directly to users ( Tang, Tsai, & Chen, 2010 ). Many methods and algorithms for novelty mining have previously been studied, but none have compared and discussed the impact of term weighting on the evaluation measures. This paper performed experiments to recommend the best term weighting function for both document and sentence-level novelty mining.

ViStruclizer: A Structural Visualizer for Multi-dimensional Social Networks

With the popularity of Web 2.0 sites, social networks today increasingly involve different kinds of relationships among different types of users in a single network. Such social networks are said to be multi-dimensional. Analyzing multi-dimensional networks is a challenging research task that requires intelligent visualization techniques. In this paper, we therefore propose a visual analytics tool called ViStruclizer to analyze structures embedded in a multi-dimensional social network. ViStruclizer incorporates structure analyzers that summarize social networks into both node clusters each representing a set of users, and edge clusters representing relationships between users in the node clusters. ViStruclizer supports user interactions to examine specific clusters of users and inter-cluster relationships, as well as to refine the learnt structural summary.

Database optimization for novelty mining of business blogs

The widespread growth of business blogs has created opportunities for companies as channels of marketing, communication, customer feedback, and mass opinion measurement. However, many blogs often contain similar information and the sheer volume of available information really challenges the ability of organizations to act quickly in todays business environment. Thus, novelty mining can help to single out novel information out of a massive set of text documents. This paper explores the feasibility and performance of novelty mining and database optimization of business blogs, which have not been studied before. The results show that our novelty mining system can detect novelty in our dataset of business blogs with very high accuracy, and that database optimization can significantly improve the performance.

A Business Zone Recommender System Based on Facebook and Urban Planning Data

We present ZoneRec—a zone recommendation system for physical businesses in an urban city, which uses both public business data from Facebook and urban planning data. The system consists of machine learning algorithms that take in a business’ metadata and outputs a list of recommended zones to establish the business in. We evaluate our system using data of food businesses in Singapore and assess the contribution of different feature groups to the recommendation quality.

Multilingual novelty detection

Novelty detection aims at reducing redundant information from a chronologically ordered list of documents or sentences. Other studies of novelty detection have been conducted on the English language, but few papers have addressed the problem of multilingual novelty detection. Likewise, research in multilingual information retrieval have rarely been applied to novelty detection. This paper attempts to bridge the two disciplines by first describing the preprocessing steps for English, Malay and Chinese, then applying document and sentence-level novelty detection for the three languages on APWSJ and TREC 2004 Novelty Track data. Experiments on sentence-level novelty detection show similar results for all three languages, which indicates that our algorithm is suitable for multilingual novelty detection at the sentence level. However, results for document-level novelty detection show a disparity across the different languages, with English and Malay outperforming Chinese. After applying sentence-level novelty detection to detect novel documents, we observe substantial improvements on all three languages. This demonstrates that segmenting documents into sentences improves document-level novelty detection in multiple languages, and has practical benefits for a real-time multilingual novelty detection system.

LASER: a living analytics experimentation system for large-scale online controlled experiments

Tracking user browsing data and measuring the effectiveness of website design and web services are important to businesses that want to attract the consumers today who spend much more time online than before. Instead of using randomized controlled experiments, the existing approach simply tracks user browsing behaviors before and after a change is made to website design or web services, and evaluate the differences. To address the effects caused by hidden factors (e.g. promotion activities on the website) and to give fair comparison of different website designs, we propose the LASER system, a unified experimentation platform that enables randomized online controlled experiments to be easily conducted with minimal human effort and modifications to the experimented websites. More importantly, the LASER system manages the various aspects of online controlled experiments, namely the selection of participants into groups, exposure of different user interface features or recommendation algorithms to these groups, measuring their responses, and summarizing the results in the visual manner.