Patrick Hennig

Mapping the Blogosphere with RSS-Feeds

The massive adoption of social media has provided new ways for individuals to express their opinions online. The blogosphere, an inherent part of this trend, contains a vast array of information about a variety of topics. It is thus a huge think tank that creates an enormous and ever-changing archive of open source intelligence. Modeling and mining this vast pool of data to extract, exploit and describe meaningful knowledge in order to leverage (content-related) structures and dynamics of emerging networks within the blogosphere is the higher-level aim of the research presented here. This paper focuses on this project’s initial phase, in which the above-mentioned data of interest needs to be collected and made available offline for further analyses. Our proprietary development of a tailor-made feed-crawler meets exactly this need. The main concept, the techniques and the implementation details of the crawler thus form the main interest of this paper and furthermore provide the basis for future project phases.

Mapping the Blogosphere--Towards a Universal and Scalable Blog-Crawler

The massive adoption of social media has provided new ways for individuals to express their opinions online. The blogosphere, an inherent part of this trend, contains a vast array of information about a variety of topics. Thus, it is a huge think tank that creates an enormous and ever-changing archive of open source intelligence. Modeling and mining this vast pool of data to extract and describe meaningful knowledge in order to leverage (content-related) structures and dynamics of emerging networks within the blogo sphere is the higher-level aim of the research presented here. While the concept of our tailor-mode feed-crawler was already discussed in two earlier publications this paper focuses on our approach to extend the earlier feed crawler to a more universal and highly scalable blog-crawler.

Identify Emergent Trends Based on the Blogosphere

Information about upcoming trends is a valuable knowledge for both, companies and individuals. Detecting trends for a certain topic is of special interest. According to the latest information over 200 million blogs exist in the World Wide Web. Hence, every day millions of posts are published. These blogs contain an enormous think tank of open-source intelligence. Considering the continuously growing nature of the World Wide Web a primary factor of success is the ability to include the latest data and focus on the complete data set of blogs. The structured as well as unstructured data of blogs are available offline via a single database for further analyses. This paper describes and evaluates an algorithm to detect trends based on the data published in blog posts.

Blogosphere and its Exploration

This book represents an attempt to fully review the phenomenon of the blogosphere. The intention is to provide a reliable guide to understanding and analyzing the world of the unimaginable number of diverse blogs, each consisting of innumerable posts, which in their entirety form the blogosphere. We go on to answer the questions of how to grasp the complexity of the blogosphere and extract useful knowledge from it. In setting out to write this book, our central aim was to increase the readers awareness and understanding of the blogosphere phenomenon, including its structure and characteristics. This can be achieved through a better understanding of individual blogs and their particular technical characteristics, as well as a deeper knowledge of how a single blog is embedded and interconnected within the entire blogosphere. The shape and form of the blogosphere can be described using the analogy of different continents. In our description the defining features and characteristics of the continents are illustrated by paradigmatic example blogs. Following on from the structural analysis we provide details of the available methods and describe the complex challenge of automatically retrieving information from the abundance of data contained in the blogosphere. Finally, we present our blog search platform, called BLOGINTELLIGENCE and describe all the tools and features we have developed during the last couple of years to explore the blogosphere.

RSS-Crawler Enhancement for Blogosphere-Mapping

The massive adoption of social media has provided new ways for individuals to express their opinions online. The blogosphere, an inherent part of this trend, contains a vast array of information about a variety of topics. It is a huge think tank that creates an enormous and ever-changing archive of open source intelligence. Mining and modeling this vast pool of data to extract, exploit and describe meaningful knowledge in order to leverage structures and dynamics of emerging networks within the blogosphere is the higher-level aim of the research presented here. Our proprieteary development of a tailor-made feed-crawler-framework meets exactly this need. While the main concept, as well as the basic techniques and implementation details of the crawler have already been dealt with in earlier publications, this paper focuses on several recent optimization efforts made on the crawler framework that proved to be crucial for the performance of the overall framework.

Blog, Forum or Newspaper? Web Genre Detection Using SVMs

In recent years, blogs have become a very popular way to publish information, express opinions and hold discussions. Hence researchers and industry have interest in analyzing the blogosphere. Due to the increasing diversity of blog usage, the initial categorization into web genres is the first necessary step before any analyses. In this research, we focus on the distinction between traditional blogs, news portals, forums and miscellaneous websites. Especially the new distinction between news portals and blogs allows analyses to adapt to the network-specific characteristics of traditional media with high journalistic effort and more personal weblogs and their authors. We present a set of 80 features and extensively experiment with possible combinations and SVM parameters to identify the best constellation for the categorization into the four different web genres. Our experiments show a maximal accuracy of 83.5% overall. This high precision was reached using a combination of trained n-grams, structural properties (e.g. Twitter links) and quantitative properties like the texts length and number of dates.

Hot spot detection — An interactive cluster heat map for sentiment analysis

The blogosphere allows analysts to track opinions and sentiments of individuals, groups or the general public with large sample sizes regarding many topics. Essential for the sentiment analysis are visualizations. The visual understanding of large corporas sentiment is far more effective than relying on textual representations of the analyzed content. Users are very interested in changes in the public opinion. Thus, the identification of patterns is of high interest. In this paper, we propose a cluster heat map visualization for sentiment visualization that displays the sentiment development of various related terms over time intervals. As we want to encourage the discovery of patterns over multiple related topics, we apply an ordering algorithm based on dimensionality reduction to the cluster heat map and improve upon the ordering algorithm to enable fast pattern recognition.

Efficient Event Detection for the Blogosphere

In this paper we come up with a novel approach for the early detection of events in blog entries. The detection of trend is already discussed pretty often. Nevertheless, in our understanding the detection of events goes one step further. The presented algorithms detects unique happenings at a given point in time by perceiving unusual frequent occurrences of words or word groups. We introduce an implementation of our algorithm, making use of the SAP HANA database in order to achieve high performance and the ability to answer live queries for events.

Identifying Domain Experts in the Blogosphere -- Ranking Blogs Based on Topic Consistency

Current ranking algorithms, such as Page Rank, Technorati authority, and BI-Impact, favor blogs that report on a diversity of topics since those attract a large audience and thus more visitors, links, and comments. On the other side, niche blogs with a very specific topic only attract a small audience and thus have only a small reach. This results in a low ranking from todays blog retrieval systems. We argue that the consistency of a blog, i.e. how focused an author reports on a single topic, is a sign for expert knowledge. To find these blogs is particular important for other domain experts to identify blogs that they would like to follow and stay in active contact. To ease the retrieval of expert blogs, i.e. to separate them from the mass of blogs that report on random topics, we introduce a metric for blogs based on topic consistency. We divide the consistency ranking in four different aspects: (1) intra-post, (2) inter-post, (3) intra-blog, and (4) inter-blog consistency. By evaluating the metric with a test data set of 12,000 crawled blogs, we demonstrate the plausibility of our approach.

Identifying Audience Attributes: Predicting Age, Gender and Personality for Enhanced Article Writing

In order to create an effective article, having great content is essential. However, to achieve this, the writer needs to target a specific audience. A target audience refers to a group of readers that a writer intends to reach with his content. Defining a target audience is substantial because it has a direct effect on adjusting writing style and content of the article. Nowadays, writers rely solely on annotated attributes of articles, such as location and language to understand his/her audience. The aim of this work is to identify the audience attributes of articles, especially not-annotated attributes. Among others, this work focuses on the detection of three key audience attributes of related articles: age, gender, and personality.We compare between multiple machine learning classifiers to detect these attributes. Finally, we demonstrate a prototypical application that enables writers to run existing algorithms such as trend detection and showing related articles that are specific to a defined target audience based on the newly detected attributes.