Matej Črepinšek

Comparing general-purpose and domain-specific languages: An empirical study

Many domain-specific languages, that try to bring feasible alternatives for existing solutions while simplifying programming work, have come up in recent years. Although, these little languages seem to be easy to use, there is an open issue whether they bring advantages in comparison to the application libraries, which are the most commonly used implementation approach. In this work, we present an experiment, which was carried out to compare such a domain-specific language with a comparable application library. The experiment was conducted with 36 programmers, who have answered a questionnaire on both implementation approaches. The questionnaire is more than 100 pages long. For a domain-specific language and the application library, the same problem domain has been used - construction of graphical user interfaces. In terms of a domain-specific language, XAML has been used and C# Forms for the application library. A cognitive dimension framework has been used for a comparison between XAML and C# Forms.

Replication and comparison of computational experiments in applied evolutionary computing: Common pitfalls and guidelines to avoid them

Abstract Replicating and comparing computational experiments in applied evolutionary computing may sound like a trivial task. Unfortunately, it is not so. Namely, many papers do not document experimental settings in sufficient detail, and hence replication of experiments is almost impossible. Additionally, some work fails to satisfy the thumb rules for Experimentation throughout all disciplines, such that all experiments should be conducted and compared under the same or stricter conditions. Also, because of the stochastic properties inherent in evolutionary algorithms (EAs), experimental results should always be rich enough with respect to Statistics. Moreover, the comparisons conducted should be based on suitable performance measures and show the statistical significance of one approach over others. Otherwise, the derived conclusions may fail to have scientific merits. The primary objective of this paper is to offer some preliminary guidelines and reminders for assisting researchers to conduct any replications and comparisons of computational experiments when solving practical problems, by the use of EAs in the future. The common pitfalls are explained, that solve economic load dispatch problems using EAs from concrete examples found in some papers.

A chess rating system for evolutionary algorithms: A new method for the comparison and ranking of evolutionary algorithms

Abstract The Null Hypothesis Significance Testing (NHST) is of utmost importance for comparing evolutionary algorithms as the performance of one algorithm over another can be scientifically proven. However, NHST is often misused, improperly applied and misinterpreted. In order to avoid the pitfalls of NHST usage this paper proposes a new method, a Chess Rating System for Evolutionary Algorithms (CRS4EAs) for the comparison and ranking of evolutionary algorithms. A computational experiment in CRS4EAs is conducted in the form of a tournament where the evolutionary algorithms are treated as chess players and a comparison between the solutions of two algorithms on the objective function is treated as one game outcome. The rating system used in CRS4EAs was inspired by the Glicko-2 rating system, based on the Bradley–Terry model for dynamic pairwise comparisons, where each algorithm is represented by rating, rating deviation, a rating/confidence interval, and rating volatility. The CRS4EAs was empirically compared to NHST within a computational experiment conducted on 16 evolutionary algorithms and a benchmark suite of 20 numerical minimisation problems. The analysis of the results shows that the CRS4EAs is comparable with NHST but may also have many additional benefits. The computations in CRS4EAs are less complicated and sensitive than those in statistical significance tests, the method is less sensitive to outliers, reliable ratings can be obtained over a small number of runs, and the conservativity/liberality of CRS4EAs is easier to control.

Is a comparison of results meaningful from the inexact replications of computational experiments

The main objective of this paper is to correct the unreasonable and inaccurate criticism to our previous experiments using Teaching–Learning-Based Optimization algorithm and to quantify the amount of error that may arise due to incorrect counting of fitness evaluations. It is shown that inexact experiment replication should be avoided in comparisons between meta-heuristic algorithms whenever possible. Otherwise, an inexact replication and margin of error should be explicitly reported.

Analysis of exploration and exploitation in evolutionary algorithms by ancestry trees

This paper introduces an ancestry tree-based approach for exploration and exploitation analysis. The approach introduces a data structure to record the evolution history of a population and a number of exploration and exploitation metrics. Such an approach not only provides insight of how and when the exploration and exploitation influence an evolution process, but also how the genetic structure of an individual is affected. It can be used to better understand inner working of an evolutionary algorithm or in evolutionary algorithm designing phase to develop suitable variation operators with good balance between exploration and exploitation. The approach is applied to the multi-objective 0/1 knapsack problem.

Ontology driven development of domain-specific languages

Domain-specific languages (DSLs) are computer (programming, modeling, specification) languages devoted to solving problems in a specific domain. The development of a DSL includes the following phases: decision, analysis, design, implementation, testing, deployment, and maintenance. The least-known and least examined are analysis and design. Although various formal methodologies exist, domain analysis is still done informally most of the time. A common reason why formal methodologies are not used as often as they could be is that they are very demanding. Instead of developing a new, less complex methodology, we propose that domain analysis could be replaced with a previously existing analysis in another form. A particularly suitable form is the use of ontologies. This paper focuses on ontology-based domain analysis and how it can be incorporated into the DSL design phase. We will present the preliminary results of the Ontology2DSL framework, which can be used to help transform ontology to a DSL grammar incorporating concepts from a domain.

Extracting grammar from programs: evolutionary approach

The paper discusses context-free grammar (CFG) inference using genetic-programming with application to inducing grammars from programs written in simple domain-specific languages. Grammar-specific heuristic operators and non-random construction of the initial population are proposed to achieve this task. Suitability of the approach is shown by small examples where the underlying CFGs are successfully inferred.

Context-free grammar induction using genetic programming

While grammar inference is used in areas like natural language acquisition, syntactic pattern recognition, etc., its application to the programming language problem domain has been limited. We propose a new application area for grammar induction which intends to make domain-specific language development easier and finds a second application in renovation tools for legacy systems. The genetic programming approach is used for grammatical inference. Our earlier work used grammar-specific heuristic operators in tandem with non-random construction of the initial grammar population and succeeded in inducing small grammars.

Extracting grammar from programs: brute force approach

Extracting grammar from programs attracts researchers from several fields such as software engineering, pattern recognition, computational linguistic and natural language acquisition. So far, only regular grammar induction has been successful, while purposeful context-free grammar induction is still elusive. We discuss the search space of context-free grammar induction and propose the Brute Force approach to the problem which, along with its various enhancements, can be further used in collaboration with the Evolutionary approach, as described in the accompanying paper.

On automata and language based grammar metrics

Grammar metrics have been introduced to measure the quality and the complexity of the formal grammars. The aim of this paper is to explore the meaning of these notions and to experiment, on several grammars of domain specific languages and of general-purpose languages, existing grammar metrics together with the new metrics that are based on grammar LR automaton and on the language recognized. We discuss the results of this experiment and focus on the comparison between grammars of domain specific languages as well as of general-purpose languages and on the evolution of the metrics between several versions of the same language.