Alex Burnap

When Crowdsourcing Fails: A Study of Expertise on Crowdsourced Design Evaluation

Crowdsourced evaluation is a promising method of evaluating engineering design attributes that require human input. The challenge is to correctly estimate scores using a massive and diverse crowd, particularly when only a small subset of evaluators has the expertise to give correct evaluations. Since averaging evaluations across all evaluators will result in an inaccurate crowd evaluation, this paper benchmarks a crowd consensus model that aims to identify experts such that their evaluations may be given more weight. Simulation results indicate this crowd consensus model outperforms averaging when it correctly identifies experts in the crowd, under the assumption that only experts have consistent evaluations. However, empirical results from a real human crowd indicate this assumption may not hold even on a simple engineering design evaluation task, as clusters of consistently wrong evaluators are shown to exist along with the cluster of experts. This suggests that both averaging evaluations and a crowd consensus model that relies only on evaluations may not be adequate for engineering design tasks, accordingly calling for further research into methods of finding experts within the crowd. [DOI: 10.1115/1.4029065]

A SIMULATION BASED ESTIMATION OF CROWD ABILITY AND ITS INFLUENCE ON CROWDSOURCED EVALUATION OF DESIGN CONCEPTS

Crowdsourced evaluation is a promising method for evaluating attributes of design concepts that require human input. One factor in obtaining good evaluations is the ratio of high-ability to low-ability participants within the crowd. In this paper we introduce a Bayesian network model capable of finding participants with high design evaluation ability, so that their evaluations may be weighted more than those of the rest of the crowd. The Bayesian network model also estimates a score of how well each design concept performs with respect to a design attribute without knowledge of the true scores. Monte Carlo simulation studies tested the quality of the estimations on a variety of crowds consisting of participants with different evaluation ability. Results suggest that the Bayesian network model estimates design attribute performance scores much closer to their true values than simply weighting the evaluations from all participants in the crowd equally. This finding holds true even when the group of high ability participants is a small percentage of the entire crowd.Copyright

Improving Design Preference Prediction Accuracy Using Feature Learning

Quantitative preference models are used to predict customer choices among design alternatives by collecting prior purchase data or survey answers. This paper examines how to improve the prediction accuracy of such models without collecting more data or changing the model. We propose to use features as an intermediary between the original customer-linked design variables and the preference model, transforming the original variables into a feature representation that captures the underlying design preference task more effectively. We apply this idea to automobile purchase decisions using three feature learning methods (principal component analysis (PCA), low rank and sparse matrix decomposition (LSD), and exponential sparse restricted Boltzmann machine (RBM)) and show that the use of features offers improvement in prediction accuracy using over 1 million real passenger vehicle purchase data. We then show that the interpretation and visualization of these feature representations may be used to help augment data-driven design decisions. [DOI: 10.1115/1.4033427]

IMPROVING PREFERENCE PREDICTION ACCURACY WITH FEATURE LEARNING

Motivated by continued interest within the design community to model design preferences, this paper investigates the question of predicting preferences with particular application to consumer purchase behavior: How can we obtain high prediction accuracy in a consumer preference model using market purchase data? To this end, we employ sparse coding and sparse restricted Boltzmann machines, recent methods from machine learning, to transform the original market data into a sparse and high-dimensional representation. We show that these ‘feature learning’ techniques, which are independent from the preference model itself (e.g., logit model), can complement existing efforts towards high-accuracy preference prediction. Using actual passenger car market data, we achieve significant improvement in prediction accuracy on a binary preference task by properly transforming the original consumer variables and passenger car variables to a sparse and high-dimensional representation.

Balancing Design Freedom and Brand Recognition in the Evolution of Automotive Brand Styling

Designers faced with the task of developing the next model of a brand must balance several considerations. The design must be novel and express attributes important to the customers, while also recognizable as a representative of the brand. This balancing is left to the intuition of the designers, who must anticipate how all customers will perceive the new design. Oftentimes, the design freedom used to meet a styling attribute such as aggressiveness can compromise the recognition of the product as a member of the brand. In this paper, an experiment is conducted measuring change in ten styling attributes common to both design freedom and brand recognition for automotive designs, using customer responses to vehicle designs created interactively. Results show that, while brand recognition is highly dependent on the particular manufacturer, tradeoffs between design freedom and brand recognition may be measured using predictive models to inform strategic design decisions.Copyright