A neural network system for transformation of regional cuisine style
Masahiro Kazama, Minami Sugimoto, Chizuru Hosokawa, Keisuke Matsushima, Lav R. Varshney, Yoshiki Ishikawa
AA neural network system for transformation of regional cuisine style
Masahiro Kazama, Minami Sugimoto, Chizuru Hosokawa, Keisuke Matsushima, Lav R. Varshney, and Yoshiki Ishikawa ∗ Habitech Inc., Tokyo, Japan Department of Public Health, Cancer Scan Inc., Tokyo, Japan Keisuke Matsushima, Nice, France University of Illinois at Urbana-Champaign, Illinois, USA
Abstract
We propose a novel system which can transform a recipe into any selected regional style (e.g.,Japanese, Mediterranean, or Italian). This system has two characteristics. First the system can identifythe degree of regional cuisine style mixture of any selected recipe and visualize such regional cuisine stylemixtures using barycentric Newton diagrams. Second, the system can suggest ingredient substitutionsthrough an extended word2vec model, such that a recipe becomes more authentic for any selected re-gional cuisine style. Drawing on a large number of recipes from Yummly, an example shows how theproposed system can transform a traditional Japanese recipe, Sukiyaki, into French style.Keywords: food, big data, regional cuisine style, newton diagram, neural network, word2vec
With growing diversity in personal food preference and regional cuisine style, personalized informationsystems that can transform a recipe into any selected regional cuisine style that a user might prefer wouldhelp food companies and professional chefs create new recipes.To achieve this goal, there are two significant challenges: 1) identifying the degree of regional cuisinestyle mixture of any selected recipe; and 2) developing an algorithm that shifts a recipe into any selectedregional cuisine style.As to the former challenge, with growing globalization and economic development, it is becoming difficultto identify a recipes regional cuisine style with specific traditional styles since regional cuisine patterns havebeen changing and converging in many countries throughout Asia, Europe, and elsewhere [1]. Regardingthe latter challenge, to the best of our knowledge, little attention has been paid to developing algorithmswhich transform a recipes regional cuisine style into any selected regional cuisine pattern, cf. [2, 3]. Previousstudies have focused on developing an algorithm which suggests replaceable ingredients based on cookingaction [4], degree of similarity among ingredient [5], ingredient network [6], degree of typicality of ingredient[7], and flavor (foodpairing.com).The aim of this study is to propose a novel data-driven system for transformation of regional cuisine style.This system has two characteristics. First, we propose a new method for identifying a recipes regional cuisinestyle mixture by calculating the contribution of each ingredient to certain regional cuisine patterns, such asMediterranean, French, or Japanese, by drawing on ingredient prevalence data from large recipe repositories.Also the system visualizes a recipes regional cuisine style mixture in two-dimensional space under barycentriccoordinates using what we call a
Newton diagram . Second, the system transforms a recipes regional cuisinepattern into any selected regional style by recommending replaceable ingredients in existing recipes. ∗ [email protected] a r X i v : . [ c s . C Y ] J un igure 1: Architecture of transformation system which transform a given recipe into any selected coun-try/region styleAs an example of this proposed system, we transform a traditional Japanese recipe, Sukiyaki, into Frenchstyle. Figure 1 shows the overall architecture of the transformation system, which consists of two steps: 1) iden-tification and visualization of a recipes regional cuisine style mixture; and 2) algorithm which transforms agiven recipe into any selected regional/country style. Details of the steps are described as follows.
Using a neural network method as detailed below, we identify a recipe’s regional cuisine style. The neuralnetwork model was constructed as shown in Figure 2. The number of layers and dimension of each layer arealso shown in Figure 2.When we enter a recipe, this model classifies which country or regional cuisine the recipe belongs to. Theinput is a vector with the dimension of the total number of ingredients included in the dataset, and only theindices of ingredients contained in the input recipe are 1, otherwise they are 0.There are two hidden layers. Therefore, this model can consider a combination of ingredients to predictthe country probability. Dropout is also used for the hidden layer, randomly (20%) setting the value ofthe node to 0. So a robust network is constructed. The final layers dimension is the number of countries,here 20 countries. In the final layer, we convert it to a probability value using the softmax function, whichrepresents the probability that the recipe belongs to that country. ADAM [8] was used as an optimizationtechnique. The number of epochs in training was 200. These network structure and parameters were chosenafter preliminary experiments so that the neural network could perform the country classification task asefficiently as possible.In this study, we used a labeled corpus of Yummly recipes to train this neural network. Yummly datasethas 39774 recipes from the 20 countries as shown in Table 1. Each recipe has the ingredients and countryinformation. Firstly, we randomly divided the data set into 80% for training the neural network and 20%for testing how precisely it can classify. The final neural network achieved a classification accuracy of 79%2igure 2: Neural network model for predicting regional cuisine from list of ingredients.Table 1: Statistics of Yummly dataset and some recipe examples.Country Recipes IngredientsItalian 7838 2929Mexican 6438 2684Southern US 4320 2462Indian 3003 1664Chinese 2673 1792French 2646 2102Cajun Creole 1546 1576Thai 1539 1376Japanese 1423 1439Greek 1175 1198 Country Recipes IngredientsSpanish 989 1263Korean 830 898Vietnamese 825 1108Moroccan 821 974British 804 1166Filipino 755 947Irish 667 999Jamaican 526 877Russian 489 872Brazilian 467 853ALL 39774 6714RecipeID Country Ingredients34466 British greek yogurt, lemon curd, confectioners sugar, raspberries44500 Indian chili, mayonaise, chopped onion, cider vinegar, fresh mint, cilantro leaves38233 Thai sugar, chicken thighs, cooking oil, fish sauce, garlic, black pepper3able 2: Example of ingredient classification by the neural network. Three top countries are listed with theprobability that the ingredient are classified into.Ingredient Top1 Top2 Top3Onions French Italian Mexican0.130 0.126 0.126Soy sauce Japanese Chinese Filipino0.246 0.233 0.122Mirin Japanese French Korean0.890 0.040 0.020on the test set. Figure 3 shows the confusion matrix of the neural network classifficaiton. Table 2 shows theexamples of ingredient classification results. Common ingredients, onions for example, that appear in manyregional recipes are assigned to all countries with low probability. On the other hands some ingredients thatappear only in specific country are assigned to the country with high probability. For example mirin that isa seasoning commonly used in Japan is classified into Japan with high probability.Figure 3: Confusion matrix of neural network classiffication.By using the probability values that emerge from the activation function in the neural network, ratherthan just the final classification, we can draw a barycentric Newton diagram, as shown in Figure 4. Thebasic idea of the visualization, drawing on Isaac Newtons visualization of the color spectrum [9], is to expressa mixture in terms of its constituents as represented in barycentric coordinates. This visualization allowsan intuitive interpretation of which country a recipe belongs to. If the probability of Japanese is high, therecipe is mapped near the Japanese. The countries on the Newton diagram are placed by spectral graphdrawing [10], so that similar countries are placed nearby on the circle. The calculation is as follows. First4igure 4: Newton diagram: visualization of probability that the recipe belongs to the several regional cuisinestyle. Countries are placed by spectral drawing.we define the adjacency matrix W as the similarity between two countries. The similarity between country i and j is calculated by cosine similarity of county i vector and j vector. These vector are defined in nextsection. W ij = sim ( vec i , vec j ). The degree matrix D is a diagonal matrix where D ii = (cid:80) j W ij . Next wecalculate the eigendecomposition of D − W . The second and third smallest eingenvalues and correspondedeingevectors are used for placing the countries. Eigenvectors are normalized so as to place the countries onthe circle. If you want to change a given recipe into a recipe having high probability of a specific country by justchanging one ingredient, which ingredient should be alternatively used?When we change the one ingredient x i in the recipe to ingredient x j , the probability value of countrylikelihood can be calculated by using the above neural network model. If we want to change the recipe to havehigh probability of a specific country c , we can find ingredient x j that maximizes the following probability. P ( C = c | r − x i + x j ) where r is the recipe. However, with this method, regardless of the ingredient x i , onlyspecific ingredients having a high probability of country c are always selected. In this system, we want toselect ingredients that are similar to ingredient x i and have a high probability of country c . Therefore, wepropose a method of extending word2vec as a method of finding ingredients resembling ingredient x i .Word2vec is a technique proposed in the field of natural language processing [11]. As the name implies,it is a method to vectorize words, and similar words are represented by similar vectors. To train word2vec,skip-gram model is used. In the skip-gram model, the objective is to learn word vector representations thatcan predict the nearby words. The objective function is5igure 5: The word2vec (skip-gram) architecture. The left panel is the traditional word2vec with windowsize n = 2. The middle panel is the word2vec for recipe data. The right panel is the word2vec for recipedata with country information. (cid:88) d ∈ D (cid:88) w i ∈ d (cid:88) − n ≤ j ≤ n,j (cid:54) =0 log P ( w i + j | w i ) (1)where D is the set of documents, d is a document, w i is a word, and n is the window size. This modelpredicts the n words before and after the input word, as described in left side of Figure 5. The objectivefunction is to maximize the likelihood of the prediction of the surrounding word w i + j given the center word w i . The probability is P ( w j | w i ) = exp( v Tw i v (cid:48) w j ) (cid:80) w ∈ W exp( v Tw i v (cid:48) w ) (2)where v w ∈ R K is an input vector of word w , v (cid:48) w ∈ R K is an output vector of word w , K is the dimensionof the vector, and W is the set of all words. To optimize this objective function, hierarchical softmax ornegative sampling method [11] are used. After that we get the vectors of words and we can calculate analogiesby using the vectors. For example, the analogy of “King - Man + Women = ?” yields “Queen” by usingword2vec.In this study, word2vec is applied to the data set of recipes. Word2vec can be applied by consideringrecipes as documents and ingredients as words. We do not include a window size parameter, since it is usedto encode the ordering of words in document where it is relevant. In recipes, the listing of ingredients isunordered. The objective function is (cid:88) r ∈ R (cid:88) w i ∈ r (cid:88) j (cid:54) = i log P ( w j | w i ) (3)where R is a set of recipes, r is a recipe, and w i is the i th ingredient in recipe r . The architecture is describedin middle of Figure 5. The objective function is to maximize the likelihood of the prediction of the ingredient w j in the same recipe given the ingredient w i . The probability is defined below. P ( w j | w i ) = exp( v Tw i v (cid:48) w j ) (cid:80) w ∈ W exp( v Tw i v (cid:48) w ) (4)where w is an ingredient, v w ∈ R K is an input vector of ingredient, v (cid:48) w ∈ R K is an output vector of ingredient, K is the dimension of the vector, and W is the set of all ingredients.6ach ingredient is vectorized by word2vec, and the similarity of each ingredient is calculated using cosinesimilarity. Through vectorization in word2vec, those of the same genre are placed nearby. In other words,by using the word2vec vector, it is possible to select ingredients with similar genres.Next, we extend word2vec to be able to incorporate information of the country. When we vectorize thecountries, we can calculate the analogy between countries and ingredients. For example, this method can tellus what is the French ingredient that corresponds to Japanese soy sauce by calculating “Soy sauce - Japan+ French = ?”.The detail of our method is as follows. We maximize objective function (5). (cid:88) r ∈ R (cid:88) w i ∈ r log P ( w i | c r ) + log P ( c r | w i ) + (cid:88) j (cid:54) = i log P ( w j | w i ) (5)where R is a set of recipes, r is a recipe, w i is the i th ingredient in recipe r , and c r is the country recipe r belongs to. The architecture is described in right of Figure 5. The objective function is to maximize thelikelihood of the prediction of the ingredient w j in the same recipe given the ingredient w i along with theprediction of the the ingredients w i given the country c r and the prediction of the the country c r given theingredient w i . The probability is defined below. P ( b | a ) = exp( v Ta v (cid:48) b ) (cid:80) c ∈ W exp( v Ta v (cid:48) c ) (6)where a is a ingredient or country, b, c are also, v a ∈ R K is an input vector of ingredient or country, v (cid:48) a ∈ R K is an output vector of ingredient or country, K is the dimension of vector, and W is the set of all ingredientsand all countries.We can use hierarchical softmax or negative sampling [11] to maximize objective function (5) and findthe vectors of ingredients and countries in the same vector space.Table 3 shows the ingredients around each country in the vector space, and which could be considered asmost authentic for that regional cuisine [12]. Also, Figure 6 shows the ingredients and countries in 2D mapby using t-SNE method [13].Table 3: Authentic ingredients for each country. Top 5 ingredients around each country in the vector space.French Japanese Italian MexicanTop1 Cognac Mirin Grated parmesan cheese Corn tortillasTop2 Calvados Dashi pecorino romano cheese SalsaTop3 Thyme springs Nori prosciutto Tortilla chipsTop4 Gruyere cheese Wasabi paste marinara sauce GuacamoleTop5 Nicoise olives Bonito flakes Sweet italian sausage Poblano peppers Our substitution strategy is as follows. First we use extended word2vec and train it by Yummly dataset.After that all ingredients and countries are vectorized into 100 dimensional vector space. Second we findsubstitution by analogy calculation. For example, to find french substitution of Mirin, we calculate “Mirin -Japanese + French” in the vector space and get the vector as result. After that we find similar ingredientsaround the vector by calculating the cosine similarity.As an example of our proposed system, we transformed a traditional Japanese “Sukiyaki” into Frenchstyle. Table 4 shows the suggested replaceable ingredients and the probability after replacing. “Sukiyaki”consists of soy sauce, beef sirloin, white sugar, green onions, mirin, shiitake, egg, vegetable oil, konnyaku,7igure 6: Ingredients and countries map by extended word2vec: Each ingredient and country is mapped in2D by using t-SNE. Also Each ingredient is colored by using t-SNE to convert 100 dimension vector into 3dimension. The 3 dimension is corresponded to RGB color. Countries are represented by bold black.and chinese cabbage. Figure 7 shows the Sukiyaki in French style cooked by professional chef KM who isone of the authors of this paper. He assesses the new recipe as valid and novel to him in terms of Sukiyakiin French. Here our task is in generating a new dish, for which by definition there is no ground truth forcomparison. Rating by experts is the standard approach for assessing novel generative artifacts, e.g. instudies of creativity [14], but going forward it is important to develop other approaches for assessment.Figure 7: Sukiyaki in French style. Professional chef KM who is one of the authors of this paper cooked therecipe suggested by our system.
With growing diversity in personal food preference and regional cuisine style, the development of data-drivensystems which can transform recipes into any given regional cuisine style might be of value for food companies8able 4: Alternative ingredients suggested by extended word2vec model and country probability of changingfood ingredients in order from the top. Professional chef KM who is one of the authors of this paper choseone alternative ingredient from top 10 suggested ingredients each.Original Ingredient Alternative Ingredient P(Japanese) P(French)
Conflict of Interest Statement
The authors declare that they have no conflict of interest.
Author Contributions
MK, LRV, and YI had the idea for the study and drafted the manuscript. MK performed the data collectionand analysis. MS, CH, and KM participated in the interpretation of the results and discussions for manuscriptwriting and finalization. All authors read and approved the final manuscript.
Funding
Varshney’s work was supported in part by the IBM-Illinois Center for Cognitive Computing Systems Research(C3SR), a research collaboration as part of the IBM AI Horizons Network.
Acknowledgments
This study used data from Yummly. We would like to express our deepest gratitude to everyone whoparticipated in this services. We thank Kush Varshney for suggesting the spectral graph drawing approachto placing countries on the circle.
References [1] C. K. Khoury, A. D. Bjorkman, H. Dempewolf, J. Ramirez-Villegas, L. Guarino, A. Jarvis, L. H.Rieseberg, and P. C. Struik, “Increasing homogeneity in global food supplies and the implications forfood security,”
Proceedings of the National Academy of Sciences of the United States of America , vol.111, no. 11, pp. 4001–4006, Mar. 2014.[2] F. Pinel, N. Shao, and L. R. Varshney, “Using physicochemical correlates of perceptual flavor similarityto enhance, balance and substitute flavors,” Aug. 13 2014, US Patent App. 14/458,315.[3] F. Pinel and L. R. Varshney, “Substitution of work products,” May 5 2014, US Patent App. 14/269,353.[4] Y. Shidochi, T. Takahashi, I. Ide, and H. Murase, “Finding replaceable materials in cooking recipe textsconsidering characteristic cooking actions,” in
Proceedings of the ACM Multimedia 2009 Workshop onMultimedia for Cooking and Eating Activities , Oct. 2009, pp. 9–14.[5] K. Nozawa, Y. Nakaoka, S. Yamamoto, and T. Satoh, “Finding method of replaceable ingredients usinglarge amounts of cooking recipes,”
IEICE Technical Report , vol. 114, no. 204, pp. 41–46, Sep. 2014.106] C.-Y. Teng, Y.-R. Lin, and L. A. Adamic, “Recipe recommendation using ingredient networks,” in
Proceedings of the 3rd Annual ACM Web Science Conference (WebSci’12) , Jun. 2012, pp. 298–307.[7] S. Yokoi, K. Doman, T. Hirayama, I. Ide, D. Deguchi, and H. Murase, “Typicality analysis of thecombination of ingredients in a cooking recipe for assisting the arrangement of ingredients,” in , Jun. 2015.[8] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 ,2014.[9] I. Newton,
Opticks: Or, a Treatise of the Reflexions, Refractions, Inflexions and Colours of Light .London: Sam. Smith and Benj. Walford, 1704.[10] Y. Koren, “On spectral graph drawing,” in
International Computing and Combinatorics Conference .Springer, 2003, pp. 496–508.[11] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of wordsand phrases and their compositionality,” in
Advances in Neural Information Processing Systems 26 ,C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, Eds. Cambridge, MA:MIT Press, 2013, pp. 3111–3119.[12] Y.-Y. Ahn, S. E. Ahnert, J. P. Bagrow, and A.-L. Barab´asi, “Flavor network and the principles of foodpairing,”
Scientific Rerports , vol. 1, p. 196, Dec. 2011.[13] L. van der Maaten and G. Hinton, “Visualizing data using t-SNE,”
Journal of Machine Learning Re-search , vol. 9, pp. 2579–2605, Nov. 2008.[14] A. Jordanous, “A standardised procedure for evaluating creative systems: Computational creativityevaluation based on what it is to be creative,”
Cognitive Computation , vol. 4, no. 3, pp. 246–279, 2012.[15] D. B. Panagiotakos, C. Pitsavos, and C. Stefanadis, “Dietary patterns: a Mediterranean diet score andits relation to clinical and biological markers of cardiovascular disease risk,”
Nutrition, Metabolism andCardiovascular Diseases , vol. 16, no. 8, pp. 559–568, Dec. 2006.[16] J. R. Kettenring, “Canonical analysis of several sets of variables,”
Biometrika , vol. 58, no. 3, pp. 433–451,Dec. 1971.[17] L. Serra-Majem, A. Trichopoulou, J. N. de la Cruz, P. Cervera, A. G. ´Alvarez, C. La Vecchia, A. Lem-touni, and D. Trichopoulos, “Does the definition of the Mediterranean diet need to be updated?”
PublicHealth Nutrition , vol. 7, no. 7, pp. 927–929, Oct. 2004.[18] O. Kinouchi, R. W. Diez-Garcia, A. J. Holanda, P. Zambianchi, and A. C. Roque, “The non-equilibriumnature of culinary evolution,”
New Journal of Physics , vol. 10, no. 7, p. 073020, 2008.[19] K. R. Varshney, L. R. Varshney, J. Wang, and D. Myers, “Flavor pairing in Medieval European cuisine:A study in cooking with dirty data,”
Int. Joint Conf. Artif. Intell. Workshops , pp. 3–12, Aug. 2013.[20] Y.-X. Zhu, J. Huang, Z.-K. Zhang, Q.-M. Zhang, T. Zhou, and Y.-Y. Ahn, “Geography and similarityof regional cuisines in China,”
PloS ONE , vol. 8, no. 11, p. e79161, Nov. 2013.[21] A. Jain, Rakhi N. K., and G. Bagler, “Analysis of food pairing in regional cuisines of India,”
PLoSONE , vol. 10, no. 10, p. e0139539.[22] S. T. Tallab and M. S. Alrazgan, “Exploring the food pairing hypothesis in Arab cuisine: A study incomputational gastronomy,”