PPersonal Food Model
Ali Rostami
University of California IrvineIrvine, United States of [email protected]
Vaibhav Pandey
University of California IrvineIrvine, United States of [email protected]
Nitish Nag
University of California IrvineIrvine, United States of [email protected]
Vesper Wang
University of California IrvineIrvine, United States of [email protected]
Ramesh Jain
University of California IrvineIrvine, United States of [email protected]
ABSTRACT
Food is central to life. Food provides us with energy andfoundational building blocks for our body and is also a majorsource of joy and new experiences. A significant part of theoverall economy is related to food. Food science, distribution,processing, and consumption have been addressed by differ-ent communities using silos of computational approaches[29]. In this paper, we adopt a person-centric multimediaand multimodal perspective on food computing and showhow multimedia and food computing are synergistic andcomplementary.Enjoying food is a truly multimedia experience involvingsight, taste, smell, and even sound, that can be captured usinga multimedia food logger. The biological response to food canbe captured using multimodal data streams using availablewearable devices. Central to this approach is the PersonalFood Model. Personal Food Model is the digitized representa-tion of the food-related characteristics of an individual. It isdesigned to be used in food recommendation systems to pro-vide eating-related recommendations that improve the user’squality of life. To model the food-related characteristics ofeach person, it is essential to capture their food-related en-joyment using a Preferential Personal Food Model and theirbiological response to food using their Biological PersonalFood Model. Inspired by the power of 3-dimensional colormodels for visual processing, we introduce a 6-dimensionaltaste-space for capturing culinary characteristics as well aspersonal preferences. We use event mining approaches to
Permission to make digital or hard copies of part or all of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contactthe owner/author(s).
MM ’20, October 12–16, 2020, Seattle, WA, USA © 2020 Copyright held by the owner/author(s).ACM ISBN 978-1-4503-7988-5/20/10.https://doi.org/10.1145/3394171.3414691 relate food with other life and biological events to build a pre-dictive model that could also be used effectively in emergingfood recommendation systems.
CCS CONCEPTS • Applied computing → Life and medical sciences ; •
Human-centered computing → Ubiquitous and mobilecomputing ; •
Computing methodologies → Modeling andsimulation . KEYWORDS
Food Computing, Personal Food Model, Food Recommenda-tion Systems, Taste Space, Event Mining, Personicle
ACM Reference Format:
Ali Rostami, Vaibhav Pandey, Nitish Nag, Vesper Wang, and RameshJain. 2020. Personal Food Model. In
Proceedings of the 28th ACMInternational Conference on Multimedia (MM ’20), October 12–16,2020, Seattle, WA, USA.
ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3394171.3414691 "One cannot think well, love well, sleep well, if one has notdined well." - Virginia WoolfFood is a significant determinant of human quality of life.Food provides the energy and nutrients essential for healthand is a significant source of personal enjoyment and socialfabric. In many instances, pleasures of eating conflict withthe optimal nutritional needs of the person’s physiologicalwell-being, and is the leading cause of the substantial in-crease in diet-related diseases such as obesity, diabetes, andhypertension [1], [50]. An important question is: why dopeople enjoy food [36]? People working on improving theenjoyment aspect of food, particularly chefs and food indus-try, have primarily ignored the health, and those focusedon health (doctors and nutritionists) usually consider theenjoyment aspect secondary [27], [18]. This disconnect inthe two approaches has led to the current situation with thewidespread increase in food-related illnesses. An importantfact is: what I like to eat is not necessarily what my bodylikes [26], [18]. Can we satisfy both me and my body? a r X i v : . [ c s . MM ] A ug M ’20, October 12–16, 2020, Seattle, WA, USA Rostami, et al.
Food and nutrition have their roots in multimedia andmultimodal elements [51]. Food experience requires the par-ticipation of audio, visual, tactile, gustatory, and olfactorysenses, and prior experiences play a crucial role [49]. Thefood we perceptually enjoy is a complete multimedia ex-perience [27], [51], which extends further to an extensivemultimodal effect in the body, impacting the physiology andbiochemistry of the individual. A multitude of sensors canmeasure the relationship between foods and the individualthrough the dynamic health state variables [34]. These in-clude readily available sensors that provide continuous datacollection for blood glucose, heart rate, perspiration rate, andbody temperature [19].Food is a multimodal experience that enriches personal lifeand enhances social rituals important to humans. However,we have not studied all aspects of food in a unified compu-tational framework like many other aspects of life, such associal networks, sports, and entertainment. Recently Minet al. [29] put together a computational framework arounddifferent silos of food. They adopt a diverse data-centricperspective and define food computing as, computational ap-proaches for acquiring and analyzing heterogeneous food datafrom disparate sources for perception, recognition, retrieval, rec-ommendation, and monitoring of food to address food-relatedissues in health, biology, gastronomy, and agronomy.
This ex-haustive and inclusive approach to food computing will helpunderstand different aspects of the food ecosystem and howthey impact each other.This paper looks at the food ecosystem from a person-centered perspective. Our goal is to study how food affects aperson’s life and how the food ecosystem may be affectedby choices made by people, as shown in Figure 1.Food serves two crucial but closely related functions ofmaintaining biological health state and personal enjoymentin life. Food items, listed in the dish-centric layer, meet thepersonal food needs of individuals. A group of food producersand distributors are part of the next layer that we show as thefood chain. Finally, each item produced, distributed, served,and consumed has a specific effect on the environment.In this paper, we present a computational framework forbuilding a Personal Food Model (PFM) that is essential tohelp people identify the right food, at the right place, inthe right situation, at the right price. PFM is an essentialcomponent of emerging food recommendation systems toaddress challenges in different aspects of businesses as wellas individuals’ health [28]. We consider the aspects of foodthat satisfy the two crucial needs of a person: enjoyment andsustenance. Different groups of people have studied thesetwo aspects. We believe that there is an excellent opportunityto bring these disjoint areas together using a computationalframework centered around multimedia. The most important contribution of this paper is the uni-fied personal model of culinary multimedia experience andbiological health aspects. We use this model in a complexrecommendation system that considers food items as a com-bination of features contributing to both enjoyment andsustenance and optimizes specific health outcomes such assleep quality. We model a person by analyzing their multi-modal food experiences as well as complex contextual factorsrelated to different food items and dishes. The recommen-dation system then tries to optimize factors related to bothenjoyment and sustenance by selecting correct food dishesin a given context.
Figure 1: Personal Food Computing Overview
We present the personal food model (PFM), as a critical, rel-evant, and timely challenge for multimedia and multimodalresearch. We present these ideas by(1) Reviewing existing work in multimedia that peripher-ally touched food computing but did not address realchallenges. We believe this was due to the absence ofa clear challenge and application. We show that char-acteristics of food items and the food preferences of aperson can be understood by combining visual, olfac-tory, culinary, and tactile (texture) aspects of food andeating environment.(2) Discussing essential aspects of personal food comput-ing that will benefit significantly from multimedia tech-nology and offer new challenges for the multimediacommunity. Notably, we discuss a multimodal foodlogging platform for building PFM and using it in anovel food recommendation platform. This may open aprominent application area for multimedia computing. ersonal Food Model MM ’20, October 12–16, 2020, Seattle, WA, USA
Figure 2: Food Recommendation Architecture: Data from the 3 digestion phases are being collected alongside other data-streams to create the PFM: the heart of Food Recommendation. (3) Presenting early components of personal food modelbased on multimedia computing, but require signifi-cant new research to create applications that may rivalany past multimedia applications.As discussed in subsequent sections, these are primarilymultimedia challenges that will open new paradigms in mul-timedia computing and communications and will help peopleenjoy good food and be healthy.
PFM is the digitized representation of the food-related char-acteristics of an individual. It can be used in food recommen-dation systems to provide eating-related recommendationsthat improve the user’s quality of life. Many factors affect andlimit a simple eating decision. However, this problem has notbeen modeled in a comprehensive framework to study foodas a multimedia experience, including taste, visual, social,and experiential factors. We show how PFM can predict theuser’s multimodal food preferences in different contexts. Weaccomplish this using different data streams captured fromthe user, such as location history [33], vital sign streams, andfood intake logged using text voice and photos. In futureworks, we plan to expand the sources of information we useto create the personal model and focus on using many otherdata streams such as the user’s calendar, social media, andtransaction history.PFM encompasses a complex nature as it contains manydimensions. The Biological part captures how Food itemscan satisfy nutritional needs for certain goals such as weightloss or improved performance in athletics [31]. Furthermore,Contextual understanding of the user needs must be layeredfor best computing real-time needs [32]. Other biologicaland life events may also impact the food events indirectlyand needs to be added to the model [40].Figure 2 shows how the personicle collects different datastreams over a long period [16]. Events from the personicle are fed to the PFM which consists of two parts. We define theBiological PFM of the user to capture the body’s reactionsto different food items including allergic reactions and nu-tritional needs. The biological model is an important factorin each food decision we make, but it is not the only factor.We also create the user’s taste profile, which constitutes thePreferential Personal Food model for the user. User’s tasteprofile contains the information about the food items whichthe user has experienced in the past, and it may also revealdishes that the user has never tried.
Figure 3: The interactions in the biological food model
Biological Personal Food Model
The Biological Personal Food Model (B-PFM) must considerhow food is related to the health state of the individual [30].This model should also extend to how the user may want tochange their health state towards a specific goal [31]. TheB-PFM focuses on the user’s dynamic health and nutritionalneeds [32], [35].Building the Biological PFM in a purely data-driven man-ner is a daunting task. Even though some apps like MyFitness-Pal collect food intake and activity data from the user, theyonly focus on a limited fitness aspect and cannot be extendedto a general biological model. However, instead of findingthe patterns solely based on user data, we propose a hybridapproach using patterns obtained from domain knowledgeto form a rule-based population model. We personalize this
M ’20, October 12–16, 2020, Seattle, WA, USA Rostami, et al. model as we collect more data. These rules capture the im-pact of food on biological parameters. For example, researchshows that eating heavy meals before bedtime could lowerthe quality of sleep. We collect a selection of such sequencesfrom expert domain knowledge and calculate the probabilityof validity for each of these patterns in different contexts.This set of context-driven rules form the B-PFM.We also need to understand how food items and foodevents impact different aspects of the health state of the indi-vidual [30], [35]. The user could have multiple health goalsthat might lead to conflicting recommendations (eg. diet forweight loss and sleep improvement). Therefore, it is impor-tant to keep the balance between different biological goalswhile also including static personal factors such as allergies,intolerances, and genetic factors in this computation. Figure3 shows how the current nutritional state is impacted bythe food intake based on the particular needs of the user atthe current biological state. In the event mining section, wedescribe how we turn the expert knowledge into active rulesand validate them based on the user’s data to predict thefuture biological health state.
Figure 4: Visualization of the US4B Taste Space. Part A: Thecollection of all taste samples from all users determine thehypervolume of the taste range of food items. Part B: Pasttaste sample values and ratings from the user determineuser’s preferred taste region within the food item taste rangehypervolume.
Preferential Personal Food Model
We propose a novel approach to quantify and describe hu-man taste perception to create the Preferential Personal FoodModel (P-PFM). Based on the current state of the art methodsfood recommendations either ignore the preference modelcompletely and just focus on healthy recommendations [46],or try to find the preferred ingredients of the user by askingthe user to rate a long list of ingredients and dishes withoutreally understanding why the user likes an ingredient [12].We introduce a taste space using six taste primaries calledthe US4B taste model. The US4B taste model is a multidimen-sional additive taste model, in which umami, salty, sweet, spicy, sour, and bitter taste (USSSSB) are added together invarious ways to reproduce a broad array of tastes. The RGBcolor space has been the foundation of many advances inmultimedia technology such as digital displays, virtual re-ality and 3D printing. The US4B taste space can be the keyto future food-related technologies that were not possiblewithout this foundation.Each food item will have an exact value in the US4B chan-nels which determine its taste. As Figure 4 shows, we createa Hyperdimensional Taste Space (HD-Taste-Space) and cal-culate a region for each food item. An unripe mango fromBrazil is going to have a different vector value in the HD-Taste-Space compared to a ripe mango from India whereaszucchini and cucumber samples share the same taste region.Therefore by sampling different instances we associate a hy-pervolume in the US4B Taste Space to each food item whichis shown in Figure 4 part A. Then we use the recipe databasesto estimate the hypervolume containing the possible tastesfor the dish in the hyperdimensional space. The state of theart finds correlations among recipes based on their ingredi-ents [20] but there has been no research to really understandthe taste of the dish based on the recipe as a multidimen-sional media. To create the P-PFM we map the food log tothe HD-Taste-Space to compute a hypervolume representingthe user’s preferred taste regions. The user’s preferred tasteregions in the US4B taste space is the most important partof the P-PFM. It contains the information about the foodthat the user likes and has experienced before, and can alsopredict the food that the user has never tried but might likebecause it lies within the user’s region of interest. Know-ing the preferred regions, we can search for healthier fooditems within the user’s preferred range of taste. Diet soda isa classic example of this concept. It has similar taste, texture,smell and visual cues compared to a normal soda but it hasdifferent effects on the biology. Having the food items inthe US4B taste space and finding the user’s preferred tasteregions in this space enables us to come up with better foodoptions tailored for each individual’s specific needs and tastepreference.
Models are built using data. Most successful search engines,social media, and e-commerce systems utilize personal mod-els to provide people with the right information, at the righttime, in the right context, usually even before a user articu-lates his need [4]. Personal food model plays the same rolein food recommendation systems [28]. We need to log foodconsumed by a person and all the relevant metadata over along period for the user. While initial food logging effortsrequired cumbersome manual food diaries, smartphones and ersonal Food Model MM ’20, October 12–16, 2020, Seattle, WA, USA cameras can drastically improve the quality and ease of log-ging. Aizawa [2] was the first multimedia researcher to cham-pion the idea of logging food using a smartphone camera andremains a very active researcher. Applications for camera-based food-logging have been developed in many other coun-tries [9], [8]. Multimedia and computer vision research com-munities have been actively exploring food-logging systems.These systems use computer vision techniques to recognizeitems, their ingredients, and even the volume consumed bythe user [9], [38]. Unfortunately, there is no generalized log-ger for international food, and identifying ingredients andvolume remains a challenge. A useful review of many visualapproaches and descriptions of databases used for trainingis included in [29].Conversational voice interfaces are becoming quite popu-lar, making rapid progress. Systems like Alexa and Siri areavailable at home, in phones, and watches. People can re-port what they eat, volume, and reaction to food using asimple sentence. Many packaged food and processed readyto prepare food items have barcodes. Since barcode readersare now omnipresent even in smartphones, one can get allfood information from these. Some sensors measure mus-cle activity and try to infer food items from that. These areplaced on the chest, near the ear, or neck [10]. These haveshown some progress in recognizing eating events but havenot gone much beyond that yet.We propose a multimedia food logging platform, shownin Figure 5, that could use many relevant sources to logfood items and find all information that may be needed tobuild a PFM. Multimedia uses complementary and correlatedinformation and provides more comprehensive and preciseinformation than any one medium. Moreover, we will keepadding new sensors and technologies to keep the loggeruseful.This is the beginning of building towards a robust multi-media solution to the problem of logging. There are threeimportant aspects to this platform: • We can design a multimedia platform that uses vi-sual food recognition, speech-based systems, paymentbased options, barcodes, sensors to determine foodchewing and content of the food, and several similaremerging approaches. • Once a dish or food item is recognized and the amountconsumed is known, systems must find the nutritionaldata using governmental or commercial databases. Sim-ilarly, weather information, social context, and othermetadata related to food required by the PFM maycome from other sources. • The log must contain the user’s reaction both in termsof enjoyment and bodily reaction. The enjoyment in-formation may come from asking the user, and the
Figure 5: Food logging will use multimedia input sourcesand complement information from online databases to logeach meal and all metadata related to the meal. It capturesinformation about the food (dish name, ingredients, quan-tity), location (place of eating), time (eating and logging), so-cial context (companions), causal aspects (nutritional andflavor information), and multimedia and experiential infor-mation about the food. bodily reactions may come from sensors such as heartrate, glucose measurement, and respiration rate.
Utilizing Other Knowledge and Data sources
We enrich the food events with associated nutritional, culi-nary, and contextual information using databases from dif-ferent public and private organizations. These include nutri-tion (NutritionIX, USDA food database), weather, air-quality(airnow provided by EPA), and place (Google Places, Yelp).We may also want to capture some biomarkers characterizingthe health of the person. These parameters may be continu-ously recorded and could be used to identify physiologicalresponses to food items [38]. A personicle like system [37]can capture this information, and the time-indexed nature ofthe data and events makes it readily available for associatingand analyzing with the food events.
Data Model for the Foodlog
Food logs are collected for • Building PFM to understand the nutritional require-ments and taste preferences of the user. • Understanding the health state of the user.These two goals may require different information fromthe food log. Building PFM requires as much longitudinaldata as available, while health state estimation requires PFMand recent lifestyle and biological data. We need to keepthese goals in mind while designing the food log. We havefollowed the HW5 (how, what, when, why, where, what)model as described in [57], [56] to identify what informationcan fully describe a food event and maximize its utility for avariety of applications. The different aspects and associated
M ’20, October 12–16, 2020, Seattle, WA, USA Rostami, et al. information are detailed in figure 5. There can be three typesof data collected:(1) Observed data: Directly captured using a sensor.(2) Derived data: We can derive some data and informationusing sensors and knowledge sources. This informa-tion will depend on the algorithms and data sourcesused.(3) Subjective data: The system may prompt the user orsome other human source to get specific information.This data is prone to errors as it depends on humanperception.We should utilize the different types of measurements indifferent manners to minimize the error in our analyses andpredictions.
Current Status
In this paper, we describe the data and knowledge neededto build a PFM directed at improving sleep quality. We con-sidered that the sleep quality is affected by stress, activity,and food [6]. We are implementing a food logging platform.We decided to focus on data collection using voice, text, andbarcode for the current version. We will include visual recog-nition approaches soon.We add food metadata in the log using weather and reverse-geo databases. The foodlogger asks the user about their re-action to each item entered. We used NutrionIX platform toget information about calories and nutrients in each fooditem. The current food logger has information about howmuch a user likes a dish to build the Preferential PersonalFood model. However, the information about the taste andflavor of a food item is not readily available from any source.We are working towards deriving such information aboutfood items from different sources. This is an excellent openopportunity for the multimedia community to take the leadin solving this critical problem.
As stated in the previous sections, the personal model shouldbe able to incorporate existing knowledge sources. We havesurveyed papers that explore the relationship between di-etary inputs and sleep outcomes. We summarize our findingsin figure 6. We found that macro nutrients have a great im-pact on sleep outcomes [53], [44], [43], [58], [5], [52]. Somemicronutrients contribute to melatonin secretion, and hencecan have significant impact on sleep quality [14], [55]. Ad-ditionally, there are some studies that explore the effect ofspecific food items such as kiwi fruit [22] and cherries [45],[25], [13] on sleep. Some chemicals responsible for specifictaste such as capsaicin [11] and sugar [48], [54] can alsoimpact sleep. Fasting contributes to the change of bedtime[7] as well.
Figure 6: Attributes of food events that impact sleep qual-ity. These relationships form the basis for the Biological Per-sonal Food Model.
We have also included some studies about the impact of ex-ercise and physical activity on sleep [23], [24], [21], [41] asit is an important confounding variable that impacts bothnutritional needs and sleep quality.
Figure 7: Event Mining workflow: Hypothesis generation op-erators allows us to find frequently occurring sequences ofevents. These can be converted to hypotheses by includingconfounding variables and can then be tested in presenceof these confounding factors using hypothesis verificationoperator. These verified hypotheses serve as a rule-basedmodel for the user’s behavior.
Multimedia research in event mining has focused on eventrecognition and situation understanding (eg., sports andsurveillance videos). There has not been much research onhow we can utilize event mining to run n-of-1 experiments ersonal Food Model MM ’20, October 12–16, 2020, Seattle, WA, USA using a person’s events and data streams and derive rulesthat describe their behavior in different situations. Eventmining allows us to find patterns and relationships betweendifferent events in our daily lives. We can find relationshipsbetween different events in a person’s lifelog data and derivean explainable personal model [39].Event mining results in rules of the form
Event i C −→ Event o ,where Event i is the input event, and we want to find out itseffect in the occurrences of the outcome, Event o . C definesthe set of confounding variables and temporal conditionsthat might affect this relationship. For Biological-PFM, theinput events are lifestyle events that have a causal impacton some observable biological outcome [40]. While, in thePreferential-PFM, the input events capture the contextualsituations that affect the user’s culinary preferences. Thisview of events and their impacts is in line with the poten-tial outcomes framework for causal inference (provided therequired assumptions, eg., SUTVA are valid) [47] and areexplored in detail later in this section.We perform a two-step analysis with a human expert act-ing as an intermediary to select non-spurious relationships.The event patterns language described in [17] allows us todescribe the relationships as temporal patterns of events. Hy-pothesis generation is used as a preliminary investigationtool that allows a human expert to identify any behavioralpatterns in the form of events co-occurrences and
Hypothe-sis verification tells us whether the relationship is causallysignificant in the presence of the confounding variables.
Hypothesis generation: Discovering new behavioralpatterns
Users’ event logs contain all of their daily habits and bio-logical responses to different events. Hypothesis generationoperators allow us to discover these habits and patterns thatcan be tested and used for prediction. This step of the anal-ysis is mostly data-driven and starts with a human expertspecifying the event streams that they believe to be corre-lated. The output is a heat map with different combinations ofevents occupying different positions (Figure 7). The patternswith relatively higher frequency may represent a significantrelationship and selected for hypothesis verification.The frequent patterns would then need to be converted tocandidate hypotheses. The user would need to specify thecause and effect events along with any confounding factors.This hypothesis can then be verified using the hypothesisverification operator.
Hypothesis verification: Verifying patterns underdifferent contexts
Users can also verify their beliefs by encoding those as pat-terns of events and specifying the variables that define the contextual situation. We defined these patterns using sci-entific literature, as described in the previous section. Eachoccurrence of the pattern represents an instance of the inputevent (treatment), and we want to measure its impact on theoutcome. Thus, each occurrence of the pattern becomes asingle unit in the potential outcomes framework [47], andwe can compare different units while matching them by theconfounding factors, to estimate the causal effect of the treat-ment.Once we have found all the pattern occurrences and the con-founding variables, we follow a two-step process to find thevalidity of the rule.(1)
Find similar situations based on confounding vari-ables (Contextual Matching) . The confounding vari-ables define the situation in which the input event(treatment) occurs, and can affect the event relation-ship we want to analyze. Therefore, we want to com-pare the events that occur in similar contexts and com-pare the impact of the input event on the outcome inan unbiased manner. We can do it by either clusteringthe values of confounding variables or converting theconfounding variables to events and find matchingconfounding event patterns.(2)
Find the validity of the relationship for each situ-ation.
Once we have performed the contextual match-ing, for each contextual group, we can find the effectof the treatment on the outcome using an appropri-ate statistical test. We can compare the difference inthe outcome for different input events, and this wouldtell us the relative causal effect of the different inputevents.This two-step hypothesis verification allows us to simulatean N-of-1 experiment on the user’s event log while alsoincorporating the existing scientific knowledge in the form ofcandidate hypotheses and identifying confounding variables.
Deriving Personal Food Model using Event mining
We need to analyze the food log in conjunction with otherevents from the personicle to create an explainable and per-sonalized food model for every individual. The model wouldpredict the impact of food events on other aspects of a per-son’s life; and how different lifestyle and biological factorsimpact our food choices. In this paper, we are exploring therelationship between food events and sleep outcomes; there-fore, we will include behavioral factors that would impactthese two events, such as physical activity (exercise, stepcount).We can identify different behavioral habits of the userusing hypothesis generation operators. We can also visual-ize the relationship between various nutritional factors anddifferent sleep outcomes to find if these are worth exploring
M ’20, October 12–16, 2020, Seattle, WA, USA Rostami, et al. further. Once we have identified such relationships, we canstart verifying these hypotheses. We derive the hypothesesfrom data or existing biomedical literature. These relation-ships have been detailed in the previous section and are alsodepicted in figure 6.Figure 7 shows the complete event mining process for thepersonal food model. The verified hypothesis contain eventrelationships that hold true in the specified contextual situ-ation. These relationships form a set of rules with varyingdegrees of accuracy in different contextual situations. Forexample, if we have verified that cow’s milk has a positiveimpact on sleep latency, then the relationship would be quan-tified in the form of minutes reduced in latency. It will havea different value for different contextual situations describedby physical activity, day’s meals, and last night’s sleep. Theserules could thus be used to identify the potential outcome ofdifferent foods and recommend items with the desired sleepoutcome.
Though this paper focuses primarily on personal food mod-els, it is really about building personal health models usingdisparate data and information sources. A personal healthmodel’s importance is apparent in these days of a pandemicthat has disrupted lives globally. In this section, we discussinteresting challenges that we need to address. We believethat multimedia computing offers concepts, techniques, andpractical experiences related to key areas mentioned in thepaper.(1) User Privacy: User privacy and data protection are inte-gral to developing a multimedia personal model. With-out adequate security measures the model is unlikelyto be widely adopted, regardless of the performanceor utility. This is an important challenge for multi-media, artificial intelligence, and privacy and securityresearch groups and we are actively looking for col-laborations in this area. There are learning techniquessuch as federated learning [59] that allow us to buildmodels and share insights without taking users’ datafrom their device. We need to incorporate such meth-ods in our platforms so that the users have completeownership of their data.(2) Taste Space: Taste and flavor of food are very complex.Food taste space depends on the ingredients and recipeas well as visual presentations. On the other hand, eachperson has their own preferred taste space that must bedetermined by observations over a long time. We areexploring 6-dimensional taste space. This is less thanthe tip of the proverbial iceberg. Such representationswill result in labeling food items better so that people can select what they will enjoy eating and will behealthy.(3) Multimedia Logging Platform: Multimedia communityhas focused on food logging using only visual recog-nition approaches and has been limited only to dishand ingredient recognition. Food logging is not justrecognizing dishes from pictures, but identifying allcharacteristics of an eating event. We need to build amultimedia logging platform to collect all food-relatedinformation relevant to building PFM. Such logs couldbe used for studying population for health as well asfor business reasons.(4) Multimodal event detection: The health state of a per-son is usually estimated by combining multimedia(audio-visual) and multimodal (heart rate, EEG, res-piration rate, Glucose content) signals. Estimation ofhealth state is a great challenge for researchers thatwill also help ALL humans.(5) Multimodal Knowledge Collection: Much of the di-agnosis and prescription related to health is multi-modal and will require extending traditional knowl-edge graph [60][15] techniques.(6) Event Mining: Mining multiple sequences of eventstreams detected from disparate data streams is essen-tial for both building models such as PFM as well asfor health state estimation. Event mining may offermore challenging problems in predictive and preven-tive approaches in several application areas, includinghealth using novel forms of machine learning than ob-ject recognition offered in computer vision. We havealready started building a platform for this.(7) Recommendation System to motivate behavioral change:In context of eating habits, a recommendation systemwhich always promotes the healthiest option is notnecessarily the best one. A good recommendation mustconsider personal food preferences and healthiness to-gether to suggest not just healthy but correct amountof ’healthy and tasty’ food. Correct recommendationshould be given at the correct place and the appropri-ate time to motivate behavioral change [42], [3]. PFMis the first step towards context-aware recommenda-tion in food domain but this is just the beginning of along journey.
REFERENCES
FoodLog: Multimedia Tool for Healthcare Applications ersonal Food Model MM ’20, October 12–16, 2020, Seattle, WA, USA [4] Gediminas Adomavicius and Alexander Tuzhilin. [n.d.].
Toward theNext Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions . Technical Report.[5] Ahmad Afaghi, Helen O’Connor, and Chin Moi Chow. 2008. Acuteeffects of the very low carbohydrate diet on sleep indices.
Nutri-tional Neuroscience
11, 4 (2008), 146–154. https://doi.org/10.1179/147683008X301540[6] Iman Azimi, Olugbenga Oti, Sina Labbaf, Hannakaisa Niela-Vilen,Anna Axelin, Nikil Dutt, Pasi Liljeberg, and Amir M. Rahmani. 2019.Personalized maternal sleep quality assessment: An objective iot-basedlongitudinal study.
IEEE Access
Journal ofResearch in Medical Sciences
18, 2 (2013), 127–131.[8] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. 2014. Food-101- Mining discriminative components with random forests. In
LectureNotes in Computer Science (including subseries Lecture Notes in Artifi-cial Intelligence and Lecture Notes in Bioinformatics) , Vol. 8694 LNCS.Springer Verlag, 446–461. https://doi.org/10.1007/978-3-319-10599-4{_}29[9] Jingjing Chen and Chong Wah Ngo. 2016. Deep-based ingredientrecognition for cooking recipe retrieval. In
MM 2016 - Proceedingsof the 2016 ACM Multimedia Conference . Association for ComputingMachinery, Inc, 32–41. https://doi.org/10.1145/2964284.2964315[10] Michael Chu, Thao Nguyen, Vaibhav Pandey, Yongxiao Zhou, Hoang N.Pham, Ronen Bar-Yoseph, Shlomit Radom-Aizik, Ramesh Jain, Dan M.Cooper, and Michelle Khine. 2019. Respiration rate and volume mea-surements using wearable strain sensors. npj Digital Medicine
2, 1 (122019), 1–9. https://doi.org/10.1038/s41746-019-0083-3[11] Stephen J. Edwards, Iain M. Montgomery, Eric Q. Colquhoun, Jo E.Jordan, and Michael G. Clark. 1992. Spicy meal disturbs sleep: an effectof thermoregulation?
International Journal of Psychophysiology
13, 2(1992), 97–100. https://doi.org/10.1016/0167-8760(92)90048-G[12] Mehdi Elahi, Mouzhi Ge, Francesco Ricci, Ignacio Fernández-Tobías,Shlomo Berkovsky, and Massimo David. [n.d.].
Interaction Design in aMobile Food Recommender System . Technical Report.[13] M Garrido, D González-Gómez, M Lozano, C Barriga, S D Paredes, andA B Rodríguez. 2013. A Jerte valley cherry product provides beneficialeffects on sleep quality. Influence on aging.
The journal of nutrition,health & aging
17, 6 (6 2013), 553–60. https://doi.org/10.1007/s12603-013-0029-4[14] Satoko Hashimoto, Masako Kohsaka, Nobuyuki Morita, Noriko Fukuda,Sato Honma, and Ken Ichi Honma. 1996. Vitamin B12 enhances thephase-response of circadian melatonin rhythm to a single bright lightexposure in humans.
Neuroscience Letters
FoodKG: A Semantics-Driven KnowledgeGraph for Food Recommendation . Technical Report. http://dbpedia.org/resource/Chocolate[16] Laleh Jal, Hyungik Oh, Siripen Pongpaichet, Ramesh Jain, Laleh Jalali,Da Huo, and Mengfan Tang. 2014.
Personicle: Personal Chronicle of LifeEvents Smart city systems View project electrical characterization of de-vices View project Personicle: Personal Chronicle of Life Events
Tracing Complexity in Food Blogging Entries . Techni-cal Report.[19] Emad Kasaeyan Naeini, Sina Shahhosseini, Ajan Subramanian, TingjueYin, Amir M. Rahmani, and Nikil Dutt. 2019. An Edge-Assisted and Smart System for Real-Time Pain Monitoring. In
Proceedings - 4thIEEE/ACM Conference on Connected Health: Applications, Systems andEngineering Technologies, CHASE 2019 . Institute of Electrical and Elec-tronics Engineers Inc., 47–52. https://doi.org/10.1109/CHASE48038.2019.00023[20] Fang Fei Kuo, Cheng Te Li, Man Kwan Shan, and Suh Yin Lee. 2012. In-telligent menu planning: Recommending set of recipes by ingredients.In
CEA 2012 - Proceedings of the 2012 ACM Workshop on Multimedia forCooking and Eating Activities, Co-located with ACM Multimedia 2012 .1–6. https://doi.org/10.1145/2390776.2390778[21] Maya J. Lambiase, Kelley Pettee Gabriel, Lewis H. Kuller, and Karen A.Matthews. 2013. Temporal relationships between physical activity andsleep in older women.
Medicine and Science in Sports and Exercise
45, 12(12 2013), 2362–2368. https://doi.org/10.1249/MSS.0b013e31829e4cea[22] Hsiao Han Lin, Pei Shan Tsai, Su Chen Fang, and Jen Fang Liu. 2011.Effect of kiwifruit consumption on sleep quality in adults with sleepproblems.
Asia Pacific Journal of Clinical Nutrition
20, 2 (6 2011),169–174. https://doi.org/10.6133/apjcn.2011.20.2.05[23] Paul D. Loprinzi and Bradley J. Cardinal. 2011. Association betweenobjectively-measured physical activity and sleep, NHANES 2005-2006.
Mental Health and Physical Activity
4, 2 (12 2011), 65–69. https://doi.org/10.1016/j.mhpa.2011.08.001[24] Paul D. Loprinzi, Kristina L. Loprinzi, and Bradley J. Cardinal. 2012.The relationship between physical activity and sleep among pregnantwomen.
Mental Health and Physical Activity
5, 1 (6 2012), 22–27.https://doi.org/10.1016/j.mhpa.2011.12.002[25] Jack N. Losso, John W. Finley, Namrata Karki, Ann G. Liu, AlfredoPrudente, Russell Tipton, Ying Yu, and Frank L. Greenway. 2018. PilotStudy of the Tart Cherry Juice for the Treatment of Insomnia andInvestigation of Mechanisms.
American journal of therapeutics
25, 2 (32018), e194–e201. https://doi.org/10.1097/MJT.0000000000000584[26] Robert Mai, Stefan Hoffmann, Jens R. Helmert, Boris M. Velichkovsky,Susann Zahn, Doris Jaros, Peter Eh Schwarz, and Harald Rohm. 2011.Implicit food associations as obstacles to healthy nutrition: The needfor further research.
British Journal of Diabetes and Vascular Disease
11, 4 (7 2011), 182–186. https://doi.org/10.1177/1474651411410725[27] David Julian Mcclements. [n.d.].
FUTURE FOODS HOW MODERNSCIENCE IS TRANSFORMING THE WAY WE EAT . Technical Report.[28] Weiqing Min, Shuqiang Jiang, and Ramesh C. Jain. 2019. Food Rec-ommendation: Framework, Existing Solutions and Challenges.
IEEETransactions on Multimedia (12 2019), 1–1. https://doi.org/10.1109/tmm.2019.2958761[29] Weiqing Min, Shuqiang Jiang, Linhu Liu, Yong Rui, and Ramesh Jain.2019. A survey on food computing.
Comput. Surveys
52, 5 (9 2019).https://doi.org/10.1145/3329168[30] Nitish Nag. 2020. Health State Estimation. (2020). http://arxiv.org/abs/2003.09312[31] Nitish Nag and Ramesh Jain. 2019. A Navigational Approach to Health:Actionable Guidance for Improved Quality of Life.
Computer
52, 4 (42019), 12–20. https://doi.org/10.1109/MC.2018.2883280[32] N. Nag, V. Pandey, and R. Jain. 2017. Live personalized nutrition rec-ommendation engine. In
MMHealth 2017 - Proceedings of the 2nd Inter-national Workshop on Multimedia for Personal Health and Health Care,co-located with MM 2017 . https://doi.org/10.1145/3132635.3132643[33] Nitish Nag, Vaibhav Pandey, Likhita Navali, Prateek Mohan, andRamesh Jain. 2019. Synchronizing Geospatial Information for Person-alized Health Monitoring. (7 2019). http://arxiv.org/abs/1907.10594[34] Nitish Nag, Vaibhav Pandey, Preston J. Putzel, Hari Bhimaraju, SrikanthKrishnan, and Ramesh Jain. 2018. Cross-modal health state estimation.In
MM 2018 - Proceedings of the 2018 ACM Multimedia Conference .Association for Computing Machinery, Inc, New York, New York, USA,1993–2002. https://doi.org/10.1145/3240508.3241913
M ’20, October 12–16, 2020, Seattle, WA, USA Rostami, et al. [35] Nitish Nag, Vaibhav Pandey, Abhisaar Sharma, Jonathan Lam, RunyiWang, and Ramesh Jain. 2017. Pocket Dietitian: Automated HealthyDish Recommendations by Location. In
Lecture Notes in ComputerScience (including subseries Lecture Notes in Artificial Intelligence andLecture Notes in Bioinformatics) , Vol. 10590 LNCS. Springer Verlag,444–452. https://doi.org/10.1007/978-3-319-70742-6{_}43[36] Eugene Newton Anderson. [n.d.].
Everyone Eats: Understanding Foodand Culture . Technical Report.[37] Hyungik Oh and Ramesh Jain. 2017. From multimedia logs to personalchronicles. In
MM 2017 - Proceedings of the 2017 ACM MultimediaConference . Association for Computing Machinery, Inc, 881–889. https://doi.org/10.1145/3123266.3123375[38] Hyungik Oh, Soundarya Soundararajan, Jonathan Nguyen, andRamesh Jain. 2018. Multimodal food journaling. In
HealthMedia 2018 -Proceedings of the 3rd International Workshop on Multimedia for Per-sonal Health and Health Care, co-located with MM 2018 . Association forComputing Machinery, Inc, 39–47. https://doi.org/10.1145/3264996.3265000[39] Vaibhav Pandey, Nitish Nag, and Ramesh Jain. [n.d.]. Ubiquitous EventMining to Enhance Personal Health. ([n. d.]). https://doi.org/10.1145/3267305.3267684[40] Vaibhav Pandey, Nitish Nag, and Ramesh Jain. 2020. ContinuousHealth Interface Event Retrieval. (4 2020). https://doi.org/10.1145/3372278.3390705[41] Subin Park. 2014. Associations of physical activity with sleep sat-isfaction, perceived stress, and problematic Internet use in Koreanadolescents.
BMC Public Health
14, 1 (11 2014), 1143. https://doi.org/10.1186/1471-2458-14-1143[42] Mitesh S. Patel, David A. Asch, and Kevin G. Volpp. 2015. Wearabledevices as facilitators, not drivers, of health behavior change. , 459–460 pages. https://doi.org/10.1001/jama.2014.14781[43] Katri Peuhkuri, Nora Sihvola, and Riitta Korpela. 2012. Diet promotessleep duration and quality. , 309–319 pages. https://doi.org/10.1016/j.nutres.2012.03.009[44] Katri Peuhkuri, Nora Sihvola, and Riitta Korpela. 2012. Dietary factorsand fluctuating levels of melatonin.
Food & Nutrition Research
56, 1 (12012), 17252. https://doi.org/10.3402/fnr.v56i0.17252[45] Wilfred R. Pigeon, Michelle Carr, Colin Gorman, and Michael L. Perlis.2010. Effects of a tart cherry juice beverage on the sleep of older adultswith Insomnia: A pilot study.
Journal of Medicinal Food
13, 3 (6 2010),579–583. https://doi.org/10.1089/jmf.2009.0096[46] Faisal Rehman, Osman Khalid, Nuhman Ul Haq, Atta Ur Rehman Khan,Kashif Bilal, and Sajjad A. Madani. 2017. Diet-right: A smart foodrecommendation system.
KSII Transactions on Internet and InformationSystems
11, 6 (2017), 2910–2925. https://doi.org/10.3837/tiis.2017.06.006[47] Donald B. Rubin. 2005. Causal inference using potential outcomes:Design, modeling, decisions.
J. Amer. Statist. Assoc.
Nutrition
48 (4 2018),77–81. https://doi.org/10.1016/j.nut.2017.11.013[49] Cecile Sarabian, Barthelemy Ngoubangoye, and Andrewj J. Macintosh.2017. Avoidance of biological contaminants through sight, smell andtouch in chimpanzees.
Royal Society Open Science
4, 11 (11 2017).https://doi.org/10.1098/rsos.170968[50] Matthias B. Schulze, Miguel A. Martínez-González, Teresa T. Fung,Alice H. Lichtenstein, and Nita G. Forouhi. 2018. Food based dietarypatterns and chronic disease prevention.
BMJ (Online)
361 (2018).https://doi.org/10.1136/bmj.k2396 [51] Charles Spence. 2015. Multisensory Flavor Perception. , 24–35 pages.https://doi.org/10.1016/j.cell.2015.03.007[52] Marie Pierre St-Onge, Amy Roberts, Ari Shechter, and Arindam RoyChoudhury. 2016. Fiber and saturated fat are associated with sleeparousals and slow wave sleep.
Journal of Clinical Sleep Medicine
12, 1(2016), 19–24. https://doi.org/10.5664/jcsm.5384[53] Eizaburo Tanaka, Hiroshi Yatsuya, Mayu Uemura, Chiyoe Murata,Rei Otsuka, Hideaki Toyoshima, Koji Tamakoshi, Satoshi Sasaki, LeoKawaguchi, and Atsuko Aoyama. 2013. Associations of protein, fat,and carbohydrate intakes with insomnia symptoms among middle-aged Japanese workers.
Journal of Epidemiology
23, 2 (2013), 132–138.https://doi.org/10.2188/jea.JE20120101[54] Robert E Thayer. [n.d.].
Energy, Tiredness, and Tension Effects of a SugarSnack Versus Moderate Exercise . Technical Report 1. 119–125 pages.[55] Maija Valtonen, Leo Niskanen, Antti Pekka Kangas, and Teuvo Kosk-inen. 2005. Effects of melatonin-rich night-time milk on sleep andactivity in elderly institutionalized subjects.
Nordic Journal of Psychia-try
59, 3 (2005), 217–221. https://doi.org/10.1080/08039480510023034[56] U Westermann, R Jain IEEE multimedia, and undefined 2007. [n.d.].Toward a common event model for multimedia applications. ieeex-plore.ieee.org ([n. d.]). https://ieeexplore.ieee.org/abstract/document/4061303/[57] L Xie, H Sundaram, M Campbell Proceedings of the IEEE, and un-defined 2008. [n.d.]. Event mining in multimedia streams. ieeex-plore.ieee.org ([n. d.]). https://ieeexplore.ieee.org/abstract/document/4468738/[58] Miwa Yamaguchi, Hirokazu Uemura, Sakurako Katsuura-Kamano,Mariko Nakamoto, Mineyoshi Hiyoshi, Hidenobu Takami, FusakazuSawachika, Tomoya Juta, and Kokichi Arisawa. 2013. Relationshipof dietary factors and habits with sleep-wake regularity.
Asia PacificJournal of Clinical Nutrition
22, 3 (2013), 457–465. https://doi.org/10.6133/apjcn.2013.22.3.01[59] Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. 2019. Fed-erated machine learning: Concept and applications.
ACM Trans-actions on Intelligent Systems and Technology
10, 2 (1 2019), 1–19.https://doi.org/10.1145/3298981[60] Unai Zulaika, Asier Gutiérrez, and Diego López-de Ipiña. 2018. En-hancing Profile and Context Aware Relevant Food Search throughKnowledge Graphs.