Augmenting reality: On the shared history of perceptual illusion and video projection mapping
aa r X i v : . [ c s . MM ] M a y Augmenting reality:On the shared history of perceptual illusion and videoprojection mapping
Alvaro PastorComputer Science, Multimedia and Telecommunications Department.Universitat Oberta de Catalunya. Barcelona, [email protected]
Abstract
Perceptual illusions based on the spatial correspondence between objects anddisplayed images have been pursued by artists and scientists since the 15th century,mastering optics to create crucial techniques as the linear perspective and devices asthe
Magic Lantern . Contemporary video projection mapping inherits and furtherextends this drive to produce perceptual illusions in space by incorporating therequired real time capabilities for dynamically superposing the imaginary ontophysical objects under fluid real world conditions. A critical milestone has beenreached in the creation of the technical possibilities for all encompassing, untetheredsynthetic reality experiences available to the plain senses, where every surface may actas a screen and the relation to everyday objects is open to perceptual alterations.
Introduction
The advancement and popularization of computer technologies over the past threedecades has had a significant effect not only on the way images are created, but alsoon the techniques and technologies which enable their presentation. One of the mostpopular emerging practices is video projection mapping, also referred as videomapping or 3D mapping, a terminology which stands for a set of imaging techniquesin service of the spatially coherent projection of bi dimensional images onto physicalthree dimensional objects [1, 2]. Despite its seemingly novelty in its use in the fields ofcommunication, advertising and visual arts in the recent years, the current practice ofvideo projection mapping is the result of several centuries of development andperfecting of a vast set of techniques derived from optics and the sciences of humanperception, as well as mathematical operations for image transformations, and theevolution projection devices. Across all these periods in history, the production ofperceptual illusions remains as the fundamental shared goal, of which importantmilestones are the invention of linear perspective techniques and its application in
Trompe l’oeil and
Quadratura installations during the Renassaince, the immersivetheatrical qualities of
Panorama and
Phantasmagoria in the 18th century, thedevelopment of real time 3D computer rendering capabilities and the popularization ofvideo projection mapping during the last part of the 20th century. The present articleanalyzes the evolution of video projection mapping in relation to the history ofperceptual illusion practices, considering common design methodologies, tools, andSep 19, 2019 1/14mplemented techniques, in a transversal selection of scientific and artistic practices,spreading from static compositions on a canvas, to dynamic substitutions in threedimensional space. A commentary on the emerging perceptual and cognitivephenomena as a result of the spatial distribution of synthetic contents is provided,presenting the notion that the current stance of real time dynamic forms of videoprojection mapping constitutes a turning point in the quest for replacing the realworld with synthetic versions that would offer unprecedented means for visual identityand meaning transformations beyond material constraints.
Spatial illusion in the Renassaince
Before the 14th Century few attempts were made to create the illusion of threedimensional space and to realistically depict the spatial world in art. The Italianmasters Giotto and Duccio during the late thirteenth and early fourteenth centuries,began to explore the idea of depth and volume in their artwork, and may be creditedwith introducing an early form of three dimensional painting, using shadows andcontrasts in colors to create the perceived illusion of depth [3, 4]. Although it was stillvery far from the accurate and robust three dimensional representations popular inimagery today, this technique known as
Chiaroscuro helped artists to create theillusion of volume and depth. Popularized during the beginning of the fifteenthcentury, Chiaroscuro was mastered especially for the processes of modeling andrepresenting the human body and was extensively used during the sixteenth andseventeenth centuries in the Mannerism and Baroque eras as one of the standards forapproximating three dimensional representation. Not satisfied with the state of theart, Florentine painters of the early fifteenth century, wanted to develop painting as ascience derived from Euclidean geometry influenced by the Neoplatonic intellectualpostulates. In this sense, they did not conform with the intuitive Chiaroscurotechnique, but sought scientific methods of representing reality based on mathematicallaws, thus giving birth to the knowledge that ultimately led to the devise of the linearperspective system. The linear perspective system projected the illusion of depth ontoa two dimensional plane by use of vanishing points to which all lines converged at eyelevel on the horizon, accurately reproducing the way that objects appear smaller whenthey are further away, and the way parallel lines appear to meet each other at a pointin the distance.Between 1415 and 1420, the Italian architect Filippo Brunelleschi, considered oneof the founding figures of the Renassaince, conducted and documented a series ofexperiments which helped him develop the correct perspective techniques foraccurately representing three dimensional volumes on a flat canvas. One of its resultsportrayed in accurate spatial proportion and organization, the Florence Baptisteryand the Palazzo Vecchio building seen from the front gate of the unfinished cathedral,affording to the spectator the full illusion of being present in the scene. Thesetechnical advancements where later systematized by Leon Battista Alberti in his 1436work
De Pictura , one of the most important treatises on painting of the Renaissance,enabling artists for the first time to paint imaginary scenes with perfectly accuratethree dimensional realism by following the set of rules of perspective studied byBrunelleschi [5–8]. Gradually, a standardized model of human visual perception wasconsolidated, from which a set of mathematical rules was derived and applied togovern the disposition of pictorial elements in order to produce a credible illusion ofthree dimensionality.For the next five centuries, Brunelleschi’s system of perspective and its derivedtechniques were used to create the illusion of depth on the picture plane, and served asthe basis of some of the great art of Western culture [9–11]. Among the earliest ofSep 19, 2019 2/14hose, The Brera Madonna created between 1472 and 1474 by the Italian artist Pierodella Francesca. It is a large painting depicting a group of religious figures and iconsat a fictitious apse of the church, meticulously rendered using the innovations of linearperspective techniques. But not only did perspective techniques allowed to produce acoherent illusion of space on a flat surface, but perspective was also used to organizeand highlight particular elements of the composition, thus allocating meaning inrelation to specific spatial positions in the canvas [12, 13]. For example, the geometriccenter of the face of the main character coincides with the vanishing point of theperspective system used in the scene, as well as the egg which hangs from a shell -shaped element below the apse, is perfectly aligned with the scene’s vanishing pointand marks a perpendicular line to the scene’s horizon. Tommaso di Ser Giovanni diSimone, best known as
Masaccio and regarded as the first great Italian painter of the
Quattrocento period of the Italian Renaissance, painted
The Holy Trinity in 1425 inthe dominican church of Santa Maria Novella in Florence. This pioneering muralpainting not only accurately presented the illusion of three dimensions using linearperspective techniques, but also engaged in a coherent composition with the preexisting architectural elements that surrounded the mural, mixing three dimensionalpainted and architectural elements to enhance the illusory effect. This specific type ofillusion supported by the coherent merger between painting techniques andarchitectural elements was known as Quadratura [14]. Artist Andrea Mantegnafurther experimented developing illusionistic ceiling paintings using perspectivetechniques to create the perceptual illusions and was responsible in 1465 for one of themost significant examples of Quadratura. Located in the Camera degli Sposi in theDucal Palace of Mantua, this Trompe l’oeil was applied to ceiling paintings to be seenby spectators from below and upward and presented the illusionistic painting of anoculus that opens to the sky and a number of fictional characters who look downtowards the spectators [15]. Performed by Andrea Pozzo in 1688,
The Apotheosis of StIgnatius in Sant Ignazio in Rome is another remarkable example of Quadratura,presenting in a 55 feet wide ceiling a composition combining fictional characters andarchitectural elements. A new type of site specific art form began maturing, unifyingarchitecture, painting and sculpture, and being conveyed to spectators by means of afull body spatial experience. In a sense, this almost theatrical experience was prescientof the aesthetic ideals of
Gesamtkunstwerk , postulated years later in mid nineteenthcentury by the German opera composer Richard Wagner.
Panorama and Phantasmagoria
Baroque painters from seventeenth century, notably Caravaggio, Bernini, Rubens,Rembrandt, Velazquez and Vermeer, used the now long established three dimensionalrepresentation system in hundreds of varied approaches, as did the painters from theNeoclassicist movement, of whom a notable example is the work of Giovanni BatistaPiranesi.
Carceri d’invenzione , a series prints produced between 1745 and 1761 byPiranesi depicting somber fictional scenes full of mysterious characters, bridges, stairsand mechanisms, trough meticulous utilization of the perspective technique. However,few artists during these decades shared the interest for the experimentation anddevelopment of perceptual illusions via spatial experience. In this sense, a majorreferent is the work of Robert Barker, an Irish painter that developed the idea oferecting a circular plan building to house near real scale paintings featuring panoramiclandscapes. In 1792 the Panorama was born, supported on the knowledge onperspective and color contrast practiced during the last two centuries, its paintingscovered 360 degrees around the spectator in the horizontal plane and reached 15 meterof height. Soon, Barker’s Panorama grew in size and complexity, and the circular planSep 19, 2019 3/14nstallation incorporated topographies and architectural and elements intended tomatch the painted scene thus increasing its illusory effect in a similar way asQuadratura incorporated architectural elements to its composition. This way, flooring,stairs, railings, water fountains or other urban furniture, not only to regulate thedisplacements of spectators inside the building, but more importantly, to serve asimmersive cues, formal and semantic links between the depicted scenes and the notionof what was accountable as real world experience [16, 17]. The Panorama continued itspopularization in most of Western Europe and North America as an entertainmentform, successfully hosting the representation of famous historical battles for the greatpublic. In a time when few people could afford to travel, the Panorama allowed towalk through distant landscapes and contemplate historical events from privilegedpoints of view, pioneering in a primitive form of narrative distributed in space andtime, in a sense similar as it is now possible to explore in a virtual reality experience ora video game [18, 19]. During the precedent decades, the rapid increase of knowledge inoptics was not only useful for development and use of perspective related techniques,but also supported a number of relevant inventions regarding optical instruments,among those, light reflection and projection devices. The Jesuit Athanasius Kircher inhis book
Ars Magna Lucis et Umbrae from 1645, described a primitive projectionsystem with a focusing lens and a concave mirror reflecting sunlight that served ascanvas to display text or pictures known as the
Steganographic Mirror . Furtherelaborating, dutch scientist Christiaan Huygens in 1659 documented the invention ofthe
Magic Lantern , an early type of image projector that used semi transparent, handpainted glass plates, one or more lenses, and a light source [20]. The Magic Lanternwas then consolidated, and increasingly used for education and entertainmentpurposes during the 18th century [21]. In this period, although the basic design of theMagic Lantern suffered various modifications in relation to the light source capabilitiesand the characteristics of lenses in use, its main fundamental concept remainunchanged as well as its basic operation. By the end of 18th century, Belgian Physicistand cleric Ettiene Gaspar Robert, and one of the first lanternists of all time PaulPhilidor, created a new form of illusion performance in the form of a Magic Lanternshow known as Phantasmagoria, a theatrical show that consisted of a mixture offrontal and retro projection of images, from simultaneous magic lanterns, on canvasesand solid elements in front of the sitting public [22–24]. During the 19th century, theinvention was further sophisticated, with more powerful light sources, incorporation ofphotographic impressions onto glass plates, improvements in optics, and mechanismsthat allow the projection of sequences of images, helping consolidate the expressivematurity of a new type visual spectacle still greatly influenced by the use ofperspective and contrast in colors to achieve volumetric realism [25]. Following thistheatrical illusory tradition, Louis Daguerre in 1822 developed the
Diorama for a lightbased type of scene representation device, that featured two immense paintings andvarious interchangeable moving elements, such as human figures, animals or fictionalcharacters, lit from the front and through the back inside an otherwise pitch black,rotating auditorium. The Diorama combined techniques of opaque and translucentpainting and manipulating light in a live spectacle, with color contrast and perspectivetechniques, to produce a plausible scene with accurate spatial representation. In alarge scale version, the Diorama frame could reach 10 meter wide and featureinterposing translucent elements of different opacities and colors, actuated by means ofropes that affect the color tone of selective parts of the frame, for example, to simulateintense fog or the shining sun [26]. A few decades later, the German theater directorand producer Erwin Piscator developed the most consistent scenography experimentsusing the technological possibilities of the early 20th century to support epic theaterworks emphasizing the socio political content of drama. Notably, his early productionSep 19, 2019 4/14 t¨urmflut from 1926 the stage was composed by a translucent background screen in asolid black frame with variable aperture serving as background displaying supportingimages from four retro projectors for performing artists.
Performatic spacetime
Although most of the works of Impressionist painters such as Renoir, Sisley, Monetand Pissarro and Post Impressionists such as Van Gogh, Gauguin and Seurat alsodepended on these mimetic spatial representation techniques, a significanttransformation took place in the broad scientific and artistic domains at the end of thenineteenth century. It challenged the absolute centralized perspective standard, inwhich the images orbited around the static spectator’s eye, increasingly giving way tothe development of the multiple viewpoints systems, which aimed to capture timelapses in which the scene should be considered not as static but in motion. The idea ofmultiplicity in viewpoints was mainly developed in the arts by the Cubist movement,fundamentally Picasso and Braque in their invention of techniques such as faceting,passage and multiple perspective. This type of representations of multiple viewpointsin space and time, fused in one single image, consolidated as one of the movement’smost acclaimed traits. Importantly, these innovations may be interpreted as themovements’ critical philosophical stance regarding perspective and the capacity ofhuman beings to faithfully grasp and represent reality. For the Cubists, art would notserve to be a mimesis of reality, as an objective mirror snapshot of what is there , butwould serve to represent things as experienced in space and time, a subjectivesnapshot of what is sensed . This new method aimed to account for change and flux ina scene, and create a performative situation in which the interpretative role of thespectator becomes central to the evocative power of the Cubist’s images [27].Following these lines, Albert Gleizes and Jean Metzinger in the 1912 book
Du”Cubisme” , the first major text on Cubism, explicitly related the pursue of multipleperspective to an attempt to understand time as a continuum. This may beinterpreted as giving symbolic expression to the notion of duration proposed by thephilosopher Henri Bergson, according to which, life is subjectively experienced as acontinuum, with the past flowing into the present and the present merging into thefuture [28, 29]. Early Futurist paintings by Umberto Boccioni, Giacomo Balla or GinoSeverini where also highly influenced by this search for multiple perspective andsimultaneity. This technique of representing multiple viewpoints in space and time isadvanced in complexity and scale in Gleizes’
Le D´epiquage des Moissons , displayed atthe 1912
Salon de la Section d’Or , and Robert Delaunay’s City of Paris, exhibited atthe
Ind´ependants in 1912. As the invention of linear perspective in fifteenth centuryRenaissance was a major manifestation of the establishment of the stronganthropocentric interpretation of the world, in which the whole of its Science andPhilosophy was founded upon, the multiple perspective paradigm that arrived at thedawn of the 20th century marked the ultimate disenchantment of human as themeasure of all things, welcoming novel ideas from Physics regarding the nature ofspace and time, as well as the atom and atomic dynamics, which ultimately favoredmany of the ideas that shaped the following decades: The relative over the absolute,multiplicity over centralization, dynamic co creation over pre determination [30].Whereas Renaissance artists strove to represent the real by offering viewers anarchetypical perceptual illusion, Cubist illusionism subverts this kind of realisticrepresentation by calling attention to its own artifice, to its own perspectivalmanipulations, and thus to the problematic nature of representation and illusion.Sep 19, 2019 5/14 xpanded cinema
During the 1950’s The Czech artist Josef Svoboda pursued a type of non verbaltheatrical format where actors perform in harmony along with moving imagesprojected by a set of Magic Lanterns and elements of the scenography that supportedthe performative meaning production. This poetic synthesis of traditional methodswith technical innovation including architecture, dance, cinema and theater, that in asense revived Richard Wagner’s
Gesamtkunstwerk , secured Czechoslovakia’s best scorein the Brussels World Exhibition of 1958. Svoboda not only mastered the classicalperspective and color contrast techniques to put adequate spatial illusion in action onthe stage, but also multiplied its evocative power by profiting from the emerginginteractions between the characteristics of the selected physical elements, theperforming actors and the content of the projected images in the composition [31].Moreover, part of this new theatrical format was Svoboda’s
Polyekran , a multiprojection system, consisting of screens of various shapes and dimensions suspendedwith steel cables in arbitrary angulations on a black background. Eight projectors ofslides and seven movie projectors, along with synchronized stereo helped completingthis instrument in service of the total work of art.The immersive quality of spatially distributed images was also among the interestsof the experimental filmmaker Stan VanDerBeek , who between 1963 and 1965developed his
Movie Drome , a type of semi spherical theater where people would liedown and experience movies all around them. Notably, influenced by BuckminsterFuller’s dome constructions and Robert Barker’s Panorama public installations,VanDerBeek envisioned the Movie Drome as a prototype for a global two waycommunication system consisting of a network of Movie Dromes linked to orbitingsatellites that would store and transmit images between them. Although far from fullyaccomplishing it vision, Movie Dromes did put in place a new set of challenges for thetraditional use of perspective and linear narratives to represent reality. Instead,VanDerBeek presented a multiple perspective design, that achieved multiple gazes overmultiple simultaneous events, resulting in a fragmented non linear narrative, aspatially distributed moving image collage [32, 33]. Distinct from traditional cinema,this new type of experience sometimes characterized as
Expanded Cinema [34] had aclear emphases on breaking the cinematography from its linear and concentratedconstraints to give way to a non linear, distributed in space, multi sensory experience,aimed at a type of audience engagement that anticipates contemporary art’sinteractivity and participatory practices. Audience engagement produced by carefullycontrolled perceptual illusions was also researched for popular entertainment purposes.As a proof, on 9 august 1969 the doors to the
Haunted Mansion spectacle atDisneyland opened to the public, featuring a number of light based perceptualillusions placed inside a three story Victorian house structure built in the New OrleansSquare in Disney’s Adventure Park in California. The project gathered the works ofartists and storytellers Yale Gracey, Marc Davis, Claude Coats, Xavier Atencio andamong those, Rolly Crump’s innovative
Singing Busts installation [35]. This workconsisted of five singing busts, brougth to life by the
Grim Grinning Ghosts actors,pre recorded in 16mm film singing the theme song of the ride, and then projected ontothe plaster busts modeled after each character. By controlling the lighting andcontrast of the scene, together with the predetermined path and vantage point ofuser’s, the designers could obtain an unprecedented multi sensory illusion that may beconsidered as direct precursor of current virtual and augmented reality paradigms.Since the mid 1980s, numerous examples exist of re enactment of cinematographicscenes by the use of video sequences in combination with immersive objects andarchitectures. Among those the works of Tony Oursler who as one of the key figures inthe development of video art experimented early on with the moving image thatSep 19, 2019 6/14xtended beyond the borders of the TV monitor [36]. Oursler used projections onsculptures and specially crafted scenographic installations. His early 1990s dolls anddummy works or
Troubler from 1996, rely entirely in the spatial and semanticcorrespondence between projected film and dolls which are stronglyanthropomorphized [37]. Using centuries old knowledge on optics and perception,controlling light and shadow and the spectator’s point of view Oursler mastered aunique type of cinematographic composition based on the combination of foundobjects, life-like moving faces and archetypal elements coming into life by virtue of thesuperimposed projected narratives [38].Michael Naimark, is one key multimedia artist and researcher who has alsoexplored spatialized representation. His
Displacements intallation, developed from1980 to 1984, further advanced the spatial and semantic interactions between imagesand surfaces, and is considered among the most important seminal works in thisdomain. An immersive installation that projected a pre recorded 16mm film on anarchetypal living room completely covered in white paint, matching the spatiallocation of the projected images with selected objects of the living room. Theprojector located on a rotating turntable at the room’s center slowly directed thespectators’ gaze towards the unwrapping narrative, with the appearance of thecharacters and objects as three dimensional. Artist Jeffrey Shaw has also undertakensince the late 1960s, research on types of Expanded Cinema and the development ofvarious innovative spatial illusion strategies. For example
Corpocinema from 1967,involved projection into a dome, and the Diadrama from 1974 was projected onto a180 degree screen surrounding the audience.In its theatrical form, an example of the combination of ancient perspective andcontrast techniques with contemporary spatialized narratives to bring threedimensional realism to life is the opera
Le Grande Macabre by Fura dels Baus.Premiered in Brussels in March 2009, the company made use of the correspondence ofprojected images and physical objects, profiting from the emerging frictions betweenthe projected images and a large scale anthropomorphic figure that mutates in identityand function within the play. The proposal of Fura dels Baus intends to avoid mimesis,and rather than repeat reality, aims at constructing a type of illusory liveexperience [39]. Several remarkable examples are found in the works of Robert Lepage,and among them,
Needles and Opium , which narratively relies on an strongengagement with the space, thus the need for a careful representation of location andcontext within a semi cubic structure that serves as a group of screens that mutateaccording to the needs of each scene [40]. The use of these types of neutral spatialcavities that serve as immersive screens has grown in popularity during the lastdecade. In 2012, as part of a marketing campaign for the Sony Playstation products,the agency Marshmallow Laser Feast presented the video mapping of an entire livingroom using choreographed live puppetry, controlling every element in the scene andperformed in front of a non stop recording camera. The resulting recordedperformances offered a high degree of illusion while consolidating the archetypicalcentralized and predetermined viewpoint, in this case, the viewpoint of the non stoprecording camera.
Interactive virtuality
The advent and popularization of the interactive computing technologies, a variety ofspatialized works of art speculate in the direction of producing full body perceptualillusions no longer tied to the spectator’s spatial positions and viewpointspredetermined beforehand, but affording contents that interact with the user, whosethree dimensional representations are dynamically adjusted to the view point of theuser in real time [41–43].Sep 19, 2019 7/14rom this point on, a significant part of research from academic and industryrelated fields, coincided in the aim for developing techniques and devices for visuallycompositing virtual objects within real environments, with ever increasing interactivefaculties for spectators to become co participants. A significant advance was made inthis regard by Carolina Cruz Neira, Daniel J. Sandin, and Thomas A. DeFanti at theUniversity of Illinois, Chicago Electronic Visualization Laboratory in 1992. Theypresented the notion of
CAVE acronym for
Cave automatic virtual environment , asemi cubic installation made up of rear high resolution projection screen walls andfloor, and a motion capture system that allowed users to interact by physicalmovements [44–46]. In 1995 a team of researchers from The University of NorthCarolina at Chapel Hill, including Ramesh Raskar, Greg Welch, and Henry Fuchspresented The office of the future a type of CAVE application not supported on retroprojected surfaces but projection on real office surfaces and objects, moreoverincorporated real time computer vision techniques to dynamically extract per pixeldepth and reflectance information for the visible surfaces in the office including walls,furniture, objects, and people, and then to either project images on the surfaces,render images of the surfaces, or interpret changes in the surfaces [47–49].In 2006 Shaw initiated research into a fully 360 degree 3D projection system at theUNSW iCinema Research Centre, a centre directed by Shaw and co founded withDennis Del Favero. AVIE
Advanced Visualisation and Interaction Environment wasfirst installed in the UNSW Scientia building in 2003 and presented a paradigm thatallowed the exploration of 360 degree panoramic projected images, within anarchitectural framework that correlated the design of the virtual landscape with thatof the installation itself, thus simultaneously co activating the virtual representationsand the real materials of the projection space.This installation also introduced a fundamental level of user control, allowing theviewer to control the rotation of a projected window. The design of AVIE follows thetraditions of the Panorama, as well as the search for embodied impact of the BaroqueTromp l’oeil fresco painting techniques [50, 51]. One remarkable implementationexample is
The Pure Land: Inside the Mogao Grottoes , a project by Sarah Kenderdineand Jeffrey Shaw working in the University City Honk Kong in 2012. Usingdigitization data from the Mogao Grottoes in China this installation emphasizes thespatial illusion via panoramic immersion and coherent image display techniques [52].The contents are staged in a 10 meter diameter by 4 meter high cylindrical AVIEtheater, while a handheld interface provides interaction with the rendered images,allowing the user to volitionally reveal key parts of the mural paintings on its walls.These real scale interactive visual display systems allowed the research on thedomain of the spatial illusion of reality and helped set the foundations for projectorbased
Augmented Reality in the following decades. As defined, Augmented realityaims at building systems that enhances the real world by superimposing computergenerated information on top of it, in one or more sensory modalities [53–56]. In itsprojector based form, Augmented Reality shares the aim to generate perceptualillusion untethered and available to the plain senses, by using synthetic informationdisplayed on real world locations, but distinguishing from mere spatialized motionimage in the strong interactive component of the experience [57, 58].
Living virtuality
In the last decades one of the most interesting objects for projector based syntheticaugmentation has been the human body. In a collaboration between 1998 and 2002choreographer Chris Haring and Vienna based director and composer KlausObermaier, created two multimedia dance performances, D.A.V.E and Vivisector, thatSep 19, 2019 8/14ave used the dancer’s body as the primary projection surface pre choreographedprojections on performers skins that led spectators to believe that the bodies ofperformers were dramatically reshaped and altered in real time [59–61]. In 2004together with Ars Electronica Futurelab, Obermaier developed and premieredApparition, an interactive dance and media performance featuring Desir´ee Kongerødand Rob Tannion whose movements on stage are captured using an infrared cameraand suited with adequate images generated in realtime.At the 8th IEEE International Symposium on Mixed and Augmented RealityISMAR in 2009, a team of researchers including Greg Welch, Henry Fuchs from TheUniversity of North Carolina at Chapel Hill, introduced Shader Lamps Avatars as anew approach for creating a video projected three dimensional avatar using faces ofreal people. Based on cameras and projectors to capture and map the dynamic motionand appearance of a real person onto a humanoid life sized styrofoam head mountedon an animatronic device, the system delivers a dynamic, real time representation of aperson to multiple viewers [62, 63]. In a similar quest, a collaboration betweenJapanese media artist Nobumichi Asai, make up artist Hiroto Kuwahara and digitalimage engineer Paul Lacroix resulted in the Omote project in 2014, using projectionmapping techniques to put make up and effects onto a model’s face in real time. Byusing a set of infrared markers painted on the model’s face, the team was able tocapture and process marker position, estimating face position and orientation,rendering face CG model with animated texture upon, and finally send the imagethrough the projector. This face changing system brings to life a fluid stance ofidentity in a way similar to the ancient Bian Lian techniques that allow SichuanOpera’s performers to unperceivably change masks in front of spectators. Furtheradvancing these achievements, The Ichikawa Senoo Laboratory from the University ofTokio aimed in 2015 at overcoming the limited static or quasi static conditionsrequired for video projection mapping. A dynamic projection mapping system wasdeveloped, capable of adjusting images in real time onto deforming non rigid surfaces.To this end, the team led by Gaku Narita, Yoshihiro Watanabe, and MasatoshiIshikawa developed two set of technologies addressing the system’s mainchallenges [64, 65]. First,
Deformable Dot Cluster Marker , a robust infrared acquisitionmethod for capturing deformations from the surface, even capable of handling largedeformation and occlusions. Second, a high speed projector system was developed,
DynaFlash , capable of displaying 8 bit images in frame rates up to 1,000fps withunprecedented 3 milliseconds latencies. These two innovations mark a breakpoint inthe quest for spatially distributed illusion, achieving the perceptual effect as if theprojected images were printed onto the target surface. The Ichikawa Senoo Laboratoryfurther demonstrated in 2016 their dynamic projection mapping systems actuatingonto a deformed sheet of paper and T shirt on one or multiple targets. Consideringthe possibility for embedding active or passive markers in clothing and everydayobjects [66], these advancements could represent the novel technical feasibility formost surfaces in the real world to become screens, displaying perceptual illusionscoherently and seamlessly [67–70].
Conclusions
Video projection mapping has been an important step in the evolution of spatiallybased perceptual illusionism, and on the base of the evolution of its associatedtechniques and idiosyncrasies a great variety of research lines have been developed, aswell as entertainment and artistic applications. Having mastered linear perspective,color contrast, and vantage point control in order to produce static realistic spatialthree dimensional representations, in recent years video projection mapping systemsSep 19, 2019 9/14aired with computer vision systems have acquired the capability of user interactionand of dynamically adjusting the spatial characteristics of the images in order toachieve almost perfect correspondence between projected contents and physical space.Considering video projection mapping as an intermediate stance in the reality -virtuality continuum [71], it has been shown to produce at least two types ofobservable results that occur almost simultaneously [72, 73]. The first occurs at asensory level, in which the shape, color and texture properties of the physical surfacesare incorporated by the superimposed images or wane in its benefit. For example thevisitors of the Camera degli Sposi are deceived into believing that a two dimensionalsolid surface such as a brick wall disappears to give way to a different landscape inperfect harmony with the existing architecture. The face in the Omote’s models canbe suddenly full of make up, and the performers in Obermaier’s works may seem to besuddenly in raw flesh. The second observable phenomena concerns the semantic levelof interaction between imaginary contents and physical objects. Depending on thesemantic load of each of the interacting elements, the merger between superimposedimages and material supports may yield a varying degree of identity transference, assuch, physical objects may contrast with projected images or completely assume itsidentity and its network of semantic associations. In these semantic agreement andopposition relationships, the visitors of the Camera degli Sposi may fall under theimpression of being at the residence of a powerful dynasty whose decorated oculusattracts angels from the skies. The models in Omote project can suddenly appear tobe skeletons or android humanoids, or it may result unimaginable to think on thebodies in Obermaier’s Vivisector as human due to their unprecedented flexibility.Many steps have been taken in making the real world wane in favor of syntheticsuperpositions that increasingly become more present, in that every surface acts as ascreen and every object contains potential of fluid identity and meaning hosted in thedomain of virtual representations. As the production of perceptual illusions appear tobe in full effect, new possibilities arise for new entertainment paradigms, spatiallydistributed learning applications, or massive dataset exploration tools that profit fromthe embodied cognition that results from spatial interaction with digitalcontents [ ? , 74]. Importantly, the relentless idea that fueled the quest for perceptualillusion and pervaded across centuries of artistic and scientific experimentation, seemsto gain momentum at the current juncture in history: reality does not satisfy desire. Acknowledgments
Support provided Virtual Sense Systems and Hangar Centre for artistic research andproduction in Barcelona was greatly appreciated.