Perspective picture from Visual Sphere: a new approach to image rasterization
PPerspective picture from Visual Sphere: a new approach to image rasterization
Jakub Maximilian Fober ∗ October 20, 2020 → Figure 1:
Example of aliasing-free, single-pass rasterization of polygon quad in real-time, from perspective vector map, where Ω 𝑑 = 270 °, 𝑘 = 0.32 , 𝑙 = 62% , 𝑠 = 86% . Abstract
In this paper alternative method for real-time 3D model rasterizationis given. Surfaces are drawn in perspective-map space which acts as avirtual camera lens. It can render single-pass 360° angle of view (AOV)image of unlimited shape, view-directions count and unrestrained projec-tion geometry (e.g. direct lens distortion, projection mapping, curvilinearperspective), natively aliasing-free. In conjunction to perspective vectormap, visual-sphere perspective model is proposed. A model capable ofcombining pictures from sources previously incompatible, like fish-eyecamera and wide-angle lens picture. More so, method is proposed formeasurement and simulation of a real optical system variable no-parallaxpoint (NPP). This study also explores philosophical and historical aspectsof picture perception and presents a guide for perspective design. ∗ [email protected] ∗ https://maxfober.space ∗ https://orcid.org/0000-0003-0414-4223 a r X i v : . [ c s . G R ] O c t ntroduction There is a great demand for perspective projection model able to produce computer-generated (CG) image up to 360° of view, with lens-distortions, directly fromthree-dimensional space, to pixel data. Currently there is no practical directmethod for rasterization of real-time graphics in curvilinear perspective. Everyreal-time perspective imagery incorporates
Pinhole Camera model as a base,some with additional layers of distortion on top. Also knowledge about relation-ship between motion and perspective has not been properly formulated, leavingvoid in that field of image science. This paper aims at solving those issues. Studypresents new, universal model for perspective projection and rasterization inCG graphics. It also explores history of perspective picture, redefines abstracttheorem of image (as recorded in common-knowledge) and establishes rules forpicture’s perspective design.This paper begins with a brief introduction to the visual-sphere perspectiveand jumps right into technical details of rasterization. If one is not familiar withthe topic or terms,
Appendix on page 27 gives great introduction and presentssome basic knowledge on picture, perspective and perception, along with briefhistory of the topic. If one is familiar with this work and looks for some reference,Section 2 on page 16 presents algorithms of various perspective maps, used invisual-sphere rasterization. Within that, subsection 2.1 on page 16 presents
Universal Perspective model. It describes variable image geometry in commonprojections. Variable (or floating) no-parallax point of real optical systems isdescribed in section 3 on page 26. Additional figures are presented at the end ofthe document. At the very end of this document are code listings in GLSL formost of presented equations.If one is looking for some artistic view on the topic, I recommend
Conclusion on page 36 and some content from
Appendix with Chapter’s 1 introduction.I hope that this work will bring a new breath into image production, openminds and possibilities for creative action and thinking.2 ontents eferences 42License notice 45 List of Figures
Listings
From Visual Pyramid to Visual Sphere
Figure 2: Projection of triangle
𝐴𝐵𝐶 onto visual unit sphere 𝑆 , where projectionorigin 𝑂 is at the sphere center. Edge of the projected triangle is always producedby an arc of a great circle, here 𝐴𝐶 ↦ 𝑏 ∈ ℎ . Grid 𝐺 represents visual spherevector map, where each pixel color is a spherical unit vector.Visual pyramid of Alberti theorem is by definition restricted to acute angles,which is limiting in terms of describable projections. This property makesstitching or layering multiple pictures defined in such space a problematictask. In the standard perspective model, introduced by L.B. Alberti, 3D pointposition is transformed into 2D plane/screen coordinates. Noticeably in somecurvilinear projections, points are stretched into lines (see example 2 on page 17),making single visible point appear in multiple spots on picture plane/screen.This makes linear 3D ↦
2D point position mapping insufficient.In the proposed visual-sphere model, every visible point in perspective pic-ture’s space has its own spherical position coordinate. Thus single point of avisual sphere can occupy multiple places in the picture, which conforms to thisprinciple of curvilinear perspective. Such perspective space (a sphere) allowsfor stitching, layering and mapping of images in any single-point projection andperspective geometry.Points in spherical projection model are no longer transformed into screenspace. Rather lines are calculated and combined forming a polygon image (seefigure 2). Straight line projected through the center point 𝑂 will always forman arc of a great circle ℎ . Such great circle can be mapped onto visual spheredescribed by incident vector map (later referred as perspective map ). This processinvolves rotating the perspective map vector data. Goal is to align one of theaxis of the map 𝐺 with a great circle forming polygon edge (see figure 2). Oneway to rotate axis component is to calculate dot product between the perspective6ector map 𝐺 and a unit vector perpendicular to points forming polygon edge,like ‖ ⃗𝐶 × ⃗𝐴‖ and the great circle ℎ .This process is essentially a half-space rasterization technique, extendedto spherical space. 𝐴𝐵𝐶 polygon triangle using ̂𝐺 vector fromperspective map Projected polygon geometry is always part of a great circle. The goal of the algo-rithm is to rasterize polygon shape formed by those spherical lines. Rasterizationprocess involves determining orientation of the great-circle. Then rotating per-spective map, so that the vertical axis of the map aligns with a great-circle. Nextthe step-function is performed on a ⃗𝐺 ′𝑖 component of the rotated vector map 𝐺 ′ .The step function algorithm is essential for aliasing-free edge rasterization. Full 𝑛 -sided convex polygon picture is defined by intersection of 𝑛 -number of suchoperations. ⎡⎢⎢⎣ ⃗𝐺 ′1 ⃗𝐺 ′2 ⃗𝐺 ′3 ⎤⎥⎥⎦ = ⎡⎢⎢⎣ ̂𝐺 ⋅ ‖ ⃗𝐴 × ⃗𝐵‖̂𝐺 ⋅ ‖ ⃗𝐵 × ⃗𝐶‖̂𝐺 ⋅ ‖ ⃗𝐶 × ⃗𝐴‖⎤⎥⎥⎦ (1a) 𝐴𝐵𝐶 ↦ 𝐺 = min { step ( ⃗𝐺 ′1 ), step ( ⃗𝐺 ′2 ), step ( ⃗𝐺 ′3 )} ◾ (1b)Rotation of perspective map vector ̂𝐺 is performed with a dot product betweeneach ̂𝐺 and rotation-direction vector. Which is a dot product between perspectivemap 𝐺 and normalized cross product of two edge points. Rotation vector doesn’thave to be normalized, but that could cause some precision errors, especiallywhen using aliasing-free step function. (a) 𝑅𝐺𝐵 as 𝑥𝑦𝑧 coor-dinates ← -1 10 (b) Unit vector- 𝑥 -110 (c) Unit vector- 𝑦 -10 (d) Unit vector- 𝑧 Figure 3: Visual sphere in vector coordinates as seen from outside, facing 𝑍 direction. Each vector component 𝑥 , 𝑦 and 𝑧 can be rotated independently.Visual sphere perspective model is ultimately a cosine -based perspective modeland can produce pictures in both tangent perspective and sine perspective . As seenin figure 3, each sphere surface point has a value associated to its position. Those7hree values also represent cosine of an angle between axis vectors [1 0 0] , [0 1 0] , [0 0 1] and the surface position vector. After rotation of ̂𝐺 , thosethree axes are represented by rotation matrix vectors. Step function is performed on rotated ⃗𝐺 ′ coordinate and determines inside andoutside of a polygon for each edge. Aliasing-free result can be achieved bychanging width of the step-function slope from binary, to one-pixel wide (seefigure 4 on the following page). Below presented are two algorithms for a stepfunction. pstep (𝑔) = { 𝑔𝜕(𝑔) + 12 } ∩ [0, 1] (2a)bstep (𝑔) = {1, if 𝑔 > 00, otherwise (2b)pstep (𝑔) function ( p standing for pixel ) gives smooth aliasing-free result. Thebstep (𝑔) function is a binary operation and gives aliased output. 𝜕(𝑥) functionis an equivalent to fwidth (𝑥) GLSL function. ∕ offset centers anti-aliasinggradient at the polygon edge (as seen in subfigure 4c compared to subfigure 4aon the next page). Global delta 𝛿 can replace denominator 𝜕(𝑔) in pstep () function. It givesmore natural look and has smaller computational footprint. Global delta 𝛿 isderived from perspective vector map ̂𝐺 . Its value can be preprocessed and savedin a texture. The difference between preprocessed delta 𝛿 and per-fragment 𝜕 isin polygon-edge slope width. 𝛿 = 1max {𝜕( ̂𝐺 𝑥 ), 𝜕( ̂𝐺 𝑦 ), 𝜕( ̂𝐺 𝑧 )} (3a)gpstep (𝑔) = {𝛿𝑔 + 12 } ∩ [0, 1] (3b)The result of 𝛿 calculation is a single-channel texture, which represents recip-rocal of the component-wise maximum fwidth (𝑥) of perspective map vector ̂𝐺 . 8 pixels (a) Gradient before the stepfunction. pixels (b) Gradient divided by itspixel 𝜕 . pixels (c) Pixel gradient clamped to [0, 1] range. Figure 4: Aliasing-free step function process, where 𝑔 is the gradient function. 𝜕 is equivalent of fwidth (𝑔) GLSL function. Horizontal axis represents pixelposition and vertical, the value of 𝑔 . Remark.
When combining polygons into polygon-strip using pixel-step function,it is important to sum each mask, otherwise visible seam may occur. See moreon combining fragments into buffer in subsection 1.4 on page 15.
In special case, when projected polygon’s edges meet at very shallow angle, cor-ners will extend beyond polygon outline (due to half-pixel offset in the pstep (𝑔) function). This visual artifact can be corrected by a miter mask. There are manyways to calculate such mask, one is to define the smallest circle over projectedtriangle. Following algorithm uses barycentric coordinates to determine thesmallest circle’s center and size. ⎧⎨⎩𝑎 = | ̂𝐵 − ̂𝐶| = ( ̂𝐵 − ̂𝐶) ⋅ ( ̂𝐵 − ̂𝐶)𝑏 = | ̂𝐶 − ̂𝐴| = ( ̂𝐶 − ̂𝐴) ⋅ ( ̂𝐶 − ̂𝐴)𝑐 = | ̂𝐴 − ̂𝐵| = ( ̂𝐴 − ̂𝐵) ⋅ ( ̂𝐴 − ̂𝐵) (4a) ⎡⎢⎢⎣ ⃗𝑂 𝑠 ⃗𝑂 𝑡 ⃗𝑂 𝑝 ⎤⎥⎥⎦ = ⎡⎢⎢⎣𝑎 (𝑏 + 𝑐 − 𝑎 )𝑏 (𝑐 + 𝑎 − 𝑏 )𝑐 (𝑎 + 𝑏 − 𝑐 )⎤⎥⎥⎦ (4b) ⃗𝑆 = ⎧⎪⎪⎨⎪⎪⎩0.5( ̂𝐵 + ̂𝐶), if ⃗𝑂 𝑠 ≤ 00.5( ̂𝐶 + ̂𝐴), if ⃗𝑂 𝑡 ≤ 00.5( ̂𝐴 + ̂𝐵), if ⃗𝑂 𝑝 ≤ 0 ⃗𝑂 𝑠 ̂𝐴+ ⃗𝑂 𝑡 ̂𝐵+ ⃗𝑂 𝑝 ̂𝐶⃗𝑂 𝑠 + ⃗𝑂 𝑡 + ⃗𝑂 𝑝 , otherwise ◾ (4c)9here ⃗𝑂 is the barycentric coordinate of circumcenter. 𝑎 , 𝑏 , 𝑐 are squaredlengths of projected triangle’s edges. ⃗𝑆 is the smallest-circle center vector whichlength is equal to the cosine of an angle between ̂𝑆 and the smallest-circle rim.Polygon triangle is degenerate if 𝑠 + ⃗𝑂 𝑡 + ⃗𝑂 𝑝 , meaning all projected ̂𝐴, ̂𝐵, ̂𝐶 points lay in one line. In such case miter mask can be omitted.Having the smallest circle center vector ⃗𝑆 , rasterization algorithm can be updatedas follows: ⎡⎢⎢⎢⎢⎣ ⃗𝐺 ′1 ⃗𝐺 ′2 ⃗𝐺 ′3 ⃗𝐺 ′4 ⎤⎥⎥⎥⎥⎦ = ⎡⎢⎢⎢⎢⎣ ̂𝐺 ⋅ ‖ ⃗𝐴 × ⃗𝐵‖̂𝐺 ⋅ ‖ ⃗𝐵 × ⃗𝐶‖̂𝐺 ⋅ ‖ ⃗𝐶 × ⃗𝐴‖̂𝐺 ⋅ ̂𝑆 − | ⃗𝑆| ⎤⎥⎥⎥⎥⎦ (5a) 𝐴𝐵𝐶 ↦ 𝐺 = min { pstep ( ⃗𝐺 ′1 ), pstep ( ⃗𝐺 ′2 ), pstep ( ⃗𝐺 ′3 ), pstep ( ⃗𝐺 ′4 )} ◾ (5b)Figure 5: Rasterization masks as RGB values, where perspective map represents Ω 𝑑 = 270 °, 𝑘 = 0.32 , 𝑙 = 62% , 𝑠 = 86% . Since rasterization involves multiple dot products (equation 5a), part of theprocess can be mitigated to matrix multiplication, where each matrix row repre-sents rotation vector. Such matrix is calculated once per polygon–per frame andis executed per fragment pixel. Full vertex rasterization process is presented onfigure 19 on page 39. ⎡⎢⎢⎢⎢⎣ ⃗𝐺 ′1 ⃗𝐺 ′2 ⃗𝐺 ′3 ⃗𝐺 ′4 ⎤⎥⎥⎥⎥⎦ = ⎡⎢⎢⎣ ̂𝐺 𝑥 ̂𝐺 𝑦 ̂𝐺 𝑧 ⎤⎥⎥⎦ ⎡⎢⎢⎢⎢⎣‖ ⃗𝐴 × ⃗𝐵‖‖ ⃗𝐵 × ⃗𝐶‖‖ ⃗𝐶 × ⃗𝐴‖̂𝑆 𝑥 ̂𝑆 𝑦 ̂𝑆 𝑧 ⎤⎥⎥⎥⎥⎦ − ⎡⎢⎢⎢⎣ 000| ⃗𝑆|⎤⎥⎥⎥⎦ (6a) 𝐴𝐵𝐶 ↦ 𝐺 = min { pstep ( ⃗𝐺 ′1 ), pstep ( ⃗𝐺 ′2 ), pstep ( ⃗𝐺 ′3 ), pstep ( ⃗𝐺 ′4 )} ◾ (6b)10his procedure can be expanded to any convex 𝑛 -sided planar polygon usingprocedural equation as follows: ⎡⎢⎢⎢⎢⎣ ⃗𝐺 ′1 ⃗𝐺 ′2 ⋮⃗𝐺 ′𝑛 ⎤⎥⎥⎥⎥⎦ = ⎡⎢⎢⎣ ̂𝐺 𝑥 ̂𝐺 𝑦 ̂𝐺 𝑧 ⎤⎥⎥⎦ ⎡⎢⎢⎢⎣ ‖𝑇 ⃗1 × 𝑇 ⃗2 ‖‖𝑇 ⃗2 × 𝑇 ⃗3 ‖⋮‖𝑇 ⃗𝑛 × 𝑇 ⃗(𝑖+1) mod 𝑛 ‖⎤⎥⎥⎥⎦ (7a)mask ( ⃗𝐺 ′ ) = min { step ( ⃗𝐺 ′1 ), step ( ⃗𝐺 ′2 ), … , step ( ⃗𝐺 ′𝑛 )} ◾ (7b)Here 𝑇 describes 𝑛 -sided polygon points 𝑛 × 3 matrix, where each row is avertex position (counting clock-wise). Each rotation vector is derived from crossproduct between 𝑖 -vertex and 𝑖 + 1 . For the the last 𝑛 th vertex, next one is №1. Rendering realistic polygon graphics involves shading and texture mapping.Values of normal, depth and UV coordinates associated to each vertex are in-terpolated across polygon surface using barycentric coordinate of the fragmentpoint.Figure 6: Barycentric coordinates as RGB values. Here perspective map repre-sents Ω 𝑑 = 270 °, 𝑘 = 0.32 , 𝑙 = 62% , 𝑠 = 86% . ⃗𝑁 = ⎧⎨⎩( ⃗𝐴 − ⃗𝐵) × ( ⃗𝐶 − ⃗𝐵)( ⃗𝐵 − ⃗𝐶) × ( ⃗𝐴 − ⃗𝐶)( ⃗𝐶 − ⃗𝐴) × ( ⃗𝐵 − ⃗𝐴) (8)Normal vector ⃗𝑁 ∈ ℝ of the triangle plane 𝐴𝐵𝐶 is derived from cross productbetween two triangle edges. Length of this vector is equal to the area of aparallelogram formed by those two edges, which is equal to double the area of11riangle
𝐴𝐵𝐶 . 𝑟 = ⃗𝐴 ⋅ ⃗𝑁̂𝐺 ⋅ ⃗𝑁 = ⃗𝐵 ⋅ ⃗𝑁̂𝐺 ⋅ ⃗𝑁 = ⃗𝐶 ⋅ ⃗𝑁̂𝐺 ⋅ ⃗𝑁 (9a) ≡ | ̂𝐺 → 𝐴𝐵𝐶| (9b)Distance 𝑟 represents multiplier of the visual sphere vector ̂𝐺 , to intersectionpoint on the 𝐴𝐵𝐶 triangle plane. Since ̂𝐺 is a unit vector, value 𝑟 can be exportedas depth (representing distance, not 𝑧 –position). Here vector ⃗𝐴 , ⃗𝐵 , ⃗𝐶 in thenumerator can be replaced by any point on the triangle plane. ⎡⎢⎢⎣ ⃗𝑏 𝑠 ⃗𝑏 𝑡 ⃗𝑏 𝑝 ⎤⎥⎥⎦ = ⎡⎢⎢⎣|( ⃗𝐵 − 𝑟 ̂𝐺) × ( ⃗𝐶 − 𝑟 ̂𝐺)||( ⃗𝐶 − 𝑟 ̂𝐺) × ( ⃗𝐴 − 𝑟 ̂𝐺)||( ⃗𝐴 − 𝑟 ̂𝐺) × ( ⃗𝐵 − 𝑟 ̂𝐺)|⎤⎥⎥⎦ ÷ | ⃗𝑁| (10a)Since ⃗𝐴 ⋅ ⃗𝐵 = | ⃗𝐴|| ⃗𝐵| cos 𝛼 and 𝐴𝐵 ⊥ ( ⃗𝐴 × ⃗𝐵) this equation can be rewrittenusing triple product, as follows. ⎡⎢⎢⎣ ⃗𝑏 𝑠 ⃗𝑏 𝑡 ⃗𝑏 𝑝 ⎤⎥⎥⎦ = ⎡⎢⎢⎣( ⃗𝐶 − 𝑟 ̂𝐺) × ( ⃗𝐵 − 𝑟 ̂𝐺) ⋅ ⃗𝑁( ⃗𝐴 − 𝑟 ̂𝐺) × ( ⃗𝐶 − 𝑟 ̂𝐺) ⋅ ⃗𝑁( ⃗𝐵 − 𝑟 ̂𝐺) × ( ⃗𝐴 − 𝑟 ̂𝐺) ⋅ ⃗𝑁⎤⎥⎥⎦ ÷ ( ⃗𝑁 ⋅ ⃗𝑁) (11a)Barycentric vector ⃗𝑏 is a proportion of surface area. From vector ⃗𝑏 , vari-ous vertex properties are interpolated (e.g. depth, normal vector and texturecoordinates), given each vertex 𝐴 , 𝐵 and 𝐶 has an associated value. 𝑓 𝑟 = 𝑟 (12a) 𝑓 ̂𝑁 = ‖ ⃗𝑏 𝑠 𝐴 ̂𝑁 + ⃗𝑏 𝑡 𝐵 ̂𝑁 + ⃗𝑏 𝑝 𝐶 ̂𝑁 ‖ (12b) 𝑓 ⃗𝑓 = ⃗𝑏 𝑠 𝐴 ⃗𝑓 + ⃗𝑏 𝑡 𝐵 ⃗𝑓 + ⃗𝑏 𝑝 𝐶 ⃗𝑓 (12c)Interpolation of fragment data 𝑓 is done through a dot product between barycen-tric coordinate vector ⃗𝑏 and values associated to each vertex. Here 𝑓 𝑟 is the depthpass (representing distance, not 𝑧 –position), 𝑓 ̂𝑁 is the interpolated normal vec-tor and 𝑓 ⃗𝑓 are the texture coordinates. All interpolations are perspective-correct.Figure 20 on page 40 shows this process in a step-by-step flowchart. Prior to geometry rasterization, pixel perspective shader pass may be performed.Its output works as a perspective vector map. This pipeline addition enablesspecial effects like dynamic perspective and projection mapping, flat mirrorreflection, screen-space refraction (e.g. concave refractive surfaces which expandAOV), etc. 12 xample . Projection mapping with dynamic view-position can be achievedby transforming vector data of world-position-pass texture 𝑆 . Knowing viewerposition ⃗𝑂 , offset can be applied to ⃗𝑆 . When normalized, ̂𝑆 ′ produces perspectivemap vector ̂𝐺 = ‖ ⃗𝑆 − ⃗𝑂‖ . Complexity of projection surface and number ofviews (used projectors) is outside of concern, as view-position transformation isperformed on a baked texture. Exact formula for projection mapping is availablein sub-subsection 2.2 on page 20. It is possible to produce screen-relative line drawing based on a perspectivemap. Following algorithm will produce wire-frame image of projected 𝐴𝐵 linesegment. ⃗𝐺 ′𝑧 = ̂𝐺 ⋅ ‖ ⃗𝐴 × ⃗𝐵‖ (13)Perspective vector map component ̂𝐺 𝑧 is rotated by 𝐴𝐵𝑂 tangent vector ‖ ⃗𝐴 × ⃗𝐵‖ ,where ⃗𝑂 is the observation point.blstep ( ⃗𝐺 ′𝑧 ) = {1, if 𝜕( ⃗𝐺 ′𝑧 ) − |2 ⃗𝐺 ′𝑧 | > 00, otherwise (14a)plstep ( ⃗𝐺 ′𝑧 ) = 1 − min ⎧⎨⎩ | ⃗𝐺 ′𝑧 |𝜕( ⃗𝐺 ′𝑧 ) , 1⎫⎬⎭ (14b)Above is a line-step function in two variants, binary-aliased (14a) and pixel-smooth (14b). 𝜕(𝑥) is equivalent to fwidth (𝑥) function of GLSL. ⃗𝐿 = ̂𝐴 + ̂𝐵2 (15a) ℎ = lstep ( ⃗𝐺 ′𝑧 ) (15b) 𝑙 = min { step ( ̂𝐺 ⋅ ̂𝐿 − |⃗𝐿|), ℎ} ◾ (15c)Radial mask combined with great circle ℎ forms 𝐴𝐵 line segment image 𝑙 . ̂𝐿 is the line-middle vector, lstep ( ⃗𝐺 ′𝑧 ) function rasterizes great-circle ℎ . Radialmask is formed by the step (𝑥) function of dot product between perspective mapvector ̂𝐺 and line-middle vector ̂𝐿 , minus |⃗𝐿| = cos 𝜃 ∕ .13 .3 Simple procedural particle rasterization Figure 7: Simple particle model, where 𝑟 is the particle radius and ⃗𝑃 is theparticle position from observation point ⃗𝑂 .Following algorithm will produce mask image of spherical particle, given posi-tion and radius.particle ( ̂𝐺, ⃗𝑃, 𝑟) = step ( ̂𝐺 ⋅ ̂𝑃 − √1 − 𝑟 ÷ ( ⃗𝑃 ⋅ ⃗𝑃)) (16)Where ⃗𝑃 ∈ ℝ is the particle position. 𝑟 is particle radius and ̂𝐺 is the perspec-tive map vector. To obtain texture coordinates within the particle, followingalgorithm can be used. ̂𝑋 = ‖ ⃗𝑃 𝑧 , 0, − ⃗𝑃 𝑥 ‖̂𝑌 = ‖ ̂𝑋 × ⃗𝑃‖ ≡ ̂𝑋 × ̂𝑃 (17a) [ ⃗𝑓 𝑃𝑠 ⃗𝑓 𝑃𝑡 ] = ⎡⎢⎢⎣ ̂𝐺 𝑥 ̂𝐺 𝑦 ̂𝐺 𝑧 ⎤⎥⎥⎦ [ ̂𝑋 𝑥 ̂𝑋 𝑦 ̂𝑋 𝑧 ̂𝑌 𝑥 ̂𝑌 𝑦 ̂𝑌 𝑧 ] | ⃗𝑃|2𝑟 + 12 ◾ (17b)Where ⃗𝑓 𝑃 is the texture coordinate of the particle, ̂𝑋 and ̂𝑌 are rotation matrixvectors. Full particle rasterization process with texture coordinates and roundmask can be described by following algorithm. ⎡⎢⎢⎣ ⃗𝐺 ′1 ⃗𝐺 ′2 ⃗𝐺 ′3 ⎤⎥⎥⎦ = ⎡⎢⎢⎣ ̂𝐺 𝑥 ̂𝐺 𝑦 ̂𝐺 𝑧 ⎤⎥⎥⎦ ⎡⎢⎢⎣ ‖ [ ⃗𝑃 𝑧 𝑥 ] ‖‖ [ ⃗𝑃 𝑧 𝑥 ] ‖ × ̂𝑃̂𝑃 𝑥 ̂𝑃 𝑦 ̂𝑃 𝑧 ⎤⎥⎥⎦ (18a) 𝑚 𝑃 = step ( ⃗𝐺 ′3 − √1 − 𝑟 ÷ ( ⃗𝑃 ⋅ ⃗𝑃)) ◾ (18b) [ ⃗𝑓 𝑃𝑠 ⃗𝑓 𝑃𝑡 ] = [ ⃗𝐺 ′1 ⃗𝐺 ′2 ] | ⃗𝑃|2𝑟 + 12 ◾ (18c)14here ⃗𝑓 𝑃 ∈ [0, 1] are the texture coordinates, 𝑚 𝑃 ∈ [0, 1] is the particle mask, ⃗𝑃 is the particle position with 𝑟 as radius. ̂𝐺 is the perspective-map vector. One of the key features of 3D polygon rasterization is hidden surface occlusion(HSO). There are many approaches that solve HSO problem. One of them isdepth-pass test, where each depth 𝑟 of the fragment pixel is tested against alreadyrasterized geometry depth 𝑟 ⊥ . This simple technique works well with the binarystep function (see sub-subsection 1.1.1 on page 8). Depth pass test techniqueproduces aliased result with some pixels being drawn multiple times during asingle pass.To utilize advantage of aliasing-free rasterization in perspective map rasteri-zation, geometry data should be rasterized in a very organized order. Optimalapproach is to render scene front-to-back and discard already painted pixelsduring polygon mask rasterization. Such ordered rasterization can be achievedwith binary space partitioning (BSP) and concurrent binary trees (CBT).This solution produces aliasing-free result, without depth-buffer check. It paintspixels only once for most of the time. {𝑚 𝑓𝑐 = min{𝑚 𝑓 , 1 − 𝑚 𝑏 }𝑚 = 𝑚 𝑏 + 𝑚 𝑓𝑐 (19)Above equation combines fragment data with the buffer mask in front-to-backrasterization. Where 𝑚 𝑓 is the currently processed fragment black-and-whitemask. 𝑚 𝑏 is the buffer mask of already rasterized geometry. Mask 𝑚 𝑓𝑐 is thefragment mask clipped (occluded) by the buffer mask 𝑚 𝑏 . Simple additioncombines current fragment mask 𝑚 𝑓𝑐 with the buffer 𝑚 𝑏 . Same addition canbe performed for any fragment data, having the 𝑚 𝑓𝑐 as a mask. For example UV coordinates for surface textures (equation 20b), normal pass (equation 20c),depth pass (equation 20a), etc. Initial value of each buffer is equal zero. 𝑟 = 𝑚 𝑓𝑐 𝑟 𝑓 + 𝑟 𝑏 , 𝑟 𝑏 (1) = 0 (20a) ⃗𝑓 = 𝑚 𝑓𝑐 ⃗𝑓 𝑓 + ⃗𝑓 𝑏 , ⃗𝑓 𝑏 (1) = [0 0] T (20b) ̂𝑁 = ‖𝑚 𝑓𝑐 ̂𝑁 𝑓 + ̂𝑁 𝑏 ‖, ⃗𝑁 𝑏 (1) = [0 0 0] T (20c)Where 𝑟 is the depth pass, ⃗𝑓 ∈ [0, 1] represents texture coordinates and ̂𝑁 ∈ [0, 1] is the normal vector. Current fragment mask (occlusion clipped) isdenoted by 𝑚 𝑓𝑐 . “ 𝑓 ” in a superscript indicates current fragment data and “ 𝑏 ”in superscript–buffer data of already rasterized geometry. Initial values of thebuffer are denoted by “ (1) ” in suffix. 15 Perspective transformations of 2D/3D data
In this section presented algorithms produce perspective picture from 3D and 2Ddata. Most of them describe 2D →
3D transformation, which outputs perspectivevector map from texture coordinates, suitable for rasterization.
Remark.
For a proper transformation, 2D coordinates must be normalized for agiven AOV type (e.g. vertical, diagonal or horizontal).
Example.
For a pixel 𝑖 in picture of aspect-ratio 16:9 and horizontal-AOV, coordi-nates (𝑖 𝑥 , 𝑖 𝑦 ) must be centered and horizontally normalized, so that 𝑖 𝑥 ∈ [−1, 1] and 𝑖 𝑦 ∈ [− , ] . [ ⃗𝑓 𝑥 ⃗𝑓 𝑦 ] = (2 [ ⃗𝑓 𝑠 ⃗𝑓 𝑡 ] − 1) ⎧⎪⎪⎪⎨⎪⎪⎪⎩[1 ] T , if Ω horizontal [𝑎 1] T , if Ω vertical [𝑎 1] T ÷ √1 + 𝑎 , if Ω diagonal [𝑎 1] T ÷ , if Ω horizontal 4 × 𝑥, 𝑦 ∈ [−1, 1] from texture coordinates 𝑠, 𝑡 ∈ [0, 1] ,where 𝑎 is the picture aspect ratio and Ω is the AOV. [ ⃗𝑓 𝑠 ⃗𝑓 𝑡 ] = 12 + 12 [ ⃗𝑓 𝑥 ⃗𝑓 𝑦 ] ⎧⎪⎪⎪⎨⎪⎪⎪⎩[1 𝑎] T , if Ω horizontal [ T , if Ω vertical [ T √1 + 𝑎 , if Ω diagonal [ T , if Ω horizontal 4 × 𝑠, 𝑡 ∈ [0, 1] from view coordinates 𝑥, 𝑦 ∈ [−1, 1] ,where 𝑎 is the picture aspect ratio and Ω is the AOV. Universal perspective model allows for a smooth adjustment of image geometryin accordance to the visible content. Presented two transforms produce perspec-tive picture (see definition on page 30). First 2D →
3D transformation producesperspective vector in various common projections, which is suitable for generat-ing universal-perspective maps for rasterization. Second 3D →
2D transformationproduces picture coordinates of various projections. Combination of both canmap between different projections. 16 emark.
Note that 3D →
2D transformation may be non-linear, meaning that a3D vector can be represented by multiple 2D coordinates in very special cases.
Example . In equidistant projection at Ω = 2𝜋 (whole sphere) visible pointopposite to the view direction is represented by a ring at the picture’s boundary.
There is total of four parameters defining perspective picture geometry in theuniversal perspective model. angle Ω defines angle of view (AKA FOV), which varies between → 𝜋 and . scalar 𝑘 defines perspective type as a value in range [−1, 1] , which interpolatesbetween various azimuthal projections: Gnomonic (rectilinear) 𝑘 = 1
Stereographic 𝑘 = 0.5
Equidistant 𝑘 = 0
Equisolid 𝑘 = −0.5
Orthographic 𝑘 = −1 scalar 𝑙 in range [0, 1] defines cylindricity of the projection, where 𝑙 = 0 repre-sents cylindrical projection and 𝑙 = 1 — spherical projection. scalar 𝑠 in range [ ∕ , 1] defines vertical anamorphic correction of a non-sphericalprojection, meaning that it is only active when 𝑙 < 1 .Base projection type is adjusted by the 𝑘 component. It manipulates imageperception. Cylindrical projection, is adjusted by the 𝑙 component. Low 𝑙 valuesshould represent view at level (see subfigure 16a on page 34). For roll motion,recommended value for 𝑙 is (see subfigure 16b). Anamorphic correctionof non-spherical image, driven by the 𝑠 component, depends on subject in view.Purpose of the 𝑠 scalar is to adjust proportions of the figure-in-focus. ⎧⎪⎪⎪⎨⎪⎪⎪⎩Ω max = 1max { ∕ , |𝑘|} ⋅ {→ 𝜋, if 𝑘 > 0𝜋, otherwise Ω ∈ (0, Ω max ]𝑘 ∈ [−1, 1]𝑙 ∈ [0, 1]𝑠 ∈ [ ∕ , 1] (23)Limits for perspective parameters in universal perspective model. Value of Ω max depends on the projection type (represented by scalar 𝑘 ) and varies betweenangle approaching 180° and 360° angle.17 .1.2 Transformation of 2D →
3D coordinates
This transformation produces visual sphere vector map from texture coordinates,that can be later used as an input for perspective map rasterizer.
𝑅 = ||||||||||[ ⃗𝑓 𝑥 ⃗𝑓 𝑦 ⋅ √𝑙]|||||||||| (24a) ≡ √ ⃗𝑓 + 𝑙 ⃗𝑓 (24b) 𝜃 = ⎧⎪⎨⎪⎩arctan ( tan (𝑘 Ω2 )𝑅) ÷ 𝑘, if 𝑘 > 0 Ω2 𝑅, if 𝑘 = 0arcsin ( sin (𝑘 Ω2 )𝑅) ÷ 𝑘, if 𝑘 < 0 (24c) ⎡⎢⎢⎣ ̂𝑣 𝑥 ̂𝑣 𝑦 ̂𝑣 𝑧 ⎤⎥⎥⎦ = ‖‖‖‖‖‖‖‖‖‖‖‖⎡⎢⎢⎣ ⃗𝑓 𝑥 ⃗𝑓 𝑦 sin(𝜃) ∕ 𝑅sin(𝜃) ∕ 𝑅 cos 𝜃 ⎤⎥⎥⎦ ÷ ⎡⎢⎢⎣ 1𝑙(1 − 𝑠) + 𝑠1 ⎤⎥⎥⎦‖‖‖‖‖‖‖‖‖‖‖‖ ◾ (24d)Picture coordinates are denoted by vector ⃗𝑓 ∈ [0, 1] , 𝑎 is picture aspect ratio.Transformed 3D coordinates are represented by normalized vector ̂𝑣 ∈ [−1, 1] .Scalar 𝑘 represents various projection types. Scalar 𝑙 ∈ [0, 1] is the sphericalprojection factor, with 𝑙 = 0 representing cylindrical projection and 𝑙 = 1 , aspherical projection. Scalar 𝑠 ∈ [ ∕ , 1] describes anamorphic correction ofnon-spherical image. For 𝑠 = 1 or 𝑙 = 1 there is no anamorphic correction. →
2D coordinates
This transformation is mainly used in a pixel shader, where basic rectilinearprojection can be mapped to spherical one. It is not suitable for mapping 3Dpoints onto 2D picture plane as in some specific cases single 3D point can mapto multiple 2D positions (see example 2 on the previous page). ̂𝑣 = ⎡⎢⎢⎣ ̂𝑣 𝑥 ̂𝑣 𝑦 ̂𝑣 𝑧 ⎤⎥⎥⎦ (25a) 𝜃 = arccos ( ̂𝑣 𝑧 ÷ √ ̂𝑣 + 𝑙 ̂𝑣 + ̂𝑣 ) (25b) 𝑅 = ⎧⎪⎨⎪⎩tan (𝑘𝜃) ÷ tan (𝑘 Ω2 ), if 𝑘 > 0𝜃 ÷ Ω2 , if 𝑘 = 0sin (𝑘𝜃) ÷ sin (𝑘 Ω2 ), if 𝑘 < 0 (25c)18 ⃗𝑓 𝑥 ⃗𝑓 𝑦 ] = [ ̂𝑣 𝑥 ̂𝑣 𝑦 ] 𝑅√ ̂𝑣 + 𝑙 ̂𝑣 [ 1𝑙 (1 − 𝑠) + 𝑠] ◾ (25d)3D coordinates are represented by a normalized vector ̂𝑣 ∈ [−1, 1] , where vieworigin is at position [0 0 0] . Transformed picture coordinates are representedby vector ⃗𝑓 ∈ ℝ , where image center is at position [0 0] . Angle 𝜃 is betweenvector ̂𝑣 and the 𝑍 axis. 𝑅 is the normalized distance between projected vector ⃗𝑓 and the image center, where ⃗𝑓 ↤ ̂𝑣 . Angle Ω is equal to AOV, where Ω max ∈[→ 𝜋, 2𝜋] . Scalar 𝑘 represents various projection types (see sub-subsection2.1.1 on page 17). Scalar 𝑙 ∈ [0, 1] is the spherical projection factor, where 𝑙 = 0 represents cylindrical projection and 𝑙 = 1 represents spherical projection.Scalar 𝑠 ∈ [ ∕ , 1] describes anamorphic correction of non-spherical image. For 𝑠 = 1 or 𝑙 = 1 there is no anamorphic correction. This transformation is areverse of the universal 2D →
3D transform on the previous page. →
2D coordinates
Combination of 3D and 2D transformation can be used to map between two dif-ferent projections, for example stereographic ↔ equidistant , using two separateperspective component sets, {Ω 𝑖 , 𝑘 𝑖 , 𝑙 𝑖 , 𝑠 𝑖 } and {Ω 𝑜 , 𝑘 𝑜 , 𝑙 𝑜 , 𝑠 𝑜 } for input and outputpicture. 𝑅 𝑖 = |||||||||[ ⃗𝑓 𝑥 ⃗𝑓 𝑦 ⋅ √𝑙 𝑖 ]||||||||| (26a) ≡ √ ⃗𝑓 + 𝑙 𝑖 ⃗𝑓 (26b) 𝜃 = ⎧⎪⎨⎪⎩arctan ( tan (𝑘 𝑖 Ω 𝑖 )𝑅 𝑖 ) ÷ 𝑘 𝑖 , if 𝑘 𝑖 > 0 Ω 𝑖 𝑅 𝑖 , if 𝑘 𝑖 = 0arcsin ( sin (𝑘 𝑖 Ω 𝑖 )𝑅 𝑖 ) ÷ 𝑘 𝑖 , if 𝑘 𝑖 < 0 (26c) ⎡⎢⎢⎣ ̂𝑣 𝑥 ̂𝑣 𝑦 ̂𝑣 𝑧 ⎤⎥⎥⎦ = ‖‖‖‖‖‖‖‖‖‖‖‖⎡⎢⎢⎣ ⃗𝑓 𝑥 ⃗𝑓 𝑦 sin(𝜃) ∕ 𝑅 𝑖 sin(𝜃) ∕ 𝑅 𝑖 cos 𝜃 ⎤⎥⎥⎦ ÷ ⎡⎢⎢⎣ 1𝑙 𝑖 (1 − 𝑠 𝑖 ) + 𝑠 𝑖 (26d) 𝑅 𝑜 = ⎧⎪⎨⎪⎩tan (𝑘 𝑜 𝜃) ÷ tan (𝑘 𝑜 Ω 𝑜 ), if 𝑘 𝑜 > 0𝜃 ÷ Ω 𝑜 , if 𝑘 𝑜 = 0sin (𝑘 𝑜 𝜃) ÷ sin (𝑘 𝑜 Ω 𝑜 ), if 𝑘 𝑜 < 0 (26e) [ ⃗𝑓 ′𝑥 ⃗𝑓 ′𝑦 ] = [ ̂𝑣 𝑥 ̂𝑣 𝑦 ] 𝑅 𝑜 √ ̂𝑣 + 𝑙 𝑜 ̂𝑣 [ 1𝑙 𝑜 (1 − 𝑠 𝑜 ) + 𝑠 𝑜 ] ◾ (26f)19nput picture coordinates are represented by ⃗𝑓 ∈ ℝ , while output picturecoordinates are denoted as ⃗𝑓 ′ ∈ ℝ . Algorithms presented below produce visual sphere vector map in various pro-jections, which can be later used as an input for perspective map rasterizer.Result vector ̂𝑣 ∈ [−1, 1] can be mapped to picture color range [0, 1] by simpletransformation: ⃗𝐺 ′ = ̂𝑣 + 12 (27) ⎡⎢⎢⎣ ̂𝑣 𝑥 ̂𝑣 𝑦 ̂𝑣 𝑧 ⎤⎥⎥⎦ = ⎧⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎩‖‖‖‖‖‖‖‖‖‖‖‖‖‖ 2 ⃗𝑓 𝑠 − 1(2 ⃗𝑓 𝑡 − 1) ÷ 𝑎cot Ω ℎ ‖‖‖‖‖‖‖‖‖‖‖‖‖‖ , if Ω horizontal ‖‖‖‖‖‖‖‖‖‖‖‖‖‖‖𝑎(2 ⃗𝑓 𝑠 − 1) ÷ √𝑎 + 1(2 ⃗𝑓 𝑡 − 1) ÷ √𝑎 + 1cot Ω 𝑑 ‖‖‖‖‖‖‖‖‖‖‖‖‖‖‖ , if Ω diagonal ‖‖‖‖‖‖‖‖‖‖‖‖‖‖𝑎(2 ⃗𝑓 𝑠 − 1)2 ⃗𝑓 𝑡 − 1cot Ω 𝑣 ‖‖‖‖‖‖‖‖‖‖‖‖‖‖ , if Ω vertical (28)Linear perspective map formula, where ̂𝑣 is the visual sphere vector, ⃗𝑓 representsscreen coordinates. 𝑎 is the screen aspect ratio and Ω the AOV. Figure 8: Curved panorama model, where aspect ratio 𝑎 = Ω ℎ ℎ , radius is equalto 1 and ⃗𝑂 denotes arc’s origin. 20 = Ω ℎ 𝐻 ◾ (29a) [ ⃗𝑓 ′𝑥 ⃗𝑓 ′𝑦 ] = [Ω ℎ 𝐻 ] ([ ⃗𝑓 𝑠 ⃗𝑓 𝑡 ] − 12 ) (29b) ⎡⎢⎢⎣ ̂𝑣 𝑥 ̂𝑣 𝑦 ̂𝑣 𝑧 ⎤⎥⎥⎦ = ‖‖‖‖‖‖‖‖‖‖‖‖‖sin ⃗𝑓 ′𝑥 ⃗𝑓 ′𝑦 cos ⃗𝑓 ′𝑥 ‖‖‖‖‖‖‖‖‖‖‖‖‖ ◾ (29c)Curved panorama perspective-map formula, where 𝑎 is the panorama aspectratio, 𝐻 in the display height-to-radius proportion with Ω ℎ as the horizontalAOV. Vector ⃗𝑓 represents screen coordinates and ̂𝑣 is the visual sphere vector. [ ⃗𝑓 ′𝑥 ⃗𝑓 ′𝑦 ] = [2 ⃗𝑓 𝑠 − 11 − 2 ⃗𝑓 𝑡 ] (30a) 𝜃 = | ⃗𝑓 ′ | (Ω + 𝜋2 ) (30b) ⎡⎢⎢⎣ ̂𝑣 𝑥 ̂𝑣 𝑦 ̂𝑣 𝑧 ⎤⎥⎥⎦ = ‖‖‖‖‖‖‖‖‖‖‖‖ ⃗𝑓 ′𝑥 sin 𝜃 ÷ | ⃗𝑓 ′ |cos 𝜃⃗𝑓 ′𝑦 sin 𝜃 ÷ | ⃗𝑓 ′ | + 𝑜‖‖‖‖‖‖‖‖‖‖‖‖ ⎡⎢⎢⎣1 0 00 cos 𝜑 − sin 𝜑0 sin 𝜑 cos 𝜑 ⎤⎥⎥⎦ ◾ (30c) 𝑚 = { 1 − | ⃗𝑓 ′ |𝜕(1 − | ⃗𝑓 ′ |) } ∩ [0, 1] ◾ (30d)Full dome perspective-map formula, where ⃗𝑓 is the screen coordinates vector. Ω is the compression angle, 𝜑 is the tilt angle. ̂𝑣 represents visual sphere vector. 𝑜 is the view position offset (in radius) and 𝑚 is the radial mask with 𝜕(𝑥) beingthe equivalent of fwidth (𝑥) function. [ ⃗𝑓 ′𝑥 ⃗𝑓 ′𝑦 ] = 𝜋 [2 ⃗𝑓 𝑠 − 1⃗𝑓 𝑡 ] (31a) ⎡⎢⎢⎣ ̂𝑣 𝑥 ̂𝑣 𝑦 ̂𝑣 𝑧 ⎤⎥⎥⎦ = ⎡⎢⎢⎣sin ⃗𝑓 ′𝑥 −1cos ⃗𝑓 ′𝑥 ⎤⎥⎥⎦ ⎡⎢⎢⎣sin ⃗𝑓 ′𝑦 cos ⃗𝑓 ′𝑦 sin ⃗𝑓 ′𝑦 ⎤⎥⎥⎦ ◾ (31b)Equirectangular projection perspective-map formula, where ⃗𝑓 represents screencoordinates and ̂𝑣 is the visual sphere vector.21 .2.5 Mirror dome projection map ⤻ Figure 9: Mirror dome projection model, with mirror center as world origin. ⃗𝐼 = ̂𝑁 − ⃗𝑃 (32a) ̂𝑅 = ‖⃗𝐼 − 2(⃗𝐼 ⋅ ̂𝑁) ̂𝑁‖ (32b) ⎧⎨⎩ 𝑟 = ̂𝑅 ⋅ ( ⃗𝑂 − ̂𝑁)𝑟 ′ = 𝑟 + √𝑑 − |𝑟 ̂𝑅 + ̂𝑁 − ⃗𝑂| (32c) ̂𝑣 = 𝑟 ′ ̂𝑅 + ̂𝑁 − ⃗𝑂𝑑 ◾ (32d) 𝑚 = ( 𝑟 ′ + |⃗𝐼|(𝑟 ′ + |⃗𝐼|) max ) ◾ (32e)Mirror dome projection perspective-map formula, where ⃗𝐼 represents incidentvector. ̂𝑁 ∈ [−1, 1] is the spherical-mirror world-normal and surface position, ⃗𝑃 is the projector position, ⃗𝑅 is reflection vector, 𝑟 ′ is the reflection distance todome intersection, as: ̂𝑅 ⋅ ( ⃗𝑂 − ̂𝑁) = ̂𝑅 ⋅ ⃗𝑂 ′ = (cid:26)(cid:26) | ̂𝑅|| ⃗𝑂 ′ | cos 𝛼 = (cid:26)(cid:26)(cid:26) | ⃗𝑂 ′ | 𝑟 (cid:26)(cid:26)(cid:26) | ⃗𝑂 ′ | = 𝑟 (33) ̂𝑣 ∈ [−1, 1] is the visual sphere vector as mirror surface color and ⃗𝑂 is the domeorigin position with 𝑑 as dome radius. Mirror radius is equal with its originat position [0 0 0] . Light dimming mask is represented by 𝑚 and it’s based22n inverse-square law approximation. In order to produce perspective mapimage, first mirror 3D model world-normal pass must be produced, as viewedfrom projector’s perspective. It is possible to render the view with additionalperspective map of the projector. Remark.
Mirror dome projection system was originally developed by P. Bourke. Figure 10: Projection mapping model, with 𝑊 as the world origin coordinates, ⃗𝑂 as observation point, ⃗𝐼 as projector position and 𝑆 as projection surface. ⎡⎢⎢⎣ ̂𝑣 𝑥 ̂𝑣 𝑦 ̂𝑣 𝑧 ⎤⎥⎥⎦ = ‖‖‖‖‖‖‖‖‖‖‖‖ ⃗𝑆 𝑥 − ⃗𝑂 𝑥 ⃗𝑆 𝑦 − ⃗𝑂 𝑦 ⃗𝑆 𝑧 − ⃗𝑂 𝑧 ‖‖‖‖‖‖‖‖‖‖‖‖ ⎡⎢⎢⎣𝑊 ′11 𝑊 ′12 𝑊 ′13 𝑊 ′21 𝑊 ′22 𝑊 ′23 𝑊 ′31 𝑊 ′32 𝑊 ′33 ⎤⎥⎥⎦ ◾ (34a) 𝑚 = ( | ⃗𝑆 − ⃗𝐼|| ⃗𝑆 − ⃗𝐼| max ) ◾ (34b)Projection mapping perspective-map formula, where ̂𝑣 represents the visualsphere unit-vector. ⃗𝑆 is the environment surface position, as seen through pro-jector’s point of view. ⃗𝑂 denotes observation point. 𝑊 ′ is the optional rotationmatrix for the world position ⃗𝑆 . Light dimming mask is denoted by 𝑚 , whichvalue is based on the inverse-square law. ⃗𝐼 represents the projector world posi-tion.In order to produce perspective map image, first 3D model of the projectionenvironment must be produced. Then world position map 𝑆 must be renderedfrom projector point-of-view. Position map 𝑆 changes only if projector position ⃗𝐼 changes, so the map can be reused at each frame even if the observation positionis dynamic. For more about projection mapping, see example 1 in sub-subsection1.1.5 on page 12. 23 .2.7 Cube-mapping perspective map Figure 11: Cube map model with face orientation. ⎧⎨⎩ ̂𝑋 = [1 0 0] T ̂𝑌 = [0 1 0] T ̂𝑍 = [0 0 1] T 𝑖 = ⌊6 ⃗𝑓 𝑠 ⌋ (35a) ⎡⎢⎢⎣ ̂𝑣 𝑥 ̂𝑣 𝑦 ̂𝑣 𝑧 ⎤⎥⎥⎦ = ‖‖‖‖‖‖‖‖‖‖‖‖(6 ⃗𝑓 𝑠 ) mod ∕ ⃗𝑓 𝑡 − ∕ ∕ ‖‖‖‖‖‖‖‖‖‖‖‖ ⋅ ⎧⎪⎪⎪⎪⎨⎪⎪⎪⎪⎩[ ̂𝑍 − ̂𝑋 − ̂𝑌] T , if 𝑖 = 0[− ̂𝑍 ̂𝑋 − ̂𝑌] T , if 𝑖 = 1[ ̂𝑋 − ̂𝑌 − ̂𝑍] T , if 𝑖 = 2[ ̂𝑋 ̂𝑍 − ̂𝑌] T , if 𝑖 = 4[− ̂𝑋 − ̂𝑍 − ̂𝑌] T , if 𝑖 = 5 ◾ (35b)Cube-mapping perspective-map formula, where ̂𝑣 represents visual sphere vec-tor and ⃗𝑓 is the screen coordinate. [ ̂𝑍 − ̂𝑋 − ̂𝑌] represents rotation matrixof each cube side. Operation 𝑥 mod is equivalent to fract (𝑥) function. Figure 12: Screen array model with Ω ℎ = ∕ 𝜋 , and 𝑂 as observation point.24 = ⌊𝑛 ⃗𝑓 𝑠 ⌋ + 1 − 𝑛2 (36a) ⎡⎢⎢⎣ ̂𝑣 𝑥 ̂𝑣 𝑦 ̂𝑣 𝑧 ⎤⎥⎥⎦ = ‖‖‖‖‖‖‖‖‖‖‖‖‖‖2((𝑛 ⃗𝑓 𝑠 ) mod
1) − 1(2 ⃗𝑓 𝑡 − 1)𝑛 ÷ 𝑎cot Ω ℎ ‖‖‖‖‖‖‖‖‖‖‖‖‖‖ ⎡⎢⎢⎣ cos(𝑖Ω ℎ ) 0 sin(𝑖Ω ℎ )0 1 0− sin(𝑖Ω ℎ ) 0 cos(𝑖Ω ℎ )⎤⎥⎥⎦ ◾ (36b) 𝑛 -screen array perspective-map formula, where 𝑛 is the number of screens, ̂𝑣 represents visual sphere vector, ⃗𝑓 is the screen coordinates vector, with Ω and 𝑎 being a single-screen AOV and aspect ratio, respectively. Figure 13: Virtual Reality model with 𝑑 as the IPD distance relative to totalscreen width. Such that 𝑑 = IPDwidth . 𝑖 = sign ( ⃗𝑓 𝑠 − ∕ ) (37a) [ ⃗𝑓 ′𝑥 ⃗𝑓 ′𝑦 ] = ([2((2 ⃗𝑓 𝑠 ) mod
1) − 12 ⃗𝑓 𝑡 − 1 ] + [𝑖(1 − 2𝑑)0 ]) [0.5𝑎1 ] (37b) [ ⃗𝑓 ′′𝑥 ⃗𝑓 ′′𝑦 ] = [ ⃗𝑓 ′𝑥 ⃗𝑓 ′𝑦 ] radial lens distortion ⏞ ⎴⎴⎴⎴⎴⎴⎴⎴⎴⎴⎴ ⏞ ⎴⎴⎴⎴⎴⎴⎴⎴⎴⎴⎴ ⏞1 + 𝑘 | ⃗𝑓 ′ | + 𝑘 | ⃗𝑓 ′ | + … + 𝑘 𝑛 | ⃗𝑓 ′ | + 𝑘 + … + 𝑘 𝑛 ⏟⎴⎴⎴⎴⎴⎴⏟⎴⎴⎴⎴⎴⎴⏟ bounds normalization (37c) ⎡⎢⎢⎣ ̂𝑣 𝑥 ̂𝑣 𝑦 ̂𝑣 𝑧 ⎤⎥⎥⎦ = ‖‖‖‖‖‖‖‖‖‖‖‖‖ ⃗𝑓 ′′𝑥 ⃗𝑓 ′′𝑦 cot Ω 𝑣 ‖‖‖‖‖‖‖‖‖‖‖‖‖ ◾ (37d)Virtual reality perspective map formula, where ⃗𝑓 represents screen coordinates, 𝑑 is the interpupillary distance (IPD) in screen-width scale, 𝑎 is the screen aspectratio, 𝑘 , 𝑘 , … , 𝑘 𝑛 represent lens-distortion coefficients. ̂𝑣 is the visual spherevector and Ω 𝑣 is the vertical AOV. 25 emark. VR perspective map can also be combined side-by-side with a regularperspective map. In extended desktop environment, extra perspective map couldprovide a monitor preview of the VR content, without an additional render call.
Creating perspective picture of a real optical system may require additionaldeformation of the vector data. Most commonly used algorithm for this purposeis the
Brown-Conrady lens distortion model. 𝑟 = ⃗𝑓 ⋅ ⃗𝑓 (38a) [ ⃗𝑓 ′𝑥 ⃗𝑓 ′𝑦 ] = [ ⃗𝑓 𝑥 ⃗𝑓 𝑦 ] + radial distortion ⏞⎴⎴⎴⎴⎴⎴⎴⎴⎴⎴⏞⎴⎴⎴⎴⎴⎴⎴⎴⎴⎴⏞(𝑘 𝑟 + 𝑘 𝑟 + ⋯ + 𝑘 𝑛 𝑟 ) [ ⃗𝑓 𝑥 ⃗𝑓 𝑦 ]+ ([𝑝 𝑝 ] ⋅ [ ⃗𝑓 𝑥 ⃗𝑓 𝑦 ]) [ ⃗𝑓 𝑥 ⃗𝑓 𝑦 ]⏟⎴⎴⎴⎴⎴⏟⎴⎴⎴⎴⎴⏟ thin prism + [𝑞 𝑟 𝑞 𝑟 ]⏟ ⏟ ⏟ decentering ◾ (38b)Where 𝑟 is the dot product of two ⃗𝑓 vectors. 𝑘 , 𝑘 and 𝑘 𝑛 are the radial distortioncoefficients. 𝑞 and 𝑞 are the decentering coefficients. Thin prism distortioncoefficients are denoted by 𝑝 and 𝑝 . Real optical systems exhibit phenomenon known as the floating no-parallaxpoint, where each incident vector originates from different position withinthe lens. Meaning that real optical systems are not complicit with pinholeprojection model. In pinhole model all incident vectors share same crossingpoint. Therefore to simulate optical projection, view position has to changeaccordingly to incident vector origin of the calibrated lens, or in contrary, visiblepoint should move the opposite way. In spherical lens, NPP offset is in 𝑧 -directionand can be described as a product of a function parallax (𝜃) . As it changesaccordingly to an angle between incident vector and the optical axis. Offsetvalue can be approximated by optical measurement of parallax miss-alignment(see figure 21 on page 41).To calibrate lens distortion with floating NPP, first static NPP picture mustbe obtained. If camera lens does not produce such image, it can be derivedfrom a sequence of images, each taken at different 𝑧 position (see subfigure 21don page 41). In such image composite, every point of the picture plane sharescommon incident vector origin (in world-space). From this, two lens maps can26e derived, parallax offset map from sequence’s pixel 𝑧 position and perspectivemap from composite image of calibration rig/chart.To render picture with floating NPP, each 3D point must be transformedprior to rasterization. Transformation is done accordingly to the perspectivemap incident position and associated parallax offset value. Either by movingthe point or the view position. Noticeably the parallax offset value can also beencoded in a graph (for simple spherical lenses), instead of a texture map.Rendering with rasterization would produce approximate result as valuesin-between vertices are interpolated, not transformed. Therefore best qualityfloating-NPP result is to be expected from ray-tracing. In such case, offset of aray-origin-position should be performed. An equivalent of visual-sphere offset(see subfigure 21b on page 41). Going back to rasterization, point offset can beachieved through projection of the parallax offset texture map onto the scenegeometry. Each geometry point can be transformed in view-space 𝑧 direction bya value from projected parallax offset texture. Perspective picture visible inside the visual space gives some sense of immersion(e.g. picture, film, computer game) even without visual illusion. That’s becauseit is perceived as a visual symbol of an abstract point of view, through whichnot picture plane is seen, but depicted space’s mind-reconstruction. The pictureimmersion does not break, as long as appearance of the objects do not exhibit toomuch deformation. Perceiving abstract point of view invokes separation fromthe surrounding. To enhance immersion, environment stimuli is being reduced.In a movie theater, to uphold the immersion lights are turned off and silence isexpected. Horror-gameplay session are usually played at night, to separate fromsafe-space of home. This approach focuses virtual presence on depicted space.
Remark.
On the opposite side, picture as an integral part of the surrounding canbe categorized under the
Trompe-l’œil technique. Through the perception of the picture, physical properties of depicted spaceand objects within it are estimated. Mind fills the gaps, as symbols are alwayssimplified versions of the real thing.
Example.
Cup of coffee laying on a table has only one side visible at once, butit can be assumed that the opposite one is there too, since shape of the cup isknown. This is a blank information filled by the mind.Physical objects have their physical properties, but their visual symbols havesome physical properties too, like angular size, perspective, shadow, etc. Thosevisual properties give some information about physical. In case of perspective,visual properties give information about depicted space and about point ofview. Since most of the time picture represents a point of view (e.g. film, videogame, visualization), it is wise to consider subject’s properties of vision when27esigning picture perspective. But instead of producing mechanical simulation,perspective should symbolize total sensory experience. Theorem 1.
To create immersive visual symbol of a visual space, it is necessary touse curvilinear perspective instead of a linear.Proof.
Geometry of human visual space contradicts linear perspective principle,as visual field extends beyond linear perspective angle of view. Linear perspec-tive, based on a tangent of an angle, exhibits limit of 179.(9)8° of view. Whilevisual field extends horizontally up to 220° for binocular vision. ◾ R θ
OrthographicEquisolidEquidistantStereographicGnomonic (a) Graph plotting ray of angle 𝜃 ∈[0 ° , Ω] — as the horizontal axis, andscreen-position radius 𝑅 ∈ [0, 1] , as thevertical axis, where
Ω = 40 ° which isequivalent to
𝑅 = 1 . R θ
OrthographicEquisolidEquidistantStereographicGnomonic (b) Graph plotting ray of angle 𝜃 ∈[0 ° , Ω] — as the horizontal axis, andscreen-position radius 𝑅 ∈ [0, 1] , as thevertical axis, where
Ω = 170 ° which isequivalent to
𝑅 = 1 . Figure 14: Chart comparison of radial compression in five major azimuthalprojections, across two different AOV ( Ω ) values: narrow (14a) and wide (14b).At narrow AOV both types of perspective are suitable for immersive picture.In such case differences between each projection are negligible. Subfigure 14apresents those differences in comparison to five major perspective projections.Same differences seem exaggerated at higher AOV values (see subfigure 14b). Corollary 1.1.
Practical limit for immersive picture in linear perspective is between60° and 110° AOV. Wider-angles exhibit deformations known as the
LeonardoParadox , which are then dominant in the image perception and break picture’simmersion. ◾ To show wider-angle picture it is necessary to use curvilinear projection. Butthere is a tendency to see the world not through sight but understanding. Weunderstand that the wall is flat, therefore we see it that way. Picture projected28nto the eye is just a visual symbol and has its own physical properties (e.g.perspective and shape). Therefore its visual representation is curvilinear, wherethe curvature symbolizes wider field of view. Reader can validate curvilinearnature of human visual space, by following A. Radley experiment : “Also when you have a moment, get a 30 cm ruler (...), and whilstlooking forward bring it close to the bottom of your nose, and noticehow its shape at the outer edges curves upwards and forwards. It maytake you a few minutes to be able to see this effect, because you are soaccustomed to not noticing it ! But once you do you will be amazedto see your curved field of view as it really is for the first time.” — A.Radley Since we came into conclusion that symbol of a visual space is curvilinear, thereis a task of selecting between many non-linear projections. Each has proper-ties that symbolize subject’s perception or information span about depictedspace. Not appearance of the symbol should dominate the picture, but projectedinformation about point of view and depicted space properties.
Problem . Which curvilinear perspective is best for visual symbol of visualspace?
Proposition . A model based on anamorphic lens geometry; a mix betweenfish-eye, panini and anamorphic projection. fish-eye as it can represent wider AOV than linear perspective (e.g. 𝜋 ) andconforms to the curvilinear nature of VS. Gives natural spatial awareness. panini to symbolize binocular vision; two spherical projections combined intoone panoramic image. a Produces picture geometry more familiar to theviewer. anamorphic as cylindrical projection, like
Panini , tends to elongate propor-tions vertically; there is a need for correction. Correction should makeobject in focus proportional as it varies depending on position and size.
Remark.
Only linear anamorphic correction will conform to the perspec-tive picture definition (see on page 35).
Remark.
Equations in subsection 2.1 on page 16 (about perspective transforma-tion) presents variables that drive all mentioned above geometrical factors. a Effect is also referred to as
Stereopsis . .2 Visual sphere as a whole image Common idea of an image is limited to a finite 2-dimensional plane. Which issubjective, due to constrains of human visual field and up-front placement ofeyes. One can construct a rectangular frame, which at certain distance from theeyes will cover full visual field (VF). In case of some animals (e.g. horse, rabbit),visual space confines much wider VF. With only few blind spots, spanning toalmost 360°AOV.
Such field cannot be enclosed by a single rectangular frame.Thus image nature is not of a frame. Another model has to be chosen instead.One able to cover full
Ω = 360 ° is a sphere.
Remark.
Cylindrical projection cannot cover full 360°AOV in all directions. Itis a hybrid between frame and spherical model. When vertically-oriented, full Ω 𝑣 < 180 °.All three-dimensional space around given observation point, can be projectedonto a sphere, with given observation point as a origin. Even doe sphere itself is a3D object, its surface (as well as image nature) is two-dimensional. Thereforecreating perspective picture is a matter of representing portion of the visualsphere on a flat surface; a fundamental topic in cartography. Concept of a sphereas a model of visual space goes back as far as 300 BC, where Greek mathematicianEuclid seem first to mention (others are L. Da Vinci and F. Aguilonius). Remark.
Each projection of sphere onto a flat surface is a compromise and canpreserve only some properties (e.g. shape, area, distance or direction), which incase of perspective picture relates to some symbolic information about physicalspace.
Definition 1.
Let us define perspective picture as the azimuthal projection,where lines converging at optical axis vanishing point remain straight, that conservation of perspective may occur (see definition on page 35).
Below are presented static properties of five major azimuthal projections. Prop-erties of motion can be found in sub-subsection 4.3.2 on page 33. It is importantto know which symbolic information about space is carried in each perspectiveprojection, so that design choice for perspective geometry may be conscious.
Gnomonic (rectilinear) projects all great circles as straight lines, thus preserv-ing directions. For 3D projection, straight lines in object-space remainstraight. It does not preserve proportions, angles nor area or distances (seesubfigure 15a on page 32). Extreme distortion occurs away from the center,in a form of radial stretch (see
Leonardo Paradox ) . AOV Ω ∈ (0, 𝜋) . Example.
Most common perspective type in painting, 3D graphics and ar-chitectural visualization. Sometimes it is used to overemphasize buildingappearance by leveraging
Leonardo Paradox . Wide AOV combined with30owered optical center creates an effect of acute corners. This produces anextraordinary look. Such visual-trick may confuse the public, as experi-ence of symbolic picture won’t match building visual-space appearance.
Stereographic (conformal) preserves angles (at line intersection point). Thereis no perceivable radial compression, thus smaller figures retain theirshape. It does not preserve distances (non-isometric), nor angular surfacearea. For 3D projection, most important factor is the conservation ofproportions (see subfigure 15b on the following page). AOV
Ω ∈ (0, 2𝜋) . Example.
In a picture with stereographic projection, face of the actorkeeps its shape and proportions, even at wide AOV. This projection alsogives best spatial-awareness sensation (where visual cues are available).Good use case is navigation through tight spaces and obstacles.
Equidistant preserves angular distance from the center point (see subfigure15c on the next page). For 3D projection, angular speed of motion ispreserved. Radial compression remains low-to-moderate at extreme Ω angles. AOV Ω ∈ (0, 2𝜋] . Example.
This projection is recommenced for target aiming or radar mapnavigation, where all targets are projected onto a Gaussian Sphere.
Equisolid preserves angular area. Gives good sensation of distance (see subfig-ure 15d on the following page). Radial compression is moderate up to 𝜋 .Near maximum Ω , compression is high. AOV Ω ∈ (0, 2𝜋] . Example.
When there are no spatial cues, this is best projection for puttingemphasis on the distance to the viewer. Good use case is flight simulation,where only sky and other aircraft are in-view.
Orthographic preserves brightness. It is a parallel projection of a visual hemi-sphere. Has extreme radial compression, especially near 𝜋 (see subfigure15e on the next page). AOV Ω ∈ (0, 𝜋] . Example.
Most commonly found in very cheap lenses, like the peepholedoor viewer. Thanks to brightness being proportional to occupied imagearea, it found common use in sky photography and scientific research.
Picture perspective affects the way motion is perceived. It can enhance certainfeatures, like proportions and shapes, movement or spatial-awareness. It canalso guide viewer attention to a specific region of image (e.g. center or periphery).Knowledge about those properties is essential for conscious image design.31 .II. III. (a) Rectilinear (Gnomonic) projection
II.I. III. (b) Stereographic projection
II.I. III. (c) Equidistant projection
II.I. III. (d) Equisolid projection
II.I. III. (e) Orthographic-azimuthal projection θ II.I. III.θ Or (f) Model of examples; an equal-size discs ( 𝐼 , 𝐼𝐼 , 𝐼𝐼𝐼 )at equal distance 𝑟 , from and facing observationpoint 𝑂 , where Ω ℎ = 170 °, 𝜃 = 30 ° and 𝜃 = 60 ° Figure 15: Example of motion in perspective picture of various azimuthal pro-jections. Here subfigure 15f presents viewed elements layout.
In film design, there are several techniques that focus viewer attention on a spe-cific portion of the picture (like motion , color , light and composition ). Attentionfocusing through composition and motion is related to picture’s perspective,as its geometry can compress and stretch the image. In composition, rule ofthirds states that viewer attention focuses on four corners of a rectangle pro-duced by division of an image into three, equally-sized rows and columns. Inmotion, attention generally drives toward objects approaching the camera orthose growing in scale. Attention also focuses on objects entering image frame.Same rules apply loosely in reverse, as attention suspense. Filmmakers tend to32rame the image so that region of interest lays in accordance to the rule of thirds .In case of computer games, region of interest is usually located at the very center,thus viewer must overcome the principle of thirds and some properties of linearperspective in order to switch attention to that region. In order to focus on thecenter, games usually incorporate some non-diegetic elements, like crosshair.Such approach may lower immersiveness of symbolic picture. Radial stretching and compression are the main attention focusing factors ofperspective projection. They give subconscious sensation of movement towardscamera, and can amplify figure screen-relative speed of motion.
Gnomonic (rectilinear), due to extreme radial stretch attention drives towardsperiphery. When approaching image bounds figure grows in scale andspeed (see subfigure 15a on the preceding page). This combined withmotion-sensitive peripheral vision adds to the effect. At wider AOV am-plified motion breaks immersion of symbolic picture.
Stereographic also draws attention towards periphery. Figures grow in scalenear bounds, but immersion does not break, as proportions are preserved,even at wide AOV (see subfigure 15b on the previous page).
Equidistant drives attention toward center, as figures in periphery are radiallycompressed (see subfigure 15c on the preceding page). This projectionpreserves screen-relative, radial speed of motion, making it uniform andrepresentative across the picture.
Equisolid also drives attention toward the center, as radial compression is evengreater (see subfigure 15d on the previous page). Figure speed of motionin screen-space slightly declines towards periphery.
Orthographic has extreme radial compression that breaks immersion of asymbolic picture (see subfigure 15e on the preceding page). When inmotion, image seem to be imposed on an artificial sphere.
Gnomonic and
Orthographic projections are the two extremes of azimuthalspectrum. They both are least suited for an immersive picture.
Cylindrical perspective type, while symbolizing binocular vision, also gives visual cue forthe vertical axis orientation. Such cue is undesirable in case of camera roll-motion, or when view is pointing up/down, as image vertical axis will then notbe aligned with depicted space orientation. In such case perspective geometryshould transition to spherical projection (see figure 16 on the next page).33 a) Example of image geometry forpitch/yaw motion (arrows), where Ω ℎ = 120 °, 𝑘 = 0 , 𝑙 = 10% and 𝑠 = 95% . ↔ (b) Example of image geometry for rollmotion (arrows), where Ω ℎ = 120 °, 𝑘 = 0 , 𝑙 = 100% and 𝑠 = 95% . Figure 16: Examples of image geometry for a given view-motion type.
Current image abstract theorem was established in 15 th century book De Pictura, by L. B. Alberti. Based on invention of F. Brunelleschi, Alberti definedgeometrical and theoretical rules for perspective projection design. These rulesare currently used in polygon-based CG graphics. Major theoretical statementthat laid foundation on image projection technology and present understandingof image nature can be traced back to Alberti abstract definition of image. Hedescribed painting being like a window in a wall : “First of all, on the surface on which I am going to paint, I draw arectangle of whatever size I want, which I regard as an open windowthrough which the subject to be painted is seen.” — L. B. AlbertiBut in times of its discovery, as for now, linear perspective introduced itself withseveral issues. When there’s a need for a wide-angle view, one close to humanvisual field, geometrical distortions appear to dominate visual aspect of thepicture. These issues were noticed by Renaissance artists, like L. Da Vinci. Heput to test the Alberti Theorem and produced paintings of accurate-perspective. In his
Treatise on Painting , Da Vinci notes that picture conforms to the idea ofa window only when viewed from one specific point in space. Stating that seenotherwise, objects appear distorted, especially in the periphery. Picture then,viewed from a different point ceases to be like a window in a wall and becomes avisual symbol of an abstract point of view. b Some 18 th century late Baroque and b Effect also referred to
Zeeman Paradox . Panini Projection , or R. Barker, who established the term Panorama . This was a new type ofperspective. In a form of cylindrical projection, where abstract window framebecomes horizontally curved, reducing deformation artifacts in wide, panoramicdepictions of architecture.Figure 17: Still from Todd–AO High-Speed Anamorphic lens (35mm T1.4)with visible curvilinear perspective. This type of lens was featured in films like
Conan the Barbarian , Dune and
Mad Max . © 2017 ShareGrid, Inc. Invention of motion picture followed by the rise of film industry, resultedin a demand for new image geometry. Previously still, now pictures had to bepleasing to the eye, in motion. 1950s brought anamorphic cinematography tothe wider audience. Lenses like CinemaScope and later Panavision becamea standard in film production. Figure 17 shows example of mixed spherical andcylindrical projection with perspective preservation, of anamorphic lens. Definition 2.
Conservation of perspective - lines converging at the optical-axisvanishing-point remain straight.
Remark.
See also perspective picture definition on page 30.CG image technology did not follow film industry in that field. Still based onAlberti theorem, computer graphics became incompatible with film, gener-ating great costs, when two had to be joined together. Stitching such picturerequires lens aberration rotoscopy, where geometry correction is performedmanually at each frame. Currently in computer-games industry, CG imageryis practically unable to produce realistic, curvilinear simulation of visual space(VS), or even simulate anamorphic lens geometry, due to limits of linear per-spective and resource costs of overcoming those issues. Some hybrid solutionswhere proposed, that combine rasterization with ray-tracing, or tessellation.Such approach allows for a semi-practical and limited production of real-timepictures in a non-linear perspective. 35
Conclusion and future work
Visual sphere perspective model expands possibilities for image creation. Like avision of a classical artist is richer and more dynamic than his final creation, soshould be a model describing image. So much that virtual vision would have tobe reduced to fit the medium, with a room for adjustment.In medieval times people were fascinated with mirror, it depicted realityas it really is, a task impossible for human hand. But mirror could not reflectthe vision of imagination, so much as painting. It did not produce realisticreflection of reality, nor reflection of imagination. In art there was a pursuitinvolving philosophy and religion to reach some imaginary reflection. Only afterBrunelleschi experiment proved that painting created by human hand, withstrict rules of geometry, reflected from a mirror fits well into reality, artists startedpursuing mathematics of art. Film and photography became a new mirror, butstill not one of imagination. Finally computer graphics became a format capableto reflect one’s vision of imagination, but it struggled to be photo-real.Language of image is ultimately a language of symbols, it works in conjunc-tion with mind, which recreates visual space of the picture. Computer gameimage is special in a sense, that rather being a viewing glass (like in a film) itimpersonates point of view of the protagonist, which invokes different symbolicmeasures. Real-time graphics should be capable to incorporate such visualsymbols. One way for achieve this is to change current perspective model fora bigger one. Proposed here change includes the use of universal perspectivemodel which defines simple variables for manipulation of perspective geom-etry. It also includes production technique for a real-time imagery using thismodel. Such new solution fits well into current graphics pipeline, replacingonly low-level rasterization processes. Aliasing-free end result is comparablein quality to 8 × MSAA and enables previously impossible visual effects in real-time graphics. Presented visual sphere model unites all types of perspectiveprojections under one technical solution, making perspective a fluid construct.Perspective vector-maps can be easily combined and transformed by sphericalinterpolation. Picture geometry can now be designed to smoothly adapt to thevisual story, giving new dimension of control over mental perception of image.Presented concepts and equations may also find their use in other fields, notrendering-related. This is a great base as well as complete solution upon whichmany new technologies and works can emerge. Some specific use cases stillrequire additional research, like hidden surface removal and no-parallax pointmapping. Further studies will include research over calibration and simulationof real optical systems with floating NPP. Also performance tests, comparison tocurrent solutions should be evaluated by future research. Psychological analysisof perspective geometry magnitude of influence on depicted space perception,performed on a large sample data, could be an interesting field to study.36 cknowledgments
Special thanks to Zbigniew Rybczyński who told that we don’t see the world inrectilinear perspective and that CG image does not match camera lens projection(during his 2010 conference in Cieszyn). This single presentation became aspark for this research. Additional thanks to Shaun Williams who combinedcylindrical and spherical projection based on a view angle. To Alan Radleyand Kim Veltman for their interest in this topic and paper. To Arash Shivawho provided permission for use of the anamorphic lens test. Final thanks toDobiesław Kaczmarek who taught principles of perspective drawing. And toall unmentioned individuals who gave much needed feedback, it helped greatlyimproving this paper. 37 a) Vector map of rectilinear perspective,where Ω 𝑑 = 140 °, 𝑘 = 1 , 𝑙 = 1 . → (b) Rasterized quad in rectilinear per-spective, where Ω 𝑑 = 140 °, 𝑘 = 1 , 𝑙 = 1 .(c) Equirectangular projection vectormap of a whole sphere, where Ω ℎ = 360 °and Ω 𝑣 = 180 °. → (d) Rasterized quad in equirectangularprojection, where Ω ℎ = 360 ° and Ω 𝑣 =180 °.(e) Vector map of mustache-type lens dis-tortion, where Ω 𝑑 = 131 °, 𝑘 = 0.32 , 𝑙 = 62% , 𝑠 = 86% , 𝑘 = −0.6 , 𝑘 = 0.4 . → (f) Rasterized quad in mustache-typelens distortion, where Ω 𝑑 = 131 °, 𝑘 =0.32 , 𝑙 = 62% , 𝑠 = 86% , 𝑘 = −0.6 , 𝑘 = 0.4 .(g) Vector map of curvilinear fish-eye per-spective, where Ω 𝑑 = 270 °, 𝑘 = 0.32 , 𝑙 = 62% , 𝑠 = 86% . → (h) Rasterized quad in curvilinear fish-eye perspective, where Ω 𝑑 = 270 °, 𝑘 =0.32 , 𝑙 = 62% , 𝑠 = 86% .(i) Vector map of five-screen horizontalarray in rectilinear projection, where sin-gle screen Ω ℎ,𝑣 = 60 °, 𝑘 = 1 and 𝑙 = 1 ,giving a total ℎ = 300 °. → (j) Rasterized quad in five-screen hori-zontal array, where single screen Ω ℎ,𝑣 =60 °, 𝑘 = 1 and 𝑙 = 1 , giving a total ℎ = 300 °. Figure 18: Examples of polygon quad rasterization directly from three-dimensional space to image, using visual-sphere vector map 𝐺 , where 𝐺 ⃗𝑠𝑡 ∈[0, 1] ↤ [−1, 1] . 38 ack-facing?Skip polygonNo region?Skip polygon Once perpolygon,per frame Once perpolygon,per frameOnce perframe,per polygon,per bin-maskpixel Figure 19: Vertex rasterization process flowchart. Chart starts with ⃗𝐴, ⃗𝐵, ⃗𝐶 triangle points (transformed to camera-space) and produces low-resolutionbinary mask (using perspective vector map 𝐺 ) and rasterization matrix 𝑅 ′ .39 nce perfragmentpixel Figure 20: Fragment rasterization process flowchart. Chart starts with rasteriza-tion matrix 𝑅 ′ , binary mask of render regions, polygon normal vector ⃗𝑁 , trianglepoints ⃗𝐴, ⃗𝐵, ⃗𝐶 and perspective vector map 𝐺 . For each pixel, aliasing-free mask isproduced and barycentric coordinates ⃗𝑏 are calculated, then interpolated vertexdata 𝑓 is passed to fragment shader as output.40 z NS m (a) Model of no-parallax point (NPP) calibrationrig. Which measures misalignment of 𝑚 markersbetween northern (𝑁) and southern hemisphere( 𝑆 ), as seen through camera 𝐶 . offsetviewposition offsetmodelposition N NS Sm r (b) Simulation of floating NPP in fish-eye camera.Black arrows represent position offset of the modelfor rasterization and view position offset for ray-tracing. When measuring the NPP (sub-figure 21c),camera travel is opposite to view position offset. m m m m m m m m m m m m m m m I.II.III. (c) Example 𝐼 presents misalignment of the camerain all three axes. Example 𝐼𝐼 presents alignmentin X, Y axis, where peripheral marker pair 𝑚 , 𝑚 are aligned near horizontal AOV, while pairs 𝑚 , 𝑚 are misaligned due to floating NPP. Example 𝐼𝐼𝐼 presents "slit-scan" composite of variable 𝑧 position,where all markers are aligned. z
1. 2. 3. (d) Example of a variable camera position 𝑧 encodedin an image sequence (1, 2, 3) . Element presentsalignment of the peripheral markers, while element , alignment of the side markers. Element number presents position in-between. Figure 21: Floating no-parallax point rendering and measurement process.41 eferences
1. Alberti, L. B. in.
On painting and on sculpture trans. by Grayson, C., 55(Phaidon, London, England, 1972). http://as.vanderbilt.edu/koepnick/Modernism_f05/materials/images/alberti_window.htm (cit. on pp. 6,34).2. Argan, G. C. & Robb, N. A. The Architecture of Brunelleschi and the Originsof Perspective Theory in the Fifteenth Century.
Journal of the Warburg andCourtauld Institutes
96. doi: (1946) (cit. on pp. 28, 34).3. Bagley, L. H. & Lavach, D. in Varga, M.
Textbook of Rabbit Medicine E-Book chap. 9.1 Ocular anatomy and physiology (Butterworth-Heinemann, 2013)(cit. on p. 30).4. Baldwin, J., Burleigh, A. & Pepperell, R. Comparing Artistic and Geomet-rical Perspective Depictions of Space in the Visual Field. i-Perception . http://journals.sagepub.com/doi/pdf/10.1068/i0668 (2014) (cit. on p. 29).5. Bourke, P. Low Cost Projection Environment for Immersive Gaming. Jour-nal of MultiMedia
10 . 4304 / jmm . 3 . 1 . 41 - 46 . http ://paulbourke.net/papers/dime2006/dime2006.pdf (2008) (cit. onpp. 22, 23).6. Casamassina, M. King Kong’s Immersive Style
IGN. http : / / ign . com /articles/2005/07/18/king-kongs-immersive-style (2019) (cit. onp. 33).7. Da Vinci, L.
Annunciation oil and tempera on panel. Painting. Uffizi Gallery,Florence, Italy, 1475-1480. http : / / en . wikipedia . org / wiki / De _pictura (cit. on p. 34).8. Da Vinci, L. in
A treatise on painting trans. by Rigaud, J. F., 52 (ProjectGutenberg, Salt Lake City, USA, 2014). http://gutenberg.org/ebooks/46915 (cit. on p. 34).9. Dixon, R. in
Mathographics
Proceedings of the ACM on Computer Graphics and InteractiveTechniques doi: . https://hal.archives-ouvertes.fr/hal-02898121/document (2020) (cit. on p. 15).11. Ericson, C. Minimum bounding circle (sphere) for a triangle (tetrahedron) http://realtimecollisiondetection.net/blog/?p=20 (2020) (cit. onp. 9). 422. Erkelens, C. J. The Perspective Structure of Visual Space. i-Perception doi:
10 . 1177 / 2041669515613672 . http : / / journals . sagepub . com /doi/pdf/10.1177/2041669515613672 (2015) (cit. on p. 29).13. Fuchs, H., Kedem, Z. M. & Naylor, B. F. On visible surface generation by apriori tree structures in Proceedings of the 7th annual conference on Computergraphics and interactive techniques - SIGGRAPH 80 (ACM Press, 1980).doi: (cit. on p. 15).14. Gascuel, J.-D., Holzschuch, N., Fournier, G. & Péroche, B.
Fast Non-linearProjections Using Graphics Hardware in ACM Symposium on Interactive3D Graphics and Games (ACM, Redwood City, California, 2008), 107–114. doi:
10 . 1145 / 1342250 . 1342267 . http : / / maverick . inria . fr /Publications/2008/GHFP08/i3d2007.pdf (cit. on p. 35).15. Glaeser, G. & Gröller, E. Fast generation of curved perspectives for ultra-wide-angle lenses in VR applications. The Visual Computer (1999) (cit. on pp. 31, 35).16. Hueck, A.
Von den Gränzen des Sehvermögens ger, 84. http : / / hans -strasburger . userweb . mwn . de / materials / hueck _ 1840 _ ocr . pdf (Müller’s Archiv, Tartu, 1840) (cit. on p. 28).17. Konigsberg, I.
The complete film dictionary http://archive.org/details/completefilmdict00koni (Nal Penguin INC, New York, USA,1987) (cit. on p. 35).18. Littlefield, R. Theory of the No-Parallax Point in Panorama Photography. http://janrik.net/PanoPostings/NoParallaxPoint/TheoryOfTheNoParallaxPoint.pdf (2006) (cit. on p. 26).19. Miller, P. E. & Murphy, C. J. in
Equine Ophthalmology (ed Gilger, B. C.)chap. Visual perspective and field of view (Elsevier Health Sciences, St.Louis, USA, 2010) (cit. on p. 30).20. Muszyński, G., Guzek, K. & Napieralski, P.
Wide Field of View ProjectionUsing Rasterization in Multimedia and Network Information Systems (edsChoroś, K., Kopel, M., Kukla, E. & Siemiński, A.) (Springer InternationalPublishing, Cham, 2019), 586–595. doi: (cit. on p. 2).21.
NIKKOR - The Thousand and One Nights
No.6. Nikon Corp. https : / /imaging.nikon.com/history/story/0006/index.htm (cit. on p. 31).22. Newell, M. E., Newell, R. G. & Sancha, T. L.
A solution to the hidden surfaceproblem in Proceedings of the ACM annual conference on - ACM72 (ACMPress, 1972). doi: (cit. on p. 15).23. Pannini, G. P.
Interior of St. Peter’s oil on canvas. Painting. National Galleryof Art, Washington DC, USA, 1754. http : / / web . archive . org / web /20130327080328/vedutismo.net/Pannini/ (cit. on p. 35).434. Pineda, J. A parallel algorithm for polygon rasterization.
ACM SIGGRAPHComputer Graphics . https://doi.org/10.1145%2F378456.378457 (1988) (cit. on p. 7).25. Radley, A. in Cipolla-Ficarra, F. V. Handbook of Research on InteractiveInformation Quality in Expanding Social Network Communications
46 (IGIGlobal, 2015). http://books.google.com/books?id=9OyfBwAAQBAJ (cit. on p. 29).26. Rybczyński, Z. a Treatise on the Visual Image (ed Oszmiańska, H.) chap. 1.2.2. http : / / books . google . com / books ? id = mUHNAQAACAAJ (Art StationsFoundation, Poznań, Poland, 2009) (cit. on p. 30).27. Rydzak, J.
Zbig Rybczyński, Oscar-Winning Filmmaker interview. Arizona,USA, 2014. http : / / youtube . com / watch ? v = 071 _ XgZ4{H } 2I (cit. onp. 35).28. Thoby, M.
About the various projections of the photographic objective lenses.Orthographic fisheye projection http://michel.thoby.free.fr/Fisheye_history_short/Projections/Various_lens_projection.html (2019)(cit. on p. 31).29.
Todd AO High-Speed Anamorphic 35mm T1.4 at T2.0log from The UltimateAnamorphic Lens Test. ShareGrid, Inc. https://learn.sharegrid.com/sharegrid-lens-test-todd-ao-anamorphic (2020) (cit. on p. 35).30. Tyler, C. W. Straightness and the Sphere of Vision.
Perception (2009) (cit. on p. 30).31. Wade, N. J. & Hughes, P. Fooling the Eyes: Trompe LOeil and ReversePerspective.
Perception . http://wexler . free . fr / library / files / wade \ %20(1999 ) \ % 20fooling \%20the \ %20eyes . \ % 20trompe \ %20l ’ oeil \ %20and \ %20reverse \%20perspective.pdf (1999) (cit. on p. 27).32. Wang, J., Shi, F., Zhang, J. & Liu, Y. A new calibration model of cameralens distortion. Pattern Recognition . http : / / dia . fi . upm . es / ~lbaumela / Vision13 /PapersCalibracion/wang-PR208.pdf (2008) (cit. on p. 26).33. Wikipedia, contributors. Robert Barker (painter). Biography
WikimediaFoundation, Inc. http://en.wikipedia.org/w/index.php?title=Robert_Barker_(painter)&oldid=907715733 (2019) (cit.on p. 35). 44 / / Pixel step f l o a t p x s t e p ( f l o a t g r a d i e n t )3 { return clamp ( g r a d i e n t / f w i d t h ( g r a d i e n t ) +0.5 , 0 . 0 , 1 . 0 ) ; }45 / / Binary pixel step bool b p x s t e p ( f l o a t g r a d i e n t )7 { return g r a d i e n t > −f w i d t h ( g r a d i e n t ) ∗ / / Partial derivative calculation, can be saved as a texture f l o a t get_map_delta ( vec3 v i s u a l _ s p h e r e )11 {12 v i s u a l _ s p h e r e = f w i d t h ( v i s u a l _ s p h e r e ) ;13 / / Reciprocal of perspective map maximum partial derivative return / / Pixel step with derivative in texture f l o a t g p x s t e p ( f l o a t g r a d i e n t , f l o a t map_del )19 { return clamp ( g r a d i e n t ∗ map_del +0.5 , 0 . 0 , 1 . 0 ) ; }2021 / / Binary pixel step with derivative in texture bool g b p x s t e p ( f l o a t g r a d i e n t , f l o a t map_del )23 { return g r a d i e n t > −0.5/ map_del ; } Listing 1: Aliasing-free step function in GLSL.45 f l o a t dot ( vec3 v e c t o r ) { return dot ( v e c t o r , v e c t o r ) ; }23 / / Once per polygon vec4 g e t M i t e r V e c t o r ( mat3 t r i a n g l e )5 {6 / / Project polygon onto sphere / / Vertex A / / Vertex B / / Vertex C / / Get polygon sides squared vec3 b a r y c e n t e r = vec3 (13 dot ( t r i a n g l e [1] − t r i a n g l e [ 2 ] ) ,14 dot ( t r i a n g l e [2] − t r i a n g l e [ 0 ] ) ,15 dot ( t r i a n g l e [0] − t r i a n g l e [ 1 ] )16 ) ;17 / / Get circumcenter
18 b a r y c e n t e r ∗ = b a r y c e n t e r . gbr+b a r y c e n t e r . brg−b a r y c e n t e r ;19 / / Get smallest circle vector i f ( b a r y c e n t e r . s <=0.0)21 b a r y c e n t e r = 0 . 5 ∗ ( t r i a n g l e [1]+ t r i a n g l e [ 2 ] ) ;22 e l s e i f ( b a r y c e n t e r . t <=0.0)23 b a r y c e n t e r = 0 . 5 ∗ ( t r i a n g l e [2]+ t r i a n g l e [ 0 ] ) ;24 e l s e i f ( b a r y c e n t e r . p <=0.0)25 b a r y c e n t e r = 0 . 5 ∗ ( t r i a n g l e [0]+ t r i a n g l e [ 1 ] ) ;26 e l s e
27 b a r y c e n t e r = ( b a r y c e n t e r . s ∗ t r i a n g l e [0]+ b a r y c e n t e r . t ∗ t r i a n g l e [1]+ b a r y c e n t e r . p ∗ t r i a n g l e [ 2 ] )28 / ( b a r y c e n t e r . s+b a r y c e n t e r . t+b a r y c e n t e r . p ) ;2930 / / Smallest circle center vector and radius return vec4 ( normalize ( b a r y c e n t e r ) , l e n g t h ( b a r y c e n t e r ) ) ;32 } Listing 2: Polygon miter mask function in GLSL, where matrix triangle represents vertices in camera-space. 46 vec3 normal ( vec3 A, vec3 B )2 { return normalize ( c r o s s (A, B ) ) ; }34 / / Once per polygon mat3 g e t R a s t e r M a t r i x ( mat3 t r i a n g l e )6 {7 return mat3 (8 normal ( t r i a n g l e [ 0 ] , t r i a n g l e [ 1 ] ) ,9 normal ( t r i a n g l e [ 1 ] , t r i a n g l e [ 2 ] ) ,10 normal ( t r i a n g l e [ 2 ] , t r i a n g l e [ 0 ] )11 ) ;12 }1314 / / Once per fragment pixel f l o a t g e t P o l y g o n O u t l i n e ( vec3 v i s u a l _ s p h e r e , mat3 r a s t e r , vec4 m i t e r)16 {17 vec4 masks ;18 masks . rgb = v i s u a l _ s p h e r e ∗ r a s t e r ;19 masks . a = dot ( v i s u a l _ s p h e r e , m i t e r . xyz )−m i t e r .w;2021 f l o a t o u t l i n e = p x s t e p ( masks [ 0 ] ) ;22 for ( int i =1; i <4; i ++)23 o u t l i n e = min ( p x s t e p ( masks [ i ] ) , o u t l i n e ) ;2425 return o u t l i n e ;26 } Listing 3: Aliasing-free rasterization function in GLSL. Function pxstep () isdescribed in listing 1 on page 45. / / Once per region bool getPolygonRegion ( vec3 v i s u a l _ s p h e r e , mat3 r a s t e r )3 {4 vec3 masks = v i s u a l _ s p h e r e ∗ r a s t e r ;5 / / extend outline ∗ bool o u t l i n e = masks [ 0 ] > 0 . 0 ;9 for ( int i =1; i <4; i ++)10 o u t l i n e &= masks [ i ] > 0 . 0 ;1112 return o u t l i n e ;13 } Listing 4: Binary (jagged) step rasterization function in GLSL. This versionproduces aliased result, suitable for low-resolution fragment region evaluation.47 f l o a t dot ( vec3 v e c t o r ) { return dot ( v e c t o r , v e c t o r ) ; }23 / / Once per polygon vec3 getHardNormal ( mat3 t r i a n g l e )5 { return c r o s s ( t r i a n g l e [0] − t r i a n g l e [ 1 ] , t r i a n g l e [2] − t r i a n g l e [ 1 ] ) ; }67 / / Once per fragment pixel vec4 getBarCoord ( vec3 v i s u a l _ s p h e r e , mat3 t r i a n g l e , vec3 hard_normal )9 {10 / / Extend onto polygon plane intersection f l o a t depth = dot ( t r i a n g l e [ 0 ] , hard_normal ) / dot ( v i s u a l _ s p h e r e ,hard_normal ) ;12 v i s u a l _ s p h e r e ∗ = depth ;1314 for ( int i =0; i <3; i ++)15 t r i a n g l e [ i ] −= v i s u a l _ s p h e r e ;1617 return vec4 ( vec3 (18 dot ( c r o s s ( t r i a n g l e [ 2 ] , t r i a n g l e [ 1 ] ) , hard_normal ) ,19 dot ( c r o s s ( t r i a n g l e [ 0 ] , t r i a n g l e [ 2 ] ) , hard_normal ) ,20 dot ( c r o s s ( t r i a n g l e [ 1 ] , t r i a n g l e [ 0 ] ) , hard_normal )21 ) / dot ( hard_normal ) , depth ) ;22 } Listing 5: Barycentric vector function with hard normal and depth function forfragment-data interpolation in GLSL. 48 f l o a t l p x s t e p ( f l o a t s c a l a r )2 { return / / Once per polygon side vec4 g e t L i n e V e c t o r ( vec3 A, vec3 B )6 {7 / / Project vertices onto sphere vec3 l i n e _ c e n t e r = (A+B ) ∗ return vec4 ( normalize ( l i n e _ c e n t e r ) , l e n g t h ( l i n e _ c e n t e r ) ) ;11 }1213 / / Once per polygon side vec3 normal ( vec3 A, vec3 B )15 { return normalize ( c r o s s (A, B ) ) ; }1617 / / Once per line’s pixel f l o a t getLineSegment ( vec3 v i s _ s p h e r e , vec3 normal , vec4 l i n e _ v e c )19 {20 / / Great circle f l o a t l i n e = l p x s t e p ( dot ( v i s _ s p h e r e , normal ) ) ;22 / / Apply segment mask return l i n e ∗ p x s t e p ( dot ( v i s _ s p h e r e , l i n e _ v e c . xyz )−l i n e _ v e c .w) ;24 } Listing 6: Aliasing-free line segment rasterization function in GLSL. Function pxstep () can be found in listing 1 on page 45. bool g e t A l i a s e d L i n e S e g m e n t ( vec3 v i s _ s p h e r e , vec3 normal , vec4 l i n e _ v e c )2 {3 f l o a t g r e a t _ c i r c l e = dot ( v i s _ s p h e r e , normal ) ;4 / / Binary line-step operation bool l i n e = f w i d t h ( g r e a t _ c i r c l e )−abs ( 2 . 0 ∗ g r e a t _ c i r c l e ) > 0 . 0 ;6 / / Apply segment mask return l i n e && dot ( v i s _ s p h e r e , l i n e _ v e c . xyz )−l i n e _ v e c .w > 0 . 0 ;8 } Listing 7: Binary (jagged) line segment rasterization function in GLSL.49 f l o a t dot ( vec3 v e c t o r ) { return dot ( v e c t o r , v e c t o r ) ; }2 f l o a t sq ( f l o a t s c a l a r ) { return s c a l a r ∗ s c a l a r ; }3 vec3 normal ( vec3 A, vec3 B )4 { return normalize ( c r o s s (A, B ) ) ; }56 / / Once per particle’s pixel f l o a t g e t P a r t i c l e M a s k ( vec3 v i s u a l _ s p h e r e , vec4 p a r t i c l e )8 {9 return p x s t e p ( dot ( v i s u a l _ s p h e r e , p a r t i c l e . xyz )10 −s q r t (1.0 − sq ( p a r t i c l e .w) / dot ( p a r t i c l e . xyz ) ) ) ;11 }1213 / / Once per particle, per frame mat2x3 g e t P a t r i c l e T e x C o o r d M a t r i x ( vec4 p a r t i c l e )15 {16 vec3 X = normalize ( vec3 ( p a r t i c l e . z , 0 . 0 , − p a r t i c l e . x ) ) ;17 return mat2x3 (X , normal (X , p a r t i c l e . xyz ) ) ;18 }1920 / / Once per particle’s pixel vec2 g e t P a r t i c l e T e x C o o r d ( vec3 v i s u a l _ s p h e r e , mat2x3 p a r t i c l e _ m a t , vec4 p a r t i c l e )22 { return ( v i s u a l _ s p h e r e ∗ p a r t i c l e _ m a t ) ∗ l e n g t h ( p a r t i c l e . xyz ) / p a r t i c l e.w ∗ Listing 8: Particle rasterization function in GLSL. Function pxstep () can befound in listing 1 on page 45. / / Returns fragment mask occluded by render buffer mask f l o a t clipFragment ( f l o a t fragMask , f l o a t maskBuffer )3 { return min ( fragMask , 1.0 − maskBuffer ) ; }45 / / Combines current fragment mask with the buffer f l o a t combineMaskPass ( f l o a t f r a g C l i p , f l o a t maskBuffer )7 { return f r a g C l i p+maskBuffer ; }89 / / Combines current fragment depth with the buffer f l o a t combinePass ( f l o a t f r a g C l i p , f l o a t fragDepth , f l o a t d e p t h B u f f e r )11 { return f r a g C l i p ∗ fragDepth+d e p t h B u f f e r ; }1213 / / Combines current fragment texture coordinates with the buffer vec2 combinePass ( f l o a t f r a g C l i p , vec2 fragCoord , vec2 t e x B u f f e r )15 { return f r a g C l i p ∗ fragCoord+t e x B u f f e r ; }1617 / / Combines current fragment normal with the buffer vec3 combinePass ( f l o a t f r a g C l i p , vec3 fragNormal , vec3 normBuffer )19 { return f r a g C l i p ∗ fragNormal+normBuffer ; } Listing 9: Fragment data occlusion and buffer merging, for front-to-backrasterization, in GLSL. 50 / / Convert texture coordinates to screen coordinates vec2 getScreenCoord ( vec2 texCoord , f l o a t a s p e c t R a t i o , int fovType )3 {4 / / FOV type: / / 0 - horizontal / / 1 - diagonal / / 2 - vertical / / 3 - horizontal 4x3 / / Center coordinates vec2 scrCoord = texCoord ∗ / / Correct aspect ratio i f ( fovType ==0) / / Horizontal
14 scrCoord . y /= a s p e c t R a t i o ;15 e l s e / / Vertical, diagonal, or 4x3
16 {17 scrCoord . x ∗ = a s p e c t R a t i o ;18 / / Diagonal or 4x3 i f ( fovType == 1 ) / / Diagonal
20 scrCoord ∗ = i n v e r s e s q r t ( a s p e c t R a t i o ∗ a s p e c t R a t i o + 1 .0 ) ;21 e l s e i f ( fovType == 3 ) / / 4x3
22 scrCoord /= 4 . 0 / 3 . 0 ;23 }2425 return scrCoord ;26 }2728 / / Convert screen coordinates to texture coordinates vec2 getTextureCoord ( vec2 scrCoord , f l o a t a s p e c t R a t i o , int fovType )30 {31 / / FOV type: / / 0 - horizontal / / 1 - diagonal / / 2 - vertical / / 3 - horizontal 4x3 / / Map aspect ratio to square i f ( fovType ==0) / / Horizontal
39 scrCoord . y ∗ = a s p e c t R a t i o ;40 e l s e / / Vertical, diagonal, or 4x3
41 {42 scrCoord . x /= a s p e c t R a t i o ;43 / / Diagonal or 4x3 i f ( fovType == 1 ) / / Diagonal
45 scrCoord ∗ = s q r t ( a s p e c t R a t i o ∗ a s p e c t R a t i o + 1 .0 ) ;46 e l s e i f ( fovType == 3 ) / / 4x3
47 scrCoord ∗ = 4 . 0 / 3 . 0 ;48 }4950 / / Map to corner return scrCoord ∗ Listing 10: Texture to screen coordinates transformation with aspect ratio andFOV conversion, in GLSL. 51 void l i m i t s ( inout f l o a t k , inout f l o a t l , inout f l o a t s , inoutf l o a t f o v )2 {3 k = clamp ( k , − 1 . 0 , 1 . 0 ) ;4 l = clamp ( l , 0 . 0 , 1 . 0 ) ;5 s = clamp ( s , 0 . 8 , 1 . 0 ) ;6 / / Field of view in degrees f l o a t fov_max = ( k >0.0? 1 7 9 . 0 : 1 8 0 . 0 ) / max ( abs ( k ) , 0 . 5 ) ;8 f o v = clamp ( fov , 1 . 0 , fov_max ) ;9 } Listing 11: Function for range limiting of universal perspective parameters ( 𝑘 , 𝑙 , 𝑠 and Ω ) in GLSL. 52 vec2 getViewCoord ( vec2 tex_coord , f l o a t a s p e c t , int f o v _ t y p e )2 {3 t e x _ c o o r d . xy = 2 . 0 ∗ t e x _ c o o r d . s t − 1 . 0 ;4 switch ( f o v _ t y p e )5 {6 default : / / horizontal FOV (type 1) break ;9 case / / diagonal FOV (type 2)
10 t e x _ c o o r d . xy /= l e n g t h ( vec2 ( a s p e c t , 1 . 0 ) ) ;11 case / / vertical FOV (type 3) and type 2 final step
12 t e x _ c o o r d . x ∗ = a s p e c t ;13 break ;14 case / / 4x3 horizontal FOV (type 4)
15 t e x _ c o o r d . xy ∗ = vec2 ( 0 . 7 5 ∗ a s p e c t , 0 . 7 5 ) ;16 break ;17 }18 return t e x _ c o o r d ;19 }2021 vec3 g e t P e r s p e c t i v e M a p ( vec2 view_coord , f l o a t fov , f l o a t k , f l o a t l, f l o a t s )22 {23 f l o a t halfOmega = r a d i a n s ( f o v ∗ f l o a t R = l e n g t h ( vec2 ( view_coord . x , l ∗ view_coord . y ) ) ;2526 f l o a t t h e t a ;27 i f ( k > 0 . 0 ) t h e t a = atan ( tan ( k ∗ halfOmega ) ∗ R) / k ;28 e l s e i f ( k < 0 . 0 ) t h e t a = a s i n ( s i n ( k ∗ halfOmega ) ∗ R) / k ;29 e l s e t h e t a = halfOmega ∗ R ;3031 view_coord . y /= l ∗ (1.0 − s )+s ;32 return normalize (33 vec3 ( view_coord ∗ s i n ( t h e t a ) /R , cos ( t h e t a ) )34 ) ;35 } Listing 12: Visual sphere vector ̂𝑣 ∈ [−1, 1] function from texture coordinates ⃗𝑓 ∈ [0, 1] in GLSL, for universal-perspective system.53 vec3 g e t R e c t i l i n e a r M a p ( vec2 tex_coord , f l o a t a s p e c t , f l o a t fov , int f o v _ t y p e )2 {3 vec3 d i r e c t i o n = vec3 (4 2 . 0 ∗ tex_coord −1.0 , / / map to range [-1,1] ∗ r a d i a n s ( f o v ) )6 ) ;7 switch ( f o v _ t y p e )8 {9 default : / / horizontal FOV (type 1)
10 d i r e c t i o n . y /= a s p e c t ;11 break ;12 case / / diagonal FOV (type 2)
13 d i r e c t i o n . xy /= l e n g t h ( vec2 ( a s p e c t , 1 . 0 ) ) ;14 case / / vertical FOV (type 3) and type 2 final step
15 d i r e c t i o n . x ∗ = a s p e c t ;16 break ;17 }1819 return normalize ( d i r e c t i o n ) ;20 } Listing 13: Rectilinear perspective visual-sphere vector ̂𝑣 ∈ [−1, 1] from texturecoordinates ⃗𝑓 ∈ [0, 1] in GLSL. f l o a t getPanormaAspect ( f l o a t h o r i z o n t a l _ f o v , f l o a t h e i g h t )2 { return r a d i a n s ( h o r i z o n t a l _ f o v ) / h e i g h t ; }34 vec3 getPanoramaMap ( vec2 tex_coord , f l o a t h o r i z o n t a l _ f o v , f l o a t h e i g h t )5 {6 t e x _ c o o r d . xy = vec2 ( h o r i z o n t a l _ f o v , h e i g h t ) ∗ ( t e x _ c o o r d . s t − 0 . 5) ;78 return normalize ( vec3 (9 s i n ( t e x _ c o o r d . x ) ,10 t e x _ c o o r d . y ,11 cos ( t e x _ c o o r d . x )12 ) ) ;13 } Listing 14: Curved panorama perspective visual-sphere vector ̂𝑣 ∈ [−1, 1] fromtexture coordinates ⃗𝑓 ∈ [0, 1] in GLSL.54 vec4 getDomeMap ( vec2 tex_coord , f l o a t compression , f l o a t t i l t , f l o a t o f f s e t )2 {3 t e x _ c o o r d . xy = vec2 (4 2 . 0 ∗ t e x _ c o o r d . s −1.0 ,5 1.0 −2.0 ∗ t e x _ c o o r d . t6 ) ;7 f l o a t R = l e n g t h ( t e x _ c o o r d ) ;8 f l o a t t h e t a = R ∗ r a d i a n s ( compression +90.0) ;910 vec4 perspective_map = vec4 (11 t e x _ c o o r d ∗ s i n ( t h e t a ) /R ,12 cos ( t h e t a ) ,13 1.0 −R14 ) . xzyw ;15 perspective_map . a = clamp ( perspective_map . a16 / f w i d t h ( perspective_map . a ) , 0 . 0 , 1 . 0 ) ;1718 perspective_map . z += o f f s e t ;19 perspective_map . xyz = normalize ( perspective_map . xyz ) ;2021 t i l t = r a d i a n s ( t i l t ) ;22 vec2 r o t a t i o n = vec2 ( s i n ( t i l t ) , cos ( t i l t ) ) ;23 perspective_map . xyz ∗ = mat3 (24 vec3 ( 1 . 0 , 0 . 0 , 0 . 0 ) ,25 vec3 ( 0 . 0 , r o t a t i o n [ 1 ] , −r o t a t i o n [ 0 ] ) ,26 vec3 ( 0 . 0 , r o t a t i o n [ 0 ] , r o t a t i o n [ 1 ] )27 ) ;2829 return perspective_map ;30 } Listing 15: Full dome visual-sphere vector ̂𝑣 ∈ [−1, 1] from texture coordinates ⃗𝑓 ∈ [0, 1] in GLSL. vec3 getEquirectangularMap ( vec2 t e x _ c o o r d )2 {3 t e x _ c o o r d . x = 2 . 0 ∗ t e x _ c o o r d . s − 1 . 0 ;4 t e x _ c o o r d . xy ∗ = r a d i a n s ( 1 8 0 . 0 ) ;56 vec2 s i n e = s i n ( t e x _ c o o r d ) ;7 vec2 c o s i n e = cos ( t e x _ c o o r d ) ;89 return vec3 ( s i n e . x ∗ s i n e . y , −c o s i n e . y , c o s i n e . x ∗ s i n e . y ) ;10 } Listing 16: Equirectangular projection visual-sphere vector ̂𝑣 ∈ [−1, 1] fromtexture coordinates ⃗𝑓 ∈ [0, 1] in GLSL.55 f l o a t dot ( vec3 v e c t o r ) { return dot ( v e c t o r , v e c t o r ) ; }2 f l o a t sq ( f l o a t s c a l a r ) { return s c a l a r ∗ s c a l a r ; }3 f l o a t p x s t e p ( f l o a t s c a l a r )4 { return clamp ( s c a l a r / f w i d t h ( s c a l a r ) , 0 . 0 , 1 . 0 ) ; }56 vec4 getMirrorDomeMap ( vec3 mirror_normal , vec3 projector_map , mat3 p r o j e c t o r _ r o t , vec3 p r o j e c t o r _ p o s , f l o a t dome_radius , vec3 dome_pos )7 {8 vec3 r e f l e c t i o n = r e f l e c t ( projector_map , mirror_normal ) ∗ p r o j e c t o r _ r o t ;9 vec3 m i r r o r _ s u r f a c e = mirror_normal ∗ p r o j e c t o r _ r o t ;10 dome_pos −= m i r r o r _ s u r f a c e ; / / set current mirror surface as origin / / Get reflection length to dome intersection f l o a t l e n g t h = dot ( r e f l e c t i o n . xyz , dome_pos . xyz ) ;14 l e n g t h += s q r t ( dome_radius ∗ dome_radius−dot ( dome_pos . xyz−l e n g t h ∗ r e f l e c t i o n . xyz ) ) ;1516 return vec4 (17 ( l e n g t h ∗ r e f l e c t i o n . xyz−dome_pos . xyz ) / dome_radius ,18 l e n g t h+l e n g t h ( m i r r o r _ s u r f a c e . xyz−p r o j e c t o r _ p o s . xyz )19 ) ;20 }2122 vec4 processMirrorDomeMask ( vec2 tex_coord , vec3 dome_pos )23 {24 vec2 p i x S i z e = 1 . 0 / s i z e o f ( MirrorDomeMap ) ;25 vec4 mirror_dome = t e x t u r e ( MirrorDomeMap , t e x _ c o o r d ) ;2627 / / Get mirror-bounds mask f l o a t alpha = t e x t u r e ( MirrorNormalPass , t e x _ c o o r d ) . a ;29 / / Generate reflected-dome-bounds mask
30 alpha ∗ = p x s t e p ( mirror_dome . z ) ; / / half dome clip
31 alpha ∗ = p x s t e p ( mirror_dome . y+dome_pos . y ) ; / / bottom clip / / Get maximum incident length for normalization f l o a t max_incident = 0 . 0 ;35 for ( int y =0; y
43 alpha ∗ = sq ( mirror_dome .w/ max_incident ) ;4445 return vec4 ( mirror_dome . xyz , alpha ) ;46 } Listing 17: Mirror-dome visual-sphere vector ̂𝑣 ∈ [−1, 1] from mirror normal-pass, as seen through projector in GLSL.56 vec3 g e t P r o j e c t i o n M a p ( vec2 tex_coord , vec3 view_pos , mat3 view_mat )2 {3 vec3 world_pos = t e x t u r e ( WorldPos , t e x _ c o o r d ) . xyz ;4 return normalize ( world_pos−view_pos ) ∗ view_mat ;5 }67 f l o a t g e t B r i g h t n e s s M a s k ( vec2 t e x _ c o o r d )8 {9 vec2 p i x S i z e = 1 . 0 / s i z e o f ( WorldPos ) ;1011 f l o a t max_dist2 = 0 . 0 ;12 for ( int y =0; y
12 h o r i z o n t a l _ f o v ∗ = index ;13 vec2 r o t a t i o n = vec2 (14 s i n ( h o r i z o n t a l _ f o v ) ,15 cos ( h o r i z o n t a l _ f o v )16 ) ;1718 return d i r e c t i o n ∗ mat3 (19 r o t a t i o n [ 1 ] , 0 . 0 , r o t a t i o n [ 0 ] ,20 0 . 0 , 1 . 0 , 0 . 0 ,21 −r o t a t i o n [ 0 ] , 0 . 0 , r o t a t i o n [ 1 ]22 ) ;23 } Listing 19: Visual sphere vector ̂𝑣 ∈ [−1, 1] function from texture coordinates ⃗𝑓 ∈ [0, 1] in GLSL, for 𝑛 –screen-array perspective. vec3 getVRMap ( vec2 tex_coord , f l o a t ipd , f l o a t v e r t i c a l _ f o v , f l o a t k1 , f l o a t k2 , f l o a t a s p e c t )2 {3 f l o a t index = s i g n ( t e x _ c o o r d . s − 0 . 5) ;4 / / Convert to stereoscopic coordinates ∗ vec2 ( f r a c t ( 2 . 0 ∗ t e x _ c o o r d . s ) , t e x _ c o o r d . t )−1.0;6 / / Convert to view coordinates with IPD offset (in screen-width ratio) ∗ (1. 0 −2.0 ∗ ipd ) ) ∗ ∗ a s p e c t ;8 / / Apply lens distortion f l o a t R2 = dot ( tex_coord , t e x _ c o o r d ) ; / / Radius squared
10 t e x _ c o o r d ∗ = ( 1 . 0 + k1 ∗ R2+k2 ∗ R2 ∗ R2 ) / ( 1 . 0 + k1+k2 ) ; / / Verticalnormalization / / Output incident vector return normalize ( vec3 (14 tex_coord ,15 1 . 0 / tan ( r a d i a n s ( v e r t i c a l _ f o v ) ∗ Listing 20: VR visual-sphere vector ̂𝑣 ∈ [−1, 1] from texture coordinates ⃗𝑓 ∈ [0, 1] in GLSL. The ipd variable is expressed in screen-width scale.58 vec3 a p p l y L e n s D i s t o r t i o n ( vec2 tex_coord , f l o a t k1 , f l o a t k2 , f l o a t p1 , f l o a t p2 , f l o a t q1 , f l o a t q2 , f l o a t a s p e c t )2 {3 f l o a t d i a g o n a l = i n v e r s e s q r t ( a s p e c t ∗ a s p e c t + 1 . 0 ) ;4 / / Center coordinates ∗ t e x _ c o o r d . s t − 1 . 0 ;6 / / Correct aspect ∗ = a s p e c t ;8 / / Normalize diagonally ∗ = d i a g o n a l ;1011 vec2 d i s t o r t _ c o o r d = t e x _ c o o r d ;12 / / Get radius squared f l o a t R2 = dot ( tex_coord , t e x _ c o o r d ) ;14 / / Radial distortion
15 d i s t o r t _ c o o r d ∗ = k1 ∗ R2+(k2 ∗ ( R2 ∗ R2 ) +1 . 0 ) ;16 / / Thin prism distortion
17 d i s t o r t _ c o o r d += t e x _ c o o r d ∗ dot ( vec2 ( p1 , p2 ) , t e x _ c o o r d ) ;18 / / Decentering distortion
19 d i s t o r t _ c o o r d += vec2 ( q1 , q2 ) ∗ R2 ;2021 / / Lens-distorted view-coordinates return d i s t o r t _ c o o r d ;23 } Listing 21: Lens distortion function for transformation of texture coordinates ⃗𝑓 ∈ [0, 1] to view coordinates ⃗𝑓 ′ ∈ ℝ2