Guilherme Augusto Ferreira Lima

CÉU-MEDIA: Local Inter-Media Synchronization using CÉU

The semantics of current multimedia languages is informal and may lead to the development of ambiguous applications. In this paper we investigate the use of the synchronous language Céu for programming local multimedia applications, in particular, those applications that can be described as a set of synchronized media objects. Céu has a deterministic, concise and accurate semantics which, along with high-level programming constructs, makes the language an attractive alternative for developing multimedia applications. We also present Céu-Media, a library for programming multimedia in Céu. Céu-Media implementation guarantees that the properties of the Céu synchronous semantics are reflected in the multimedia presentation, ensuring inter-media synchronization. We compare the synchronization paradigm of Céu with those of NCL and SMIL, and examine the implementation of representative use cases.

Deepening the separation of concerns in the implementation of multimedia systems

The presentation of multimedia content requires devices with proper computational resources to guarantee intermedia synchronization requirements. When a multimedia system is executed in resource-constrained devices, some synchronization mismatches may occur. This problem becomes harder to solve when considering support for features like user interaction, systemic resilience, and distributed multimedia processing. This paper discusses the separation of concerns between multimedia presentation orchestration and multimedia content decoding and rendering. A multimedia system architecture and its implementation are also presented in this paper, where the separation of concerns goes beyond modularization by being deeply realized as process isolation. This design decision not only assures improved system resilience to the proposed solution but also enables it to be applied in distributed multimedia processing/presentations. Experiments carried out have indicated that the presentation orchestration does not require much resource, and may be run on lightweight devices. On the other hand, multimedia decoding/rendering is suitable to run on devices that are more resourceful.

Reducing the complexity of NCL player implementations

In this paper, we present an approach for reducing the complexity of NCL player implementations. This approach consists, basically, in introducing in the players architecture an initial conversion step that removes all syntactic sugar and reuse features from the source language. The output of this step, a redundancy-free version of the original input, is then fed to the player that interprets it and creates a corresponding multimedia presentation. In particular, we propose the use of the NCL Raw profile as this intermediate language. The Raw profile is an (almost) redundancy-free profile that is compatible with the NCL 3.0 EDTV (Enhanced Digital TV) profile, a property that guarantees a seamless integration with current EDTV profile implementations. The main targets of the proposed approach are NCL players running on HTML browsers. We discuss how the solutions presented by NCL4Web, WebNCL, and Ginga Plug-in can be tuned to overcome some problems pointed their authors. The same problems arise in similar contexts for other declarative languages, e.g., SMIL, and the solutions presented here can also be extended to those systems.

Two normal forms for link-connector pairs in NCL 3.0

In this paper, we investigate the problem of normal forms for links and connectors in NCL 3.0. We identify two such forms, called the First and Second Normal Forms (NF1 and NF2), in which links and connectors appear in simple terms. We also present normalization procedures (proofs), which show that for every NCL~3.0 program, there is an equivalent program in each of the forms. The mere existence of NF1 and NF2 makes the semantic analysis of programs simpler. Moreover, the symmetry exhibited by these forms suggests that the same principle of arbitrarily ordered evaluation underlies both the evaluation of link conditions and the execution of non-sequential compound actions.

Converting NCL Documents to Smix and Fixing Their Semantics and Interpretation in the Process

In this paper, we present the conversion of NCL to Smix and discuss its main implications. NCL is a declarative language for the specification of interactive multimedia presentations which was adopted by the ITU-T H.761 recommendation for interoperable IPTV services. Smix is a synchronous domain-specific language with a similar purpose, but with a simpler and more precise semantics. By implementing NCL over Smix, we bring to the former the notions of reaction and execution instants, and with them some benefits. From a practical perspective, we fix the semantics of the converted documents, get a leaner NCL player (the Smix interpreter), and simplify further conversions. From a systems-design perspective, the structured conversion of NCL to Smix helps us tame the complexity of mapping the user-oriented constructs of NCL into the machine-oriented primitives that realize them as a multimedia presentation. In the paper, we present NCL and Smix, discuss related work on document conversion, and detail the conversion process and a prototype implementation.

A memory-bounded, deterministic and terminating semantics for the synchronous programming language Céu

Céu is a synchronous programming language for embedded soft real-time systems. It focuses on control-flow safety features, such as safe shared-memory concurrency and safe abortion of lines of execution, while enforcing memory bounded, deterministic, and terminating reactions to the environment. In this work, we present a small-step structural operational semantics for Céu and a proof that reactions have the properties enumerated above: that for a given arbitrary timeline of input events, multiple executions of the same program always react in bounded time and arrive at the same final finite memory state.

Dynamic Integration of Foreign-Language Parsers into an NCL Player

We describe how we modified an NCL player to accept as input, in addition to NCL documents, Lua scripts. These Lua scripts evaluate to a table in a particular format, called NCL-ltab, which is a Lua table encoding of the NCL players internal model. One advantage of our modifications is that they allow the NCL parsing to occur in the Lua script, i.e., outside the formatter but integrated in its execution flow. The same applies for parsers of dialects of NCL or similar languages. Another advantage is that new parsers can be plugged dynamically into the formatter (if they are written in Lua or can be called from Lua). In this paper, we detail the internal model of the NCL player we are using and the NCL-ltab input format. To evaluate our proposal, we present two parser-integration use cases, one for NCL itself, using the DietNCL parsing toolkit, and another for sNCL, an alternative, user-friendlier syntax for NCL.

A graphics composition architecture for multimedia applications based on layered-depth-image

We present a graphics architecture for the seamless integration of 3DV objects into ordinary 2D/3D multimedia applications. The architecture is based on layered-depth-image (LDI) and supports the real-time rendering of multiple views for multi-view 3D displays. We also present a prototype implementation of the proposed architecture in OpenCL, and discuss experimental results. The results indicate that the proposed approach leads to better results than other state-of-art graphics architecture for multimedia application.

An Approach to Convert NCL Applications into Stereoscopic 3D

This paper presents and discusses the internal operation of NCLSC (NCL Stereo Converter): a tool to convert a 2D interactive multimedia application annotated with depth information to a stereoscopic-multimedia application. Stereoscopic-multimedia applications are those that codify both the left-eye and right-eye views, as required by stereoscopic 3D displays. NCLSC takes as input an NCL (Nested Context Language) document and outputs an NCL stereoscopic application codified in side-by-side or top-bottom format (both common input formats for 3DTV sets). NCL is the declarative language adopted in most Latin America countries for terrestrial digital TV middleware systems and the ITU-T H.761 Recommendation for IPTV services. However, the proposed approach is not restricted to NCL and can be used by other languages. The depth annotation allows for positioning each 2D graphical component in a layered (2.5D or 2D+depth) user interface. It is used by NCLSC to compute the screen parallax (offset) between the graphical elements in the left and right views of the resulting stereoscopic application. When the resulting application is presented on stereoscopic 3D displays, such screen parallax induces retinal disparity, which creates the illusion of floating flat-2D graphical elements. NCLSC does not require any additional native middleware support to run in currently available 3D-enabled TV sets. Moreover, NCLSC can adapt, at run-time, the output application to different display sizes, viewer distances, and viewer preferences, which are usually required for a proper balance between artistic effects and user experience.