Nicholas Mastronarde

Fast Reinforcement Learning for Energy-Efficient Wireless Communication

We consider the problem of energy-efficient point-to-point transmission of delay-sensitive data (e.g., multimedia data) over a fading channel. We propose a rigorous and unified framework for simultaneously utilizing both physical-layer and system-level techniques to minimize energy consumption, under delay constraints, in the presence of stochastic and unknown traffic and channel conditions. We formulate the problem as a Markov decision process and solve it online using reinforcement learning. The advantages of the proposed online method are that i) it does not require a priori knowledge of the traffic arrival and channel statistics to determine the jointly optimal physical-layer and system-level power management strategies; ii) it exploits partial information about the system so that less information needs to be learned than when using conventional reinforcement learning algorithms; and iii) it obviates the need for action exploration, which severely limits the adaptation speed and run-time performance of conventional reinforcement learning algorithms.

Collaborative resource exchanges for peer-to-peer video streaming over wireless mesh networks

Peer-to-peer collaboration paradigms fundamentally change the passive way wireless stations currently adapt their transmission strategies to match available resources, by enabling them to proactively influence system dynamics through exchange of information and resources. In this paper, we focus on delay-sensitive multimedia transmission among multiple peers over wireless multi-hop enterprise mesh networks. We propose a distributed and efficient framework for resource exchanges that enables peers to collaboratively distribute available wireless resources among themselves based on their quality of service requirements, the underlying channel conditions, and network topology. The resource exchanges are enabled by the scalable coding of the video content and the design of cross-layer optimization strategies, which allow efficient adaptation to varying channel conditions and available resources. We compare our designed low complexity distributed resource exchange algorithms against an optimal centralized resource management scheme and show how their performance varies with the level of collaboration among the peers. We measure system utility in terms of the multimedia quality and show that collaborative approaches achieve ~50% improvement over non-collaborative approaches. Additionally, our distributed algorithms perform within 10% system utility of a centralized optimal resource management scheme. Finally, we observe 2-5 dB improvement in decoded PSNR for each peer due to the deployed cross-layer strategy

Joint Physical-Layer and System-Level Power Management for Delay-Sensitive Wireless Communications

We consider the problem of energy-efficient point-to-point transmission of delay-sensitive data (e.g., multimedia data) over a fading channel. Existing research on this topic utilizes either physical-layer centric solutions, namely power-control and adaptive modulation and coding (AMC), or system-level solutions based on dynamic power management (DPM); however, there is currently no rigorous and unified framework for simultaneously utilizing both physical-layer centric and system-level techniques to achieve the minimum possible energy consumption, under delay constraints, in the presence of stochastic and a priori unknown traffic and channel conditions. In this paper, we propose such a framework. We formulate the stochastic optimization problem as a Markov decision process (MDP) and solve it online using reinforcement learning (RL). The advantages of the proposed online method are that 1) it does not require a priori knowledge of the traffic arrival and channel statistics to determine the jointly optimal power-control, AMC, and DPM policies; 2) it exploits partial information about the system so that less information needs to be learned than when using conventional reinforcement learning algorithms; and 3) it obviates the need for action exploration, which severely limits the adaptation speed and runtime performance of conventional reinforcement learning algorithms. Our results show that the proposed learning algorithms can converge up to two orders of magnitude faster than a state-of-the-art learning algorithm for physical layer power-control and up to three orders of magnitude faster than conventional reinforcement learning algorithms.

To Relay or Not to Relay: Learning Device-to-Device Relaying Strategies in Cellular Networks

We consider a cellular network where mobile transceiver devices that are owned by self-interested users are incentivized to cooperate with each other using tokens, which they exchange electronically to “buy” and “sell” downlink relay services, thereby increasing the networks capacity compared to a network that only supports base station-to-device (B2D) communications. We investigate how an individual device in the network can learn its optimal cooperation policy online, which it uses to decide whether or not to provide downlink relay services for other devices in exchange for tokens. We propose a supervised learning algorithm that devices can deploy to learn their optimal cooperation strategies online given their experienced network environment. We then systematically evaluate the learning algorithm in various deployment scenarios. Our simulation results suggest that devices have the greatest incentive to cooperate when the network contains (i) many devices with high energy budgets for relaying, (ii) many highly mobile users (e.g., users in motor vehicles), and (iii) neither too few nor too many tokens. Additionally, within the token system, self-interested devices can effectively learn to cooperate online, and achieve up to 20 percent throughput gains on average compared to B2D communications alone, all while selfishly maximizing their own utilities.

Markov Decision Process Based Energy-Efficient On-Line Scheduling for Slice-Parallel Video Decoders on Multicore Systems

We consider the problem of energy-efficient on-line scheduling for slice-parallel video decoders on multicore systems with Dynamic Voltage Frequency Scaling (DVFS) enabled processors. In the past, scheduling and DVFS policies in multi-core systems have been formulated heuristically due to the inherent complexity of the on-line multicore scheduling problem. The key contribution of this paper is that we rigorously formulate the problem as a Markov decision process (MDP), which simultaneously takes into account the on-line scheduling and per-core DVFS capabilities; the power consumption of the processor cores and caches; and the loss tolerant and dynamic nature of the video decoder. The objective of the MDP is to minimize long-term power consumption subject to a minimum Quality of Service (QoS) constraint related to the decoders throughput. We evaluate the proposed on-line scheduling algorithm in Matlab using realistic video decoding traces generated from a cycle-accurate multiprocessor ARM simulator.

Transmitting Important Bits and Sailing High Radio Waves: A Decentralized Cross-Layer Approach to Cooperative Video Transmission

We investigate the impact of cooperative relaying on uplink multi-user (MU) wireless video transmissions. We analyze and simplify a MU Markov decision process (MDP), whose objective is to maximize the long-term sum of utilities across the video terminals in a decentralized fashion, by jointly optimizing the packet scheduling and physical layer, under the assumption that some nodes are willing to act as cooperative relays. The resulting MU-MDP is a pricing-based distributed resource allocation algorithm, where the price reflects the expected future congestion in the network. Compared to a non-cooperative setting, we observe that the resource price increases in networks supporting low transmission rates and decreases for high transmission rates. Additionally, cooperation allows users with feeble direct signals to significantly improve their video quality, with a moderate increase in total network energy consumption that is far less than the energy these nodes would require to achieve the same video quality without cooperation.

Online Reinforcement Learning for Dynamic Multimedia Systems

In our previous work, we proposed a systematic cross-layer framework for dynamic multimedia systems, which allows each layer to make autonomous and foresighted decisions that maximize the systems long-term performance, while meeting the applications real-time delay constraints. The proposed solution solved the cross-layer optimization offline, under the assumption that the multimedia systems probabilistic dynamics were known a priori, by modeling the system as a layered Markov decision process. In practice, however, these dynamics are unknown a priori and, therefore, must be learned online. In this paper, we address this problem by allowing the multimedia system layers to learn, through repeated interactions with each other, to autonomously optimize the systems long-term performance at run-time. The two key challenges in this layered learning setting are: (i) each layers learning performance is directly impacted by not only its own dynamics, but also by the learning processes of the other layers with which it interacts; and (ii) selecting a learning model that appropriately balances time-complexity (i.e., learning speed) with the multimedia systems limited memory and the multimedia applications real-time delay constraints. We propose two reinforcement learning algorithms for optimizing the system under different design constraints: the first algorithm solves the cross-layer optimization in a centralized manner and the second solves it in a decentralized manner. We analyze both algorithms in terms of their required computation, memory, and interlayer communication overheads. After noting that the proposed reinforcement learning algorithms learn too slowly, we introduce a complementary accelerated learning algorithm that exploits partial knowledge about the systems dynamics in order to dramatically improve the systems performance. In our experiments, we demonstrate that decentralized learning can perform equally as well as centralized learning, while enabling the layers to act autonomously. Additionally, we show that existing application-independent reinforcement learning algorithms, and existing myopic learning algorithms deployed in multimedia systems, perform significantly worse than our proposed application-aware and foresighted learning methods.

A Bargaining Theoretic Approach to Quality-Fair System Resource Allocation for Multiple Decoding Tasks

In this paper, we propose a new resource allocation framework for multimedia systems that perform multiple simultaneous video decoding tasks. We jointly consider the available system resources (e.g., processor cycles) and the video decoding tasks characteristics such as the sequences content, the bit-rate, and the group of pictures (GOP) structure, in order to determine a fair and optimal resource allocation. To this end, we derive a quality-complexity model that determines the quality [in terms of peak signal-to-noise ratio (PSNR)] that a task can achieve given a certain system resource allocation. We use these quality-complexity models to determine a quality-fair and Pareto-optimal resource allocation using the Kalai-Smorodinski Bargaining Solution (KSBS) from axiomatic bargaining theory. The KSBS explicitly considers the resulting multimedia quality when performing a resource allocation and distributes quality-domain penalties proportional to the difference between each video decoding tasks maximum and minimum quality requirements. We compare the KSBS with other fairness policies in the literature and find that, because it explicitly considers multimedia quality, it provides significantly fairer resource allocations in terms of the resulting PSNR compared with policies that operate solely in the resource domain. To weight the quality impact of the resource allocations to the different decoding tasks depending on application-specific requirements or user preferences, we generalize the existing KSBS solution by introducing bargaining powers based on each video sequences motion and texture characteristics.

Cross-layer Video Streaming Over 802.11e-Enabled Wireless Mesh Networks

We propose an integrated cross-layer optimization algorithm for maximizing the decoded video quality of delay-constrained streaming in a quality-of-service (QoS) enabled multi-hop wireless mesh network. The key to our algorithm is the synergistic optimization of control parameters at each node of the multi-hop network, across the protocol layers - application, network, medium access control (MAC) and physical (PHY) layers, as well as end-to-end, i.e. across the various network nodes. To drive this optimization, we assume an overlay network infrastructure, which conveys information on the conditions of each link. Quantitative results are presented that demonstrate the merits and the need for cross-layer optimization in an efficient solution for real-time video transmission using existing protocols and infrastructures

A Queuing-Theoretic Approach to Task Scheduling and Processor Selection for Video-Decoding Applications

We propose a cross-layer design for resource-constrained systems that simultaneously decode multiple video streams on multiple parallel processors, cores, or processing elements. Our proposed design explicitly considers the coder specific application characteristics such as the decoding dependencies, decoding deadlines, and distortion impacts of different video packets (e.g., frames, slices, groups of slices etc.). The key to the cross-layer design is the resource management control plane (RMCP) that coordinates the scheduling and processor selection across the active applications. The RMCP deploys a priority-queuing model that can evaluate the system congestion and predict the total expected video quality for the set of active decoding tasks. Using this model, we develop a robust distortion-and delay-aware scheduling algorithm for video packets. This algorithm aims to maximize the sum of achieved video qualities over all of the decoded video sequences. Additionally, we propose a processor selection scheme intended to minimize the delays experienced by the queued video packets. In this way, the number of missed decoding deadlines is reduced and the overall decoded video quality is increased. We compare queuing-theoretic based scheduling strategies to media agnostic scheduling strategies (i.e., earliest-deadline-first scheduling) that do not jointly consider the decoding deadlines and distortion impacts. Our results illustrate that by directly considering the video applications properties in the design of a video decoding system, significant system performance gains on the order of 4 dB peak-signal-to-noise ratio can be achieved.