David Braines

Tasking and sharing sensing assets using controlled natural language

We introduce an approach to representing intelligence, surveillance, and reconnaissance (ISR) tasks at a relatively high level in controlled natural language. We demonstrate that this facilitates both human interpretation and machine processing of tasks. More specically, it allows the automatic assignment of sensing assets to tasks, and the informed sharing of tasks between collaborating users in a coalition environment. To enable automatic matching of sensor types to tasks, we created a machine-processable knowledge representation based on the Military Missions and Means Framework (MMF), and implemented a semantic reasoner to match task types to sensor types. We combined this mechanism with a sensor-task assignment procedure based on a well-known distributed protocol for resource allocation. In this paper, we re-formulate the MMF ontology in Controlled English (CE), a type of controlled natural language designed to be readable by a native English speaker whilst representing information in a structured, unambiguous form to facilitate machine processing. We show how CE can be used to describe both ISR tasks (for example, detection, localization, or identication of particular kinds of object) and sensing assets (for example, acoustic, visual, or seismic sensors, mounted on motes or unmanned vehicles). We show how these representations enable an automatic sensor-task assignment process. Where a group of users are cooperating in a coalition, we show how CE task summaries give users in the eld a high-level picture of ISR coverage of an area of interest. This allows them to make ecient use of sensing resources by sharing tasks.

Deep learning for situational understanding

Situational understanding (SU) requires a combination of insight — the ability to accurately perceive an existing situation — and foresight — the ability to anticipate how an existing situation may develop in the future. SU involves information fusion as well as model representation and inference. Commonly, heterogenous data sources must be exploited in the fusion process: often including both hard and soft data products. In a coalition context, data and processing resources will also be distributed and subjected to restrictions on information sharing. It will often be necessary for a human to be in the loop in SU processes, to provide key input and guidance, and to interpret outputs in a way that necessitates a degree of transparency in the processing: systems cannot be “black boxes”. In this paper, we characterize the Coalition Situational Understanding (CSU) problem in terms of fusion, temporal, distributed, and human requirements. There is currently significant interest in deep learning (DL) approaches for processing both hard and soft data. We analyze the state-of-the-art in DL in relation to these requirements for CSU, and identify areas where there is currently considerable promise, and key gaps.

Agile sensor tasking for CoIST using natural language knowledge representation and reasoning

We describe a system architecture aimed at supporting Intelligence, Surveillance, and Reconnaissance (ISR) activities in a Company Intelligence Support Team (CoIST) using natural language-based knowledge representation and reasoning, and semantic matching of mission tasks to ISR assets. We illustrate an application of the architecture using a High Value Target (HVT) surveillance scenario which demonstrates semi-automated matching and assignment of appropriate ISR assets based on information coming in from existing sensors and human patrols operating in an area of interest and encountering a potential HVT vehicle. We highlight a number of key components of the system but focus mainly on the human/machine conversational interaction involving soldiers on the field providing input in natural language via spoken voice to a mobile device, which is then processed to machine-processable Controlled Natural Language (CNL) and confirmed with the soldier. The system also supports CoIST analysts obtaining real-time situation awareness on the unfolding events through fused CNL information via tools available at the Command and Control (C2). The system demonstrates various modes of operation including: automatic task assignment following inference of new high-importance information, as well as semi-automatic processing, providing the CoIST analyst with situation awareness information relevant to the area of operation.

MIPS: A service-based aid for intelligence analysis

The Management of Information Processing Services (MIPS) project has two main objectives; the notification to analysts of the arrival of relevant new information and the automatic processing of the new information. Within these objectives a number of significant challenges were addressed. To achieve the first objective, the team had to demonstrate the capability for specific analysts to be “tipped-off” in real-time that textual reports and sensor-data have been received that are relevant to their analytical tasks, including the possibility that such reports have been made available by other nations. In the case of the second objective, the team had to demonstrate the capability for the infrastructure to automatically initiate processing of input data as it arrives, consistent with satisfying the analytical goals of teams of analysts, in as an efficient a manner as possible (including the case where data is made available by more than one nation). Using the Information Fabric middleware developed as part of the International Technology Alliance (ITA) research program, the team created a service based information processing infrastructure to achieve the objectives and challenges set by the customer. The infrastructure allows existing software to be wrapped as a service and/or specially written services to be integrated with each other as well as with other ITA technologies such as the Controlled English (CE) Store or the Gaian Database. This paper will identify the difficulties in designing and implementing the MIPS infrastructure together with describing its architecture and illustrating its use with a worked example use case.

Knowledge Management for Coalition Information Sharing at the Network Edge

This article describes ongoing research on data integration and query services for knowledge management. For such services, Controlled English (CE), a human-friendly, machine-readable language, can represent information content and context.

Model-driven SOA for sensor networks

Our previous work has explored the application of enterprise middleware techniques at the edge of the network to address the challenges of delivering complex sensor network solutions over heterogeneous communications infrastructures. In this paper, we develop this approach further into a practicable, semantically rich, model-based design and analysis approach that considers the sensor network and its contained services as a service-oriented architecture. The proposed model enables a systematic approach to service composition, analysis (using domain-specific techniques), and deployment. It also enables cross intelligence domain integration to simplify intelligence gathering, allowing users to express queries in structured natural language (Controlled English).

Coalitions of things: supporting ISR tasks via Internet of Things approaches

In the wake of rapid maturing of Internet of Things (IoT) approaches and technologies in the commercial sector, the IoT is increasingly seen as a key ‘disruptive’ technology in military environments. Future operational environments are expected to be characterized by a lower proportion of human participants and a higher proportion of autonomous and semi-autonomous devices. This view is reflected in both US ‘third offset’ and UK ‘information age’ thinking and is likely to have a profound effect on how multinational coalition operations are conducted in the future. Much of the initial consideration of IoT adoption in the military domain has rightly focused on security concerns, reflecting similar cautions in the early era of electronic commerce. As IoT approaches mature, this initial technical focus is likely to shift to considerations of interactivity and policy. In this paper, rather than considering the broader range of IoT applications in the military context, we focus on roles for IoT concepts and devices in future intelligence, surveillance and reconnaissance (ISR) tasks, drawing on experience in sensor-mission resourcing and human-computer collaboration (HCC) for ISR. We highlight the importance of low training overheads in the adoption of IoT approaches, and the need to balance proactivity and interactivity (push vs pull modes). As with sensing systems over the last decade, we emphasize that, to be valuable in ISR tasks, IoT devices will need a degree of mission-awareness in addition to an ability to self-manage their limited resources (power, memory, bandwidth, computation, etc). In coalition operations, the management and potential sharing of IoT devices and systems among partners (e.g., in cross-coalition tactical-edge ISR teams) becomes a key issue due heterogeneous factors such as language, policy, procedure and doctrine. Finally, we briefly outline a platform that we have developed in order to experiment with human-IoT teaming on ISR tasks, in both physical and virtual settings.