Project Details
Description
This project looks to improve intelligence data processing and enhance situational awareness. It was awarded to Polaris Consulting, with support from the University of Portsmouth (UoP), as a result of a submission to the CDE Big Data and Autonomy call in 2015, and the work has been undertaken between January and June 2016. Polaris and UoP developed a proof-of-concept machine learning tool, to help intelligence analysts process and prioritise data in order to extract actionable intelligence. The software is based on a Dominance based Rough Set Approach (DRSA), which is a powerful, flexible and efficient machine learning technique that monitors users' behaviours in order to generate rules that can then be applied to predict future user behaviour and automate the intelligence analysis process.
The hypothesis of the project was that the DRSA tool could deliver increased situational awareness by enabling analysts to make more rapid and accurate intelligence analyses. To assess this hypothesis, a proof-of-concept DRSA tool was developed for an experiment focused on the analysis of Signals Intelligence (SIGINT) data. SIGINT was selected as the team had extensive experience of working in this area, understood the existing data burden challenges faced by the MOD SIGINT community and the SIGINT data formats were well-aligned to DRSA.
An experiment was ran at the UoP on 7th June 2016, with eleven intelligence analyst volunteers drawn from Dstl, Polaris and the UoP, reflecting a range of intelligence analysis experience and skills. The analysts were split into two groups (non-DRSA, DRSA) and given background briefings and training on their respective tools. The non-DRSA group had the same analysis tool as the DRSA group but had the DRSA capabilities hidden, to enable a comparison between the groups. A test set of SIGINT data was generated drawing on team’s experiences creating scenario data for MOD SIGINT training exercises, which was supported by ex UK SIGINT / intelligence analyst Subject Matter Experts (SMEs). The data consists of a series of SIGINT intercept reports from terrorists (Red), army/police (Green) and civilian (White), over a series of days leading up to a planned terrorist attack.
Analysts were asked to use their analysis tool to review reports, score their interest levels and extract intelligence to report within an Intelligence Summary (INTSUM) at the end of each exercise. The DRSA group was also able to take advantage of the DRSA capability, which predicted new potentially interesting reports. At the end of each exercise, analysts were asked to produce an Intelligence Summary (INTSUM) to enable a measure of intelligence. Following the experiment, additional feedback was captured with user surveys.
The hypothesis of the project was that the DRSA tool could deliver increased situational awareness by enabling analysts to make more rapid and accurate intelligence analyses. To assess this hypothesis, a proof-of-concept DRSA tool was developed for an experiment focused on the analysis of Signals Intelligence (SIGINT) data. SIGINT was selected as the team had extensive experience of working in this area, understood the existing data burden challenges faced by the MOD SIGINT community and the SIGINT data formats were well-aligned to DRSA.
An experiment was ran at the UoP on 7th June 2016, with eleven intelligence analyst volunteers drawn from Dstl, Polaris and the UoP, reflecting a range of intelligence analysis experience and skills. The analysts were split into two groups (non-DRSA, DRSA) and given background briefings and training on their respective tools. The non-DRSA group had the same analysis tool as the DRSA group but had the DRSA capabilities hidden, to enable a comparison between the groups. A test set of SIGINT data was generated drawing on team’s experiences creating scenario data for MOD SIGINT training exercises, which was supported by ex UK SIGINT / intelligence analyst Subject Matter Experts (SMEs). The data consists of a series of SIGINT intercept reports from terrorists (Red), army/police (Green) and civilian (White), over a series of days leading up to a planned terrorist attack.
Analysts were asked to use their analysis tool to review reports, score their interest levels and extract intelligence to report within an Intelligence Summary (INTSUM) at the end of each exercise. The DRSA group was also able to take advantage of the DRSA capability, which predicted new potentially interesting reports. At the end of each exercise, analysts were asked to produce an Intelligence Summary (INTSUM) to enable a measure of intelligence. Following the experiment, additional feedback was captured with user surveys.
Key findings
The main experiment findings showed that the DRSA tool made a significant difference by enabling analysts to identify a greater proportion of relevant reports (i.e. Red) and filter out irrelevant reports (e.g. White/Green). By comparison the non-DRSA analysts struggled to identify relevant reports and wasted time looking at a greater proportion and number of irrelevant reports. The INTSUM analysis showed that the DRSA group extracted more accurate intelligence, which suggested that the DRSA tool enabled analysts to perform better. This current and potential DRSA utility was further supported by the experiment analysts who provided an overwhelmingly positive response.
The primary conclusions of the experiment has shown that DRSA has promise as a method to augment the work of intelligence analysts. Although the small sample size and limited nature of a proof-of-principle experiment must be recognised, it showed that analysts using a DRSA enabled tool were able to quickly identify and prioritise high interest reports and ignore less important reports. The results also suggest that this enabled analysts to extract more useful intelligence, more quickly and gain better situational awareness. However, it is recognised that there were some limitations within the current proof-of-concept tool and further development would be needed to fully exploit its potential to improve the intelligence analysis process.
Based on the positive results from this work, it is recommended the DRSA tool is further developed and tested in a more representative environment.
The primary conclusions of the experiment has shown that DRSA has promise as a method to augment the work of intelligence analysts. Although the small sample size and limited nature of a proof-of-principle experiment must be recognised, it showed that analysts using a DRSA enabled tool were able to quickly identify and prioritise high interest reports and ignore less important reports. The results also suggest that this enabled analysts to extract more useful intelligence, more quickly and gain better situational awareness. However, it is recognised that there were some limitations within the current proof-of-concept tool and further development would be needed to fully exploit its potential to improve the intelligence analysis process.
Based on the positive results from this work, it is recommended the DRSA tool is further developed and tested in a more representative environment.
| Status | Finished |
|---|---|
| Effective start/end date | 4/01/16 → 30/06/16 |
Funding
- Defence Science and Technology Laboratory: £11,050.00
Keywords
- Military intelligence
- Intelligence analysis process
- Situational awareness
- Dominance based Rough Set Approach
Fingerprint
Explore the research topics touched on by this project. These labels are generated based on the underlying awards/grants. Together they form a unique fingerprint.