MORS logo

MORS Special Meeting on Gaming

Background for the meeting in Fairfax, Virginia, 29 September to 1 October 2015.

Analyze and Report Phase

The fifth and final phase of a wargame project is to analyze the collected data and related material and using this to produce reports of results. As indicated in the diagram this consists of two steps.

Step 14: Reviewing and Processing Data

The data that analysts may use comes from many sources. The three links below provide a few examples of how data might be collected. These examples are not exhaustive, and some will be inappropriate for certain types of gaming. Nevertheless they are offered as a launching pad for a new study.

  • Numbers from sources where quantification is appropriate
  • Feedback from players, and from others
  • Records of events and other activity from the conduct of the game

• Real-time Review. Reviewing data should be conducted in parallel with conducting the game. This level of review need not be comprehenive, however it is critical to impose quality control while the data is being collected.

• Data Quality Control. This real-time quality Some discrepacies in data collection that may be diagnosed from real-time review include:

  • observers were out of position to collect data at the time and location required
  • computer-based systems malfunctions (hardware or software problem)
  • collection procedures were inappropriate or inadequate, e.g., computer systems were overwhelmed by volume or pace of responses
  • data is inconsistent with expectations (due to bad procedures, software failure, etc.)
  • equipment fails (e.g., audio or video recording)
The data-collection malfunctions listed above can happen, and the list is not exhaustive. The sooner malfunctions are discovered, the sooner rectification can start. Because war gaming involves considerable resources, it is critical to take corrective action while data collection opportunities are still available (while the game is in progess).

• Processing Data. Given the diversity of data that may be collected in a wargame project, it would be a challenge to provide procedures that would be universally applicable. Some games will depend largely on a gist of the discussions, possibly supported by interviews with participants. Other games will be more dependent on numerical data, perhaps casualty numbers from a combat simulation or message loads on a command and control system or a communications network.

• Emerging Insights. As data review proceeds the analysis group should scan for emerging insights.

Insight – The synthesis of a set of observations that reveal a capability or a warfighting impact. Insights include new thoughts or patterns that emerge as an analysis team looks at observations and reviews them in light of a larger body of knowledge within an operational context.

Emerging insight – The evolving insughts that are producted on a daily basis during the conduct of the exercise or experiment. They are intended to capture in real time, what the analysis team is learning and are used to periodically update the Exercise Director.

Initial Insights Report (IIR) – An evolving document that should grow as events are conducted during the exercise. The intent of the IIR is to compile one comprehensive document that ties all insights together from all of the events or phases of the coalition exercise. The IIR is generally produced within 30 days of the end of the exercise or experiments.

Final Operational Assessment (OA) Report – A detailed analytical report containing the final interoperability insights and supporting evidence for the exercise or experiment, based on post-event analysis and synthesis of observations, instrumented data, surveys, and interviews obtained throughout the exercise or experiment in order to produce the interoperability insights. The final OA Report is generally produced within six months of the end of the exercise or experiment.

• Preparation for the Quick-look Report.

Step 15: Reporting Results

A Candidate Table of Contents

Most analysis organizations will have report format suitable for the results of a war game. If this is lacking the outline below can be considered, with parts that are obviously inappropriate discarded.

    Chapter 1 Introduction
    • Purpose
    • References
    • Objectives
    • Scope
    • Intent
    • Report Organization
  • Chapter 2 Literature Review
    • Relevant Past Findings
    • Parallel Activity
    • Data Sources
  • Chapter 3 Exercise Framework
    • Characteristics
    • Architecture
    • Organizational Design
    • Key Systems
    • Schedule of Events
    • Training
  • Chapter 4 Analysis Framework
    • Methods, Models, and Tools
    • Constraints, Limitations, and Assumptions
    • Analysis Products
  • Chapter 5 Analysis Results
    • Supported Insights, with summary of evidence
    • Nascent Insights, more study required
    • Unsupported Assertions, with supporting and contrary evidence
  • Chapter 6 Summary
  • Annexes
    • Participants or Participating Organizations
    • The Scenario and Vignettes
    • Backgrounds of Roles and Organizations Represented in the Scenario
    • Orders of Battle
    • Descriptions of Weapon, Sensor, and Communication Systems and of Platforms
    • Detailed Data (supporting Chapter 5)

    Other Issues

    • Data Visualization.

    • PowerPoint and Printed Reports.

    • Web Pages and Wikis.

Documenting the Progress

• Project Workbook. In times past this was typically a large binder kept in the Study Director's office for safekeeping. In modern times, it is less likely to look like a physical book. Rather it may be a folder on a file system or even a wiki. The workbook may include copies of emails that discuss aspects of the study, and certainly emails where commitments of resources have been made. Whatever the format, the Project Workbook should be a repository of all material relevant to the study.

• Study Plan. The Study Plan should be issued early in the life of the project. It should be signed off by the sponsor and will constitute a contract between the sponsor's organization and the study team. Within some organizations there may be other documents that are equivalent, e.g., a tasking directive. See more on the Study Plan in the Design Phase. When the Study Plan is issued it may not yet be clear that a war game will be part of the study. Alternatively, if it is clear that wargaming should be a major part of the study, the study plan may lay out aspects like the format of the game activities and where to find players.

• Analysis Plan. The analysis plan will go into considerably more detail on how the study will be structured and what methods, models, and tools may be used. A significant part of the Analysis Plan will be the Data Collection and Management Plan.

• Data Collection and Management Plan (DCMP). The DCMP is a flexible document usually presented in a spreadsheet format. On the left of the spreadsheet will be a decomposition of the sponsor's objectives, including issues, and sub-issues (if their is sufficient complexity to go this far). For the analysis group, there should be a row (perhaps more) for each essential element of analysis (EEA). Reading across such a row should identify what measure will be used to determine what the analysis team will be looking for to address the EEA. Depending on the complexity of the project, many other aspects may be provided in other columns in the row. The DCMP format is flexible and analysts may wish to add columns to address, for example, what sort of methods, models, and tools may be required; when and where to anticipate that the players will be dealing with the corresponding issue or sub-issue, what training may be required for observers so they will be competent to assess the player responses.

• Other Inputs. For most studies that include a war game, the above documents are the minimum that should be developed and maintained by the study team. For a small study these may be sufficient to proceed.

 • Specifications for Models and Simulation Support.

 • Data for Models, Methods, and Tools.

 • Network Diagrams for Communications.