Analyze and Report Phase
The fifth and final phase of a wargame project is to analyze the collected data and related material and using this to produce reports of results. As indicated in the diagram this consists of two steps.
Step 14: Reviewing and Processing Data
The data that analysts may use comes from many sources. The three links below provide a few examples of how data might be collected. These examples are not exhaustive, and some will be inappropriate for certain types of gaming. Nevertheless they are offered as a launching pad for a new study.
- Numbers from sources where quantification is appropriate
- Feedback from players, and from others
- Records of events and other activity from the conduct of the game
• Real-time Review. Reviewing data should be conducted in parallel with conducting the game. This level of review need not be comprehenive, however it is critical to impose quality control while the data is being collected.
• Data Quality Control. This real-time quality Some discrepacies in data collection that may be diagnosed from real-time review include:
- observers were out of position to collect data at the time and location required
- computer-based systems malfunctions (hardware or software problem)
- collection procedures were inappropriate or inadequate, e.g., computer systems were overwhelmed by volume or pace of responses
- data is inconsistent with expectations (due to bad procedures, software failure, etc.)
- equipment fails (e.g., audio or video recording)
• Processing Data. Given the diversity of data that may be collected in a wargame project, it would be a challenge to provide procedures that would be universally applicable. Some games will depend largely on a gist of the discussions, possibly supported by interviews with participants. Other games will be more dependent on numerical data, perhaps casualty numbers from a combat simulation or message loads on a command and control system or a communications network.
• Emerging Insights. As data review proceeds the analysis group should scan for emerging insights.
Insight – The synthesis of a set of observations that reveal a capability or a warfighting impact. Insights include new thoughts or patterns that emerge as an analysis team looks at observations and reviews them in light of a larger body of knowledge within an operational context.
Emerging insight – The evolving insughts that are producted on a daily basis during the conduct of the exercise or experiment. They are intended to capture in real time, what the analysis team is learning and are used to periodically update the Exercise Director.
Initial Insights Report (IIR) – An evolving document that should grow as events are conducted during the exercise. The intent of the IIR is to compile one comprehensive document that ties all insights together from all of the events or phases of the coalition exercise. The IIR is generally produced within 30 days of the end of the exercise or experiments.
Final Operational Assessment (OA) Report – A detailed analytical report containing the final interoperability insights and supporting evidence for the exercise or experiment, based on post-event analysis and synthesis of observations, instrumented data, surveys, and interviews obtained throughout the exercise or experiment in order to produce the interoperability insights. The final OA Report is generally produced within six months of the end of the exercise or experiment.
• Preparation for the Quick-look Report.
Step 15: Reporting Results
A Candidate Table of Contents
Most analysis organizations will have report format suitable for the results of a war game. If this is lacking the outline below can be considered, with parts that are obviously inappropriate discarded.
Chapter 1 Introduction
- Report Organization
- Chapter 2 Literature Review
- Relevant Past Findings
- Parallel Activity
- Data Sources
- Chapter 3 Exercise Framework
- Organizational Design
- Key Systems
- Schedule of Events
- Chapter 4 Analysis Framework
- Methods, Models, and Tools
- Constraints, Limitations, and Assumptions
- Analysis Products
- Chapter 5 Analysis Results
- Supported Insights, with summary of evidence
- Nascent Insights, more study required
- Unsupported Assertions, with supporting and contrary evidence
- Chapter 6 Summary
- Participants or Participating Organizations
- The Scenario and Vignettes
- Backgrounds of Roles and Organizations Represented in the Scenario
- Orders of Battle
- Descriptions of Weapon, Sensor, and Communication Systems and of Platforms
- Detailed Data (supporting Chapter 5)
• Data Visualization.
• PowerPoint and Printed Reports.
• Web Pages and Wikis.