Small Data Intelligence vs. Big Data Analysis & the Future of Intelligence Analytics

Introduction

“If you know the enemy and know yourself, you need not fear the result of a hundred battles. If you know yourself but not the enemy, for every victory gained you will also suffer a defeat. If you know neither the enemy nor yourself, you will succumb in every battle.” -Sun Tzu, The Art of War

This quote often serves as a credo for the U.S. Intelligence Community (IC). The IC, both the so-called “alphabet agencies” such as the CIA, FBI, and NSA, as well as the various intelligence components of the Department of Defense are constantly seeking to know the enemy. Whether it is to gain a strategic advantage in war, defend the homeland against foreign threats, or preventing terrorism, the IC uses every tool at its disposal to know and defeat the enemy.

The IC has numerous disciplines, collection methods, and tools that cumulatively collect immense amounts of raw data that it uses to piece together the enemy and their intentions. Yet there is a flaw in the system - not one due to improper practice or ignorance, but an inherent limitation in any organization. At its core, any intelligence organization is made up of and relies on people to analyze the raw data, and convert it into actionable data. This limitation is an inherent bottleneck - the amount of raw data to be analyzed is vastly disproportionate to the available manpower. This reliance on Small Data, only that which has already been analyzed by a human, limits our potential to know our adversaries.

Further, even the most studious of analysts is still imperfect. Whether you consider the effects of fatigue and being overworked, or the effects of inherent cognitive bias, or even simply overlooking crucial data, the answer remains the same. Humans simply cannot accurately analyze information on the scale increasingly needed to effectively contend with the adversaries the U.S. now faces. 

It is no surprise, then, that the U.S. and its adversaries and peers around the world are increasingly looking to technological solutions to bridge this gap. In particular, there is a worldwide push to increase the use of Big Data and Artificial Intelligence (AI) for military and national security applications, especially in intelligence analysis. For the remainder of this article, we will discuss the use of Small Data, Big Data, and efforts by the IC to incorporate AI and other technological solutions to enhance their capabilities.

DOWNLOAD to continue reading......