CTOlabs.

com

Recorded Future:
August 2012

Detecting and analyzing changes through space and time

An Introduction to Recorded Future for the national security community

Inside:

Background on Recorded Future Special Operations Command (SOCOM) use cases An Open Source Example

CTOlabs.com

Recorded Future: Detecting and Analyzing Changes Through Space and Time
Recorded Future is a company providing new ways of exploring what is known and projected about coming events. The analytic tools provided by Recorded Future extract meaning and relevance of information and apply this knowledge to organizational missions. This paper gives insights into how. The public cloud-based capabilities of Recorded Future are already serving commercial clients with predictive capabilities. Current use cases span from media analytics to market assessment to financial forecasts. Government missions are also being served with public cloud offerings through Recorded Futures ability to analyze and provide information on open source information. Recorded Future has a proven ability to analyze web-based information to detect changes through space and time.

The Need For Analysis Over Open Source Plus Classified

Most enterprises need more than analysis over open source. Enterprises have vast stores of internal, more private information that could also benefit from the capabilities of Recorded Future. Most of those who have seen the powerful visualizations and analytic capabilities provided by Recorded Future come away wanting these capabilities inside their enterprise where protected documents can be analyzed with these predictive tools. In most cases, the best possible solution would be a means of taking all the predictive insights harvested from the collective resources of the Internet into private enclaves or protected (even classified) networks where sensitive internal documents could also be added to the analytic process. Recorded Future now has an offering that does just that. Their platform, called Foresite, is designed to run in both private clouds and classified networks. Foresite can extract temporal signals from internal documents, third-party content and sensitive data sources. What can this mean for operational military users? Consider a Special Operations Command use-case.

A White Paper for the Government IT Community

SOCOM Mission Needs: Use cases of strategic importance

Organizations like Special Operations Command (SOCOM) frequently articulate needs for capabilities to detect and analyze changes through space and time. For example, documents such as SOCOMs 2013 Budget Highlights Document and the stated technology needs on the SOCOM.mil website call for capabilities that can help meet the following mission needs: Ability to process, display, disseminate and exploit diverse information sources and databases through the use of multi-level security systems that employ advanced data mining and data warehousing techniques. Grouping nodes, identifying local patterns, comparing and contrasting networks, groups, and individuals. Analysis of network changes through space and time with change detection techniques. Analytical technologies showing socio-cultural, economic and demographic factors. Defendable and repeatable processes, models, and measurement technologies that allow for the ability to detect changes in behavior or belief over time along with the associated factors that caused the changes. Including reporting of shifts in reaction to stimulus.

An Open Source, Self-Generated Proof Of Concept

Recorded Future and their Foresite capability show great promise in addressing mission needs like SOCOMs. For example, consider the fast open analysis their system enabled over officially released documents provided the research community by the Combating Terrorism Center (CTC). The released documents were a collection of 175 pages of Osama Bin Ladens letters discovered at his Abbottabad compound. 17 total letters were analyzed using Recorded Futures temporal analytic technology and intelligence analysis tools. The letters were treated like any other source in the Recorded Future system. Linguistic algorithms extracted a variety of data points available in the text, and then they were visualized in the Recorded Future user interface.

CTOlabs.com

Some patterns and insights became immediately apparent. For example, a network graph generated from connections found in the letters show clear focal points around topics of God, Yemen and Afghanistan:

Tools also show, rapidly, locations mentioned the most via visualizations over terrain and in text as desired:

A White Paper For The Federal IT Community

Multiple views are possible into the same dataset, and other views can extract meaning of importance to decision-makers seeking information on human relationships. For example, this view is of those individuals associated with Iran in the collected letters:

Temporal analysis over timelines is a particular strength of the Recorded Future system. This system is built to scale to the size of the Internet, but works great over smaller sets like these 175 pages as well. Looking at all events in time in these letters produces a display like the following:

CTOlabs.com

This data can be interacted with in multiple ways. Analysts are empowered with tools that let them dive into and interact with data and display correlations. What If scenarios can also be examined. The image below shows a deeper look at the years from which quite a bit of data is collected.

As you can imagine in a well architected modern system, the Recorded Future system enables data to be interlinked and cross referenced easy. Clicking on any point reveals what is known about the point and can lead to source text that analysts will want before making assessments. Information can also be extracted for use in other systems. This system was designed to be interoperable and work with existing technologies and it does that very well. Information can be exported many ways, including automated machine to machine connections or via export or via direct publication to documents and reports. This is a very easy system to export data from.

A White Paper For The Federal IT Community

Also of importance to todays analyst is an ability to find insights into the future. Where documents make reference to future events these are plotted using easy to navigate and explore visualizations. From this particular set of documents, one future reference emerged related to planning the foundation of a Muslim state.

A Sampling of Capabilities
The graphics above were produced with a small sampling of public data. Imagine the results if this was operating over all your data, plus all the data of the Internet, in ways your analysts can interact with and extract knowledge from. You can leverage a powerful temporal analytics engine designed to scale to Internet size and empower your analysts and decision makers with this predictive power.

Recorded Future Foresite Inside Your Networks

You can have this capability running inside your networks on your servers. Most small deployments of the Recorded Future system take five Virtual Machines, and these can easily scale up to meet any mission need. This architecture scales to billions and billions of records, but is simple to express and understand. Of the five Virtual Machines, two are for a data store (leveraging MongoDB), one VM is for temporal processing, clustering and scoring, and one VM is dedicated to the analytics key-value store for the UI. Another VM is for the front end.

CTOlabs.com

All common enterprise tools for data integration and ETL can be leveraged, and many are available as part of the Recorded Future deployment if desired.

The Result
With Recorded Future, some of the most awe-inspiring analytical tools and visualizations in available to humanity can now run inside your enterprise. Analysts can take advantage of the information extraction, analysis and visualization capabilities of Recorded Future to do things like search for influencers in terror networks or find the primary money laundering points of interest or extract evidence of fraud from large data stores or detect fraudulent visa applications. The architecture is modular and can be used in existing document enrichment pipelines. Data can just as easily flow in and back out using APIs. Many analytic tools are a roach motel for data it goes in but doesnt flow out in a nice way. Too many tools want to be the central data store for the organization. Not so with Recorded Future Foresite. You can use Foresite that way if you desire, but it can also participate well in your existing architecture. Analysts can also take advantage of enhanced predictive power of who will be where when so assessments and analysis can benefit from this knowledge. All the above can be done with all sources, not just open sources.

Concluding Thoughts
The analytic tools provided by Recorded Future helps extract the meaning and relevance of information and apply this knowledge to organizational missions.

Recommendation
A proof of concept can be up and running in your enterprise in a matter of days. Your analysts can be interacting with your data in new ways on very short order and you can decide the value of scaling this system up to address more of your mission needs. Recorded Future stands ready to support your mission with demos and a working prototype. Contact jason@recordedfuture.com to schedule your proof of concept.

A White Paper For The Federal IT Community

About The Author

The assessments here are from the perspective of Bob Gourley, an intelligence professional with direct and personal experience applying technology to the evaluation of our nations adversaries in operational situations. As a Naval Intelligence officer he was the first director of intelligence for the Department of Defenses Joint Task Force for Computer Network Defense, where he worked with every element of the intelligence community to enhance operational support to this emerging mission. Bob remains a student of the cyber threat. Following retirement from the Navy, Mr. Gourley was a senior executive with TRW and Northrop Grumman, and then returned to government service as the Chief Technology Officer (CTO) of the Defense Intelligence Agency (DIA). Mr. Gourley was named one of the top 25 most influential CTOs in the globe by Infoworld in 2007, and selected for AFCEAs award for meritorious service to the intelligence community in 2008. He was named by Washingtonian magazine as one of DCs Tech Titans in 2009; and one of the Top 25 Most Fascinating Communicators in Government IT by the Gov2.0 community GovFresh. Forbes named him one of the most influential in Big Data in 2012.
Bob Gourley

For More Information

If you have questions or would like to discuss this report, please contact me. As an advocate for better IT in government, I am committed to keeping the dialogue open on technologies, processes and best practices that will keep us moving forward. Contact: Bob Gourley bob@crucialpointllc.com 703-994-0549 All information/data 2012 CTOLabs.com.

CTOlabs.com