Beating the News and the IARPA Project that I Could Have Been Part of -- Data Eye in the Sky

Last week, when I was in Paris, I picked up a copy of The International Herald Tribune only to find an article on a project that I had been asked by a corporation to partner on.

The article, "Government Aims to Build a Data Eye in the Sky," also appeared in The New York Times and, in it, I enjoyed reading the quotes from two of my favorite professional colleagues, Dr. Sandy Pentland of the Media Lab at MIT and Dr. Albert-Laszlo Barabasi of Northeastern University (and renowned in network science). We had hosted both of them in our INFORMS Speaker Series at UMass Amherst and Pentland also was a keynote speaker at our regional INFORMS conference at UMass last May and gave a tutorial at the SBP conference in Maryland last March that I served as the tutorial chair of.

The project is a three-year experiment, and is to begin in April. The title is rather dry -- Open Source Indicators (OSI) program. The RFP was sent out by the Intelligence Advanced Research Projects Activity, which us part of the Office of the Director of National Intelligence and which is the funding agency for this project.

I very much enjoyed the conversations with the researchers in the company that approached me and I found the challenges very intriguing. Essentially, in a nutshell, the goal is to develop a platform by which one can "beat the news" in terms of certain events. The focus, interestingly, was to be on Latin America (all 21 countries, no less).

According to the program overview in the RFP:

Many significant societal events are preceded and/or followed by population-level changes in communication, consumption, and movement. Some of these changes may be indirectly observable from publicly available data, such as web search queries, blogs, micro-blogs, internet traffic, financial markets, traffic webcams, Wikipedia edits, and many others. Published research has found that some of these data sources are individually useful in the early detection of events such as disease outbreaks, political crises, and macroeconomic trends. For example, much has been published on extracting indicators useful in forecasting political unrest from news feeds; public sentiment has been inferred from blogs and microblogs; and disease outbreaks have been detected from web search queries. In addition, Government-funded programs such as the Integrated Crisis Early Warning System (ICEWS), the Political Instability Task Force (PITF), and the Aggregative Contingent Estimation (ACE) Program have focused on methods to forecast pre-defined events. But few methods have been developed for anticipating or detecting unexpected events by fusing publicly available data of multiple types from multiple sources.

OSI aims to fill this gap by developing methods for continuous, automated analysis of publicly available data1 in order to anticipate and/or detect significant societal events, such as political crises, humanitarian crises, mass violence, riots, mass migrations, disease outbreaks, economic instability, resource shortages, and responses to natural disasters. Open Source Indicators (OSI) performers will develop methods that ―beat the news by fusing early indicators of events.

Performers will be evaluated on the basis of warnings that they deliver about real-world events.

After spending time thinking through how I and a student and perhaps colleague could contribute, ultimately, I decided that the time-frame for the preparation of the proposal was too tight, given that it was the beginning of the new academic year and it was not entirely clear to me how big my role was to be. There were regular project deliverables and it seemed that there were some fundamental research questions that one should answer before working on deliverable software.

Imagine my surprise when The New York Times highlighted this project!

It will be interesting to see which organizations/universities/corporations end up with the winning proposals.