The military, government and other organizations are only starting to realize the full impact of big data analysis, according to Bill Franks, a prominent big data advocate who currently serves as chief analytics officer for Teradata.
Franks, author of the 2012 book Taming the Big Data Tidal Wave, sees analysis of the floods of data coming in from new sensors and other sources as offering tremendous potential for the military to add offensive and defensive capabilities. To take full advantage of that, however, analysts must take advantage of new approaches being adopted by the private sector, target organizational policies that may hold back the effectiveness of big data, and take a realistic approach to dealing with volumes of data at an unprecedented scale.
One place for the military and government to begin, Franks observed in a recent interview, is to adopt an approach that is growing in the private sector, which is that big data isn’t just a separate thing that can be analyzed on its own, but provides the most value when it is combined and mixed with other data.
“For example, you wouldn’t use sensor data from a tank alone to understand the way it is operating, but would combine it with information about who was driving the tank and what were the conditions in which the tank was operating. You need all of that information to get the full picture, and big data adds tremendous detail and context on top of the information that has been traditionally available,” Franks said.
Another trend has been the expansion of the scope of the analytic environment. “People are beginning to see the need for multiple different types of not only analytic algorithms, but in some cases the type of data is different enough that it requires a different platform for initial storage,” he explained. “For example, going through images or text at scale is much different from going through numbers at scale, which is what we traditionally have done. So there is a necessity to expand the underlying platforms to handle more types of data and analytics.”
There is also increased importance on what Franks called “discovery analytics, which is not about solving a problem that you pretty much understand, with data you understand, and applying it in a different way.
That’s much of what we’ve done over the past years, where you can have a fairly good confidence in both the effort and the outcome before you begin. “But when you are suddenly inundated with a new type of data about a problem you’ve never attacked before, a lot of time needs to be spent in discovery mode, which is about going after a broad goal without a fully formed plan as work is started,” he continued. “Start by exploring the data and figuring out the data quality issues. Once we have the data cleaned up, at what level is the data relevant to the decisions we have to make? Do we need it at the millisecond level, or can it be aggregated to the second or minute level? What problems can you apply the data to?”
One of the biggest and most widely misunderstood challenges with big data is that attacking a problem with big data for the first time is going to be more difficult than attacking a new problem with the same type of data that the analyst is used to working with, Franks contended. “That sometimes leads people to underestimate the amount of effort that that they are in for, and therefore the analysts get frustrated or in trouble because they get behind schedule.”
A key issue in this area is that many organizations have sub-optimal policies about access to data and the ease with which employees can analyze it. “Having data in a system at an organization is not the same as the people who need to analyze it having the ability to quickly analyze it,” Franks said. “There may be security concerns that prevent access, or system capacity issues, such that I don’t get enough resources to get the job done, and big data has only forced that issue more.”
In response, organizations have been upgrading underlying platforms to handle the extra volume, and putting in place new tools and approaches. At Teradata, for example, one of the key themes has been in-database analytics. “The idea is that you don’t move data out of the systems where it resides just to run an analysis, as typically occurs, but rather bring more algorithms into where the data is sitting. That provides a lot of extra scale—you’re not moving a petabyte of data from one place to another just to analyze it, but analyzing it right where it is.
“People are changing in the sense of upgrading skill sets and learning new techniques such as text and graph analytics,” he continued. “It is necessary to apply all of the new and old skills to the problems that must be solved for the organization. It’s not a challenge that can’t be overcome, but it is one that you have to be prepared for.”
Internet of Things
The emergence of the “Internet of things,” or the interconnections of sensors and other devices communicating automatically, is another factor creating new opportunities and challenges for big data analysis.
“With the data that organizations have struggled with in the past, somebody typically had to do something for that data to be generated, such as making a purchase or approving a shipment,” Franks observed. “There are only a certain number of things that an individual could do in one day that you would want to track. So while the amount of data is large, it had a limit.
“But the sensors take things up a notch, because once you turn a sensor on, it can transmit information every millisecond until someone shuts it off. You might have dozens or hundreds of those sensors within a single engine, for example. As those sensors get distributed more broadly, there are implications across the board,” he added.
On the other hand, while private industry, government agencies and the military have different needs and concerns, they are more alike than not in that they are large organizations. “In working across different types of industries, I’ve found there are certain challenges that an organization of a certain size faces, regardless of what its core mission is. As the organization grows, the inefficiencies that analytics can address rise to the level that it is worth the effort to address them. So while the military may not have much in common with a bank, they both have a huge scale, and that scale brings with it challenges in procurement, logistics and decision making,” Franks said.
“One push I’m seeing is around making analytics operational,” he said. “By that, I mean that we have spent a couple of years with organizations that are looking at the various pieces of big data that they have, trying to understand it and figure out how it can help their business. Now they’ve found ways that big data can help their business, and the next struggle is how to build an analytic process that can be embedded into their business on a daily basis. The next challenge is to translate what you learn into the way the business operates.
“Given the need to collect and use this information for an organization the size of the U.S. military, the opportunity is massive,” Franks said. “It’s not that they aren’t doing a lot today, but I think the military must continue to do more.” ♦
- Issue: 3
- Volume: 18