Deploying Big Data for Security

Many companies are tempted to deploy Big Data to detect fraud, instead of management systems for security information and events. The management induced by the processing of the results for traditional security information and events and logging systems, is too burdensome for most IT departments and Big Data is considered a potential savior. There are commercially available alternatives for current log management systems, or the technology can be deployed to function as a single data store for managing and enhancing security events.

By pushing this idea a step further, it is possible to address the problem of detecting and preventing persistent advanced threats using Big Data style analysis. These techniques can play a key role in early detection of threats, through more sophisticated trend analysis and examination of multiple combined data sources. It is also possible to identify anomalies by extracting functionality.

Today, log files are often ignored except in the case of an incident. Big Data provides the ability to automatically consolidate and analyze log files from multiple sources rather than in isolation. It can therefore reveal information that different log files do not provide, and possibly improve Intrusion Detection System (IDS) and Intrusion Prevention System (IPS) through continuous adjustments and effective learning of "good" and "bad" behaviors.

Integration of information provided by physical security systems, such as building access controls and even video surveillance, can also strengthen IDS and IPS systems to address external attacks and social engineering in the process of detection. This allows a much greater detection of fraudulent and criminal activities.

Organizational silos are proven to often reduce the effectiveness of security systems, so companies should be aware that the potential performance of a Big Data style analysis can also be diluted if these problems are not resolved.

At the very least, Big Data can enable much more practical and efficient implementations of the security information and events, IDS and IPS systems.