Big data is a big trend these days, referring to the massive volume of both structured and unstructured data that is so large it is difficult to process using traditional database management tools. Data is increasing at an unprecedented rate; 40 zettabytes of data (that’s 43 trillion gigabytes) will be created by 2020, which is an increase of 300 times from 2005. Big data in the oil and gas industry controls a 10% market share in terms of revenue in the global big data market, and is expected to become the fifth largest industry in terms of its market share position in 2020.

The oil and gas sector generates various types of structured and unstructured data; for example, a typical oil rig produces about 1 TB of data per day. In order for the data to be used correctly, it needs to be sent somewhere to be analyzed. Big data solutions are required for processing and analyzing enterprise data that is too large for traditional data processing tools to manage. This is where the need for fast file transfer comes in.

Oil companies are geographically dispersed, many with dozens of locations worldwide, including remote locations where network data bandwidth is of limited throughput and suffers from high latency. Data needs to be moved quickly between the various locations and external partners. It is commonplace in the oil and gas industry to send data over a USB hard drive; however this is not ideal when the data is time sensitive, especially since courier delivery to remote locations can be extremely unreliable. Sending data by courier also presents a security risk, since the device could be stolen or passed on to third parties, leading to a loss of company information which may have a financial or reputational impact.

[accordion][pane title=”Related Articles”]

PGS: Speeding the Delivery of Big Data in Oil & Gas to 10 Gbps

Learn how FileCatalyst moves large data in the energy sector

Learn how FileCatalyst helped NICE Software with their file transfer needs.


The solution is an accelerated file transfer solution. FileCatalyst solutions deliver data securely, reliably, and quickly. Oil and gas organizations have diverse networks that consist of a collection of satellite, wireless, and terrestrial links. FileCatalyst’s software-based solution is designed to accelerate and optimize file transfers across these networks, maximizing throughput. We offer fast and reliable file transfers with on-the-fly compression and byte level incremental transfers to maximize transfer speeds of large data sets, even to remote locations on unreliable networks. We take measures to make sure data is secure by incorporating a variety of security measures into our software. Authentication against enterprise directories or database ensure data is as secure as the rest of your enterprise. Data is secured using industry standard AES encryption, while all communications are sent over a secure SSL channel. Access to the server is further secured using a series of IP filters and intrusion detection features.

We have a variety of customers in the oil and gas industry, and our President, John Tkaczewski will be representing FileCatalyst at OilComm 2015. John will be on a panel leading the workshop “How to work with latency when you must (and avoid latency when you can)”, on Thursday, November 5th from 10:30 am – 12:00 pm. OilComm is the leading forum for communications professionals in the oil and gas community, so if you are in the Houston, TX from November 4-6, 2015, don’t miss out. You can register now to attend the OilComm Expo for FREE, or to save 30% on full conference passes.