With compression, an algorithm is applied to the data which stores repetitive bits of information as a “shorthand.” The receiving end can decode the “shorthand” and restore the data back to its original state. This can help make these humongous transfers shorter, and when moving exabytes, any bit helps.
In some scenarios, similar files may exist on both sides of the transfer, but changes may have been made at the source location. Incremental transfer features allow you to only send the changes, and sending a 4 MB delta instead of a multi-exabyte dataset will create an incredible difference in transfer time.
When transferring big data across the globe, connections are unfortunately bound to drop. When moving exabytes, restarting a failed transfer is quite the inconvenience to say the least. Some solutions, including FileCatalyst, have the ability to automatically perform an MD5 checksum and resume the transfer without any data loss. Clearly, this isn’t a feature you want to use often, but when it comes to the rescue it will be a feature to be grateful for.
Automation makes any task easier, and the same can be said for this project. The ability to schedule transfers that synchronize data across endpoints at set intervals can be a major time saver. Imagine you’re a scientist, and you forgot to start your little 11 exabyte transfer of collected data before arriving at the office. Talk about “hurry up and wait.”
The SKA project is certainly reaching for the stars, and it will truly change the face of science by helping answer some of mankind’s hardest questions. But, to truly make this incredible amount of data mobile, accessible, and more valuable, file transfer acceleration will be at the core of making it a possibility.
You may not be moving multiple exabytes or discovering the origins of the universe, but do you have a project that requires you to move large amounts of data? If so, take a look at the FileCatalyst Solution Suite and see how we can help your project reach the stars.