VPRO: Fighting Unpredictable Network Conditions at Sea using FileCatalyst
VPRO Television of the Netherlands produced a television show requiring data transfer from a sailing ship to the mainland, using a pair of VSAT domes installed on the ship and connecting via an unpredictable satellite link. Their original system was slow and unreliable. VPRO's first priority was increasing reliability of file transfer; the next was maximizing available link speed; budget was only a tertiary concern, but FileCatalyst also cost a tenth of the hardware alternative.
150 years after the publication of Darwin's book On the Origin of Species, the Clipper Stad Amsterdam, by request of the VPRO Television, sails out following the route of the HMS Beagle. Over the course of a 35-part series titled “Beagle: On the future of species,” the authentic clipper and its crew will recreate the voyage partly as a tribute to Charles Darwin but primarily as a scientific exploration, answering the questions, “Is there reason to worry about the future? Can science offer us sufficient answers to some of our most compelling questions? Are we leaving behind an inhabitable world for future generations?”
In order to meet the scientific goals of the journey as well as provide a media-rich experience to its audience, VPRO has had the ship outfitted with modern technology. From the journey's official website at http://beagle.vpro.nl/ :
Our VPRO-Beagle ship the Clipper Stad Amsterdam is an experiment in itself. This three mast sailing ship is rigged with advanced scientific measurement equipment. Scientists from various disciplines and from all over the world execute experiments on board this sailing science lab. The ship also has a built-in TV studio from which parts of the series are produced and broadcast.
Beagle has strong interactive components. All scientific findings made during the journey can be closely watched on VPRO's Beagle website, and are broadcast by both radio and television; personal journals and video logs about life on board are made by the producers, the sailing crew, and by guests on board. These are updated on a daily basis. The route of the entire voyage can be closely followed by viewers on the internet.
Digital information such as data and media files are sent over satellite, as described in the Communications Scenario below. Early in the voyage, it became evident that the original file transfer solution was not adequate. After some testing, FileCatalyst technology was adopted because of its speed, reliability, management options, pricing, and customer support.
The Communications Scenario
The retrofitted Stad Amsterdam uses a VSAT connection for both internet connection and point-to-point file transfer. A VSAT dome couldn't be attached to the masts because of its weight and because of Lloyds' Register safety regulations. A solution was engineered that uses two raised VSAT domes just behind the front mast. Sails and rigging can interfere with the dome, but if one VSAT dome is blocked, the other can take over. Given the circumstances, this provides the best possible way to connect to satellites. One exception occurs when the ship sails far from the equator with a low viewing angle to a satellite and with a very south or north heading: in that case, both VSAT domes point into the sails and rigging. In that situation, slight course alterations are needed so at least one dome can contact a satellite.
With load balancing routers and systems, the outgoing and incoming connection uses the best available signal via one of the two modems, each with its own IP address. Because a (relatively unstable) sailing vessel on high waves is considerably different than “steaming ahead” on a very well stabilized cruise liner, there could never be a 24/7 connection guarantee. In stormy weather the mechanical inertia of the motorized domes causes interruptions of the satellite link.
Apart from the internet connection, streaming video (WME) and VOIP the link also had to provide for point to point transfer of H.264 HD video files for TV-production, GPS data in XML format for the website to be updated every 5 minutes, Flash video for the website at irregular intervals and scientific data at regular intervals. At VPRO HQ those files are distributed to various destinations.
Based on calculations and testing with compression techniques it was decided that on average the bandwidth needed was 512 Kbps up and 384 Kbps down, with a weekly slot of 2 Mbps for the transfer of the high definition video files.
During the first leg of the journey, VPRO-Beagle soon experienced problems with file transfer. The original software had difficulties handling high latency (from 600 miliseconds up to 3 seconds) and also with reconnecting when switching between the two modems and IPs. The software could only generate “file transfer tunnels” of no more than 70 Kbps per file, which is normal for high-latency satellite links. An initial workaround was to cut up files manually and send those partial files simultaneously to utilize maximum bandwidth. The inefficient file transfer method resulted in serious delays in the TV production process and made it extremely difficult to update the website with content. Delays in production carry a financial cost, but more importantly the audience experience was suffering.
The challenge came in two parts: first, increase reliability and speed of ongoing transfers, while keeping a portion of the bandwidth available for activity other than file transfer. Second, to maximize bandwidth during short weekly windows of 2 Mbps connectivity so that the H.264 high-definition files could be sent back to VPRO's production facilities in a timely fashion and meet the demands of the production schedule.
Two possible solutions emerged: the FileCatalyst software solution and a very expensive change of hardware that included new switches with hardware acceleration cards both on board the clipper and at VPRO.
The FileCatalyst Solution
VPRO requested a trial version of the FileCatalyst software solution. After carefully listening to and understanding the file transfer scenario, we recommended using the FileCatalyst HotFolder application for scheduling and managing transfers to the FileCatalyst Server. For initial testing, VPRO used a satellite link with very similar conditions to those that would be faced on the VPRO-Beagle. After successful testing, FileCatalyst was quickly put into production on the ship. Francis van Gorp, technology consultant to VPRO Television had this to say:
As soon as we deployed the software we immediately saw that FileCatalyst used full bandwidth and decided to install the software on the live link to see how it behaved under extreme conditions. The guys from FileCatalyst were extremely helpful by quickly providing a new license for the ship, configuration tips, and later an online session to tune the software to our specific needs.
File transfer is no longer an issue: files get across on time because bandwidth is used to the max; when the systems routes traffic from one dome to the other transfer picks up automatically; and even under extreme weather conditions with a very unstable satellite link, FileCatalyst uses every moment of connection time to send files across.
HotFolder's bandwidth scheduler allows the system to meet the first of its goals, that being improving file transfer while not monopolizing bandwidth. By limiting the maximum transfer speed at the client side with a hard limit, there is always a portion of bandwidth being reserved for activities outside of file transfer. The scheduler itself may be set to raise or lower this amount depending on time of day as well as day of the week.
The second goal, that of maximizing the 2 Mbps “window of opportunity” is met from either the client side or the server side, and only possible because of FileCatalyst's ability to saturate the full 2 Mbps link (as mentioned earlier, the previous solution would only reach 70 Kbps). This 2 Mbps link speed may be set on the client side via the bandwidth scheduler, but administrators of the FileCatalyst Server may also set the transfer to arrive either “ASAP” or at a guaranteed delivery time. The ability to manage bandwidth from either the client or server gives greater flexibility to ad hoc transfers and the ability to accomodate rushed or emergency transfers.
After the change to FileCatalyst we have a very stable and highly automated file transfer situation, without the need to purchase additional hardware. If files don't get across, other issues than the file transfer solution are involved.
Files (including H.264 high-definition video) arrive on time, and that the production schedule does not get delayed. There is a cost savings associated with fast delivery, but the key benefit as stated by VPRO is that their audience is being treated to a more up-to-the-minute stream of information.
Although not strictly motivated by price point, FileCatalyst has also saved money for VPRO. Previously, production schedules were reworked and compressed to accommodate for delayed delivery of media, which incurred a cost in terms of human resources. Going with the FileCatalyst software-based solution was also an up-front savings, costing roughly a tenth of the hardware option.
- Bandwidth management
- Optimization of upload bandwidth usage
- Reconnection without problems even with routing changes
- Scheduled and highly automated file transfers with HotFolder
- Reliability and stability of the software
- During heavy weather conditions, even short periods of connection are used for file transfer
VPRO had a tight timeline. They needed a new solution as quickly as possible since the series itself is scheduled for a 35-episode run, and the voyage will only occur over a finite period of time. Each extra day would have meant a diminished return on their time and money investment in the larger picture. FileCatalyst places customer support as a critical priority. Quick responsiveness from our team of engineers was an influencing factor on VPRO's decision.
Pre-sales support from FileCatalyst provided us with answers we would never have found by ourselves if we only would've had the software to test. We were pointed in the right direction and were able to make a very well motivated decision. Instead of telling us how well their software performs they supported us and gave us to the opportunity to find out ourselves.
Connection issues 2 VSAT domes
One of the key features of FileCatalyst is its stability in the face of adverse network conditions such as packet loss, high RTT and frequent disconnects. Traditional file transfer applications are not equipped to handle these types of connections. When one of the VPRO-Beagle's domes is disconnected unexpectedly, a partial file is left on the server and the transfer needs to be restarted. Not only does FileCatalyst detect the change in IP address and automatically reconnect, but each time it performs checksums of the partially transferred files to ensure the integrity of the files before resuming the transfer. Even when outages last for several minutes, FileCatalyst will resume the transfer immediately when connectivity is available.