FileCatalyst » Fast File Transfer by FileCatalyst |

FileCatalyst Joins International Association of Broadcasting Manufacturers

Cara — Thu, 09 May 2013 17:25:22 +0000

FileCatalyst recently joined the International Association of Broadcasting Manufacturers, (IABM), the only international association that represents the broadcast and media technology supply industry worldwide. Controller by a 15 strong Member’s Broad, the IABM is the authoritative voice for the industry and offers member companies a range of services including market intelligence, technical services, event participation services, training, as well as other resources.

Joining 300+ worldwide members, FileCatalyst is now part of an association representing over 80% of the global broadcast technology market. Members range from small niche providers to global corporate businesses who each value different aspects of the services available from their trade association.

FileCatalyst’s accelerated file transfer technology helps end users in the broadcast space to eliminate slow or unreliable file delivery of media files which have skyrocketed in size with the adoption of 4K and HD resolutions. These resolutions go hand in hand with large format files and methods of file transfer like FTP and email are not able to keep up with these larger files, often leading to failed or never-ending transfers. Fast and reliable movement of files are crucial for these workflows, especially in an industry where fast delivery is critical.

To learn more on the IABM please visit: www.theiabm.org

REST – Why HTTP Status Codes Are Important

Marcel — Wed, 01 May 2013 14:50:45 +0000

There are many ways in which REST developers can tackle error handling. Most REST services will send some kind of error condition structure which embeds an error message describing the error and some kind of code. This is a good start.

However, for some REST services the HTTP status code is not well defined. In some cases the REST services send an OK status code of 200 regardless of whether an error was encountered or not. The problem here is that client developers have no sure way of detecting if a request was successful or if it failed. The only way the client application can detect the appropriate status is by interpreting the actual payload that was returned. This adds an unwanted dependency to client application development.

A further problem of simply returning a “200 – OK” status for all requests is that for certain requests one will not be sending any content. This can be an update, status check, or perhaps a deletion which requires no content sent back to the client application. In these cases a more suitable “204 – No Content” status should be sent.

In the case of an addition operation, such as adding a new contact in a phone list, it would be best to send a “201 – Created” status. Hence, a better approach is to set the HTTP status code as the HTTP specification describes. For instance:

The most common codes to use for any HTTP request are:
200 – OK
404 – Not found
500 – Internal Server Error

One can then expand to a more detailed set of statuses if required, such as:
201 – Created
204 – No Content
304 – Modified
400 – Bad Request
401 – Unauthorized
403 – Forbidden
501 – Not Implemented

One can see the entire list available at HTTP Status Codes

As a best practice at FileCatalyst, we do set the HTTP status code based on the request result. For example, if a request has no content the appropriate “204 – No Content” HTTP status is sent. Or if a new record is being created the appropriate “201- Created” HTTP status is sent.

We have also standardized our error structure to reflect the same structure as a FAULT SOAP service document. This was done deliberately to give SOAP developers a sense of familiarity with error handling using our REST services. Hence whenever an error condition is encountered by any of our REST services we will return the following structure:

{
 
 	 	 "fault": {
  
 	 	 	 "code": { 
 
 	 	 	 	 "value": "Sender",
  
 	 	 	 	 "subcode": { 
 
 	 	 	 	    "value": "InternalError" 
 
 	 	 	 	 }
 
 	 	    	}, 
 
 	 	 	 "reason": { 
 
 	 	 	 " text": "Internal Error Detected."
 
 	 	    	}
 
 	   	} 
 
 	 }

The above structure makes it easy for our client developers to use our REST services, knowing that all error conditions will be represented the same way. But more importantly with all errors returned to client requests, the appropriate HTTP status code is set. For instance, the above error would return an HTTP status code of “500 – Internal Server Error”.

Conclusion

Remember, the request payload is not the only important aspect of a REST service. What is also important is that when returning a client request, one must also set the HTTP status to the appropriate code which reflects the current request condition encountered. Sending “200 - OK” on every request regardless of the request content type is not acceptable.

Moving Big Data Fast: 5 Reasons Why Accelerated and Managed File Transfer Should be Part of Every Big Data Strategy

Chris — Thu, 25 Apr 2013 16:59:22 +0000

Big data is growing - in every sense of the word. As we’ve mentioned in previous blogs, the amount of data produced every day is growing exponentially. Presently, 2.5 quintillion bytes of data are created every day. And 90% of the data in the world today has been created in the last two years.(1)

Furthermore, an increasing number of companies across a variety of industries are beginning to realize the benefits of leveraging big data and adopting a big data strategy in the workplace. In a recent survey conducted by Gartner it was found that 42% of IT leaders have invested in big data, or plan to do so within 12 months.(2)

“Organizations have increased their understanding of what big data is and how it could transform the business in novel ways. The new key questions have shifted to 'What are the strategies and skills required?' and 'How can we measure and ensure our return on investment?'" said Doug Laney, research vice president at Gartner.

As Doug Laney mentioned, when implementing big data within an organization a strategy must be put in place to fully leverage its benefits. One extremely important big data strategy aspect and often overlooked is how to move this big data, as file transfer bottlenecks or failed data transfers are commonly experienced when transferring massive amounts into the terabytes or even petabytes.

Fortunately accelerated and managed file transfer is the perfect solution (pun intended) for moving big data. Accelerated and managed file transfer should be part of every big data strategy for the following 5 reasons:

Accelerated file transfer is faster than other methods: Legacy tools like FTP and SMB are not up to the challenge of handling large file sizes and quickly delivering them. In fact, moving such large volumes of data with these tools is next to impossible. Accelerated file transfer fully utilizes a network’s link speed - resulting in the fastest transfer speeds possible, even over wide transfer distances (between cities, countries, and even continents).
Data reduction capabilities: When it comes to duplication or keeping big data files in sync, certain accelerated and managed file transfer solutions can detect files and also the bytes within those files, that have changed. For examples, if only 1 file in 100 has changed and only 5% of that file has changed, only send the 5% of that 1 file would be transferred to during the sync, to dramatically reduce transfer time. And to further increase efficiency, the changed 5% detected be compressed to half the size for transfer.
Data transfer reliability: Managed file transfer solutions drastically reduce the amount of failed file transfers versus other methods, and resume their previous progress if interrupted. There is nothing worse than a 50 TB file transfer failing at 75% completion and not being able to resume due to data corruption!
Security of big data during transit: In general, all enterprise data should be secured during transit, and managed file transfer solutions typically offer industry-standard encryption to protect organizational data. Certain industries utilizing big data, like financial and medical data, place a special importance in this area and may require enhanced secured built into their tools.
Scalability for large data sets: Many older tools just weren't designed to deal with big data. For instance, many solutions will fail before a single file is transferred because handling a data set containing millions of files is beyond their capabilities. A good managed file transfer solution will be able to scale into the millions of files and have no limits on data size.

An important lesson on achieving success with big data, is that the collaboration between business and IT is essential.(3) And we believe that this is exactly where accelerated and managed file transfer comes in.

Resources

The Lighter Side of Cutting Edge Technology

Cara — Thu, 18 Apr 2013 18:41:43 +0000

Contributed by Jack Scuncio

I am approaching my first year at Unlimi-Tech Software, Inc and I've got to say a big part of this company’s success comes not just from great minds but also a great work environment. The level of professionalism exhibited by my co-workers astounds me daily, but what makes this office a pleasure to arrive at every day is our sense of fun.

That being said, we have a sense of humour (an amazing one) in not only employees, but managers as well.

If a developer breaks the build, there is no reprimand. Instead they are gleefully given a trophy that sits on their desk, showing everyone that they were indeed the last person to break the build. You do not want the trophy and you are more than happy to give it away. The end result, less broken builds, better code practices, more due diligence.

For exceptionally grave offenses, there is the dreaded "shame cake". The cake will have the offense written in delicious icing and will be served by the offending party. For the life of me, I cannot think of a repeat offender who was the recipient of a shame cake.

There is a yoyo club.

We hang out with each other at the pub for lunch.

There is a specific rule of no nerf guns (not without cause).

We go golfing, some of us wear kilts in the spirit of the sport.

We go bowling.

We have made pyramids of muffins.

Finally, if you do your job well, you earn a Hero Cookie.

In the end, I am excited to come to work every day. I love sitting down to write some new feature or make our product better. When you remove the tedious aspects of an office environment, when you replace stress with laughter, you end up with a special kind of alchemy that results in effortless excellence. Most of all, I just love hero cookies.

NAB Show 2013 Wrap Up: Exciting Announcements and New Releases for the Digital Media Industry!

Cara — Tue, 16 Apr 2013 18:15:27 +0000

Last week, the FileCatalyst Team attended NAB Show in Las Vegas. This was a great show with many new, exciting technologies announced (including our own new FileCatalyst Central and FileCatalyst Direct releases) breathing new life into the constantly evolving digital media industry.

Along with showcasing our latest FileCatalyst Central and FileCatalyst Direct releases offering extended integration features and enhanced mobile capabilities, the FileCatalyst Team also demonstrated our 10 Gbps transfers of media files. As discussed by CEO and Co-Founder Chris Bailey while being interviewed at NAB Show, FileCatalyst’s 10 Gbps file transfers address the recent shifts to HD and 4K (and even 8K) video formats, leading to larger file sizes requiring acceleration to be delivered quickly and optimize digital workflows.

During NAB Show, some competitor announcements mentioned they are now sending data at 10 Gbps on brand new chips. We’re happy to say that we’ve have been offering 10 Gbps file transfer speeds to our customers for over 1 1/2 years. Furthermore, we’ve done so using 3 year old chips (Intel or AMD) meaning we can offer our customers the same speeds with half the hardware. Providing a more in-depth look at our file transfer speeds is an independent scientific publication of our protocols by the ex-CEO of one of our competitors.

All in all, NAB Show this year was a great gathering of technology and broadcasting professionals, reflecting recent industry shifts in speaking sessions and exhibitor stands as well. Thanks to all those who stopped by the FileCatalyst booth!

FileCatalyst at NAB – Unveiling our Latest Product Releases and Showcasing the Fastest Media File Transfers

Cara — Fri, 05 Apr 2013 19:07:38 +0000

Next week the FileCatalyst Team will be heading to Las Vegas for the always entertaining and informative NAB Show. Bringing together approximately 100,000 digital media professionals the NAB Show is the essential industry event for creative inspiration and next-generation technologies to help breathe new life into content.

The FileCatalyst Team will be at Booth SL13013 providing demonstrations of our accelerated and managed file transfer solution, sending media files at blazing speeds of up to 10 Gbps. We will also be unveiling our latest software updates to FileCatalyst Direct and FileCatalyst Central, with enhanced mobile capabilities and integration features.

These v3.2 feature highlights including the following:
FileCatalyst Central v3.2 features:

Ability to completely monitor and control FileCatalyst nodes using any mobile device, with web based control panels optimized for iPads and other tablets
Mobile device capability to remotely initiate point to point file transfers from FileCatalyst HotFolder to FileCatalyst Server locations
System administrator may manage entire deployment from anywhere via tablet, with capabilities identical to traditional desktop machines

FileCatalyst Direct v3.2 features:

Extensive new API capabilities, further expanding integration possibilities for workflow providers
Advanced REST API, enabling the control all aspects of the FileCatalyst Server and HotFolder software
Extension of the FileCatalyst Android application, allowing large media files to be sent to any email address through its tight integration with FileCatalyst Workflow

Stop by the FileCatalyst booth (SL13013) if you will be in Las Vegas attending the show. The FileCatalyst Team will also have a business card draw to give away a free iPad mini to one lucky winner and host an open bar reception at the event. The open bar reception takes place from 4 to 6 pm on Wednesday April 10th and the iPad Mini winner wlil draw takes place at 5 pm on Wednesday as well, so stick around to see if you’ve won!

We look forward to seeing you there!

Pride in our work. Confidence in our performance.

Christian — Tue, 02 Apr 2013 13:30:11 +0000

Last year, the FileCatalyst team was approached with an interesting proposition.

A PhD researcher from the Anhalt University, Dmitry Kachan, wanted to test multiple transfer solutions from competing vendors as part of his work. He would take a copy of our software, run some tests to see how it performed under various network conditions, and lay down an objective comparison of various products.

Part of me wanted to say "No, thank you" - the risks are huge for a company like ours as performance is our bread and butter.

If the paper's findings are off (a misconfigured system at 10 Gbps can behave sluggishly) and these figures are published, it could harm our reputation. If the paper's author has ulterior motives, this provides an easy mark to setup our product to shine in negative light.

From speaking with customers who have evaluated multiple products, we knew indirectly how our technology stacked up to those of competitors. However, these findings had never been published and the methodology never peer reviewed.

It's one thing to claim to be the best. It's quite another to show up to the fight in a neutral ring.

To add to the risks, the paper's co-author was the ex-CEO of one of our competitors. Another concern was if the results would be an accurate reflection of our product's performance.

Although these risks were perhaps too much for a few of our competitors to take, part of me was singing a different tune. It's the voice all software developers feel after they develop a product. A product where the code runs. Fast. Part of you wants to scream out how good it is, and you want to share this with the rest of the world. We had spent months working, refining our algorithm. I knew what our software was capable of doing and just how good our solution to the problem was.

Our customers know. But many of them view us as a competitive advantage, so our achievements are not always widely published.

So we said "Yes".

To keep it fair and impartial, we maintained a hand-off approach to the software test. We were contacted on a few occasions in July, and walked the researcher through our protocol and setup much like we do with a customer needing a high speed transfer solutions (more care is required when you perform 1+Gbps transfers). The 2-4 hour process normally involves talking about tuning options in our protocol, and how to make it scream.

Then silence.

The fall development routine started to kick in, and the project more or less drifted out of mind as the daily activities kept me busy at work.

Last week, we got word the the results are being published (ICNS 2013 : The Ninth International Conference on Networking and Services) and my fears proved to be unfounded.

Out of 5 vendors tested, only 3 of them could push data past 100Mbps under even light packet loss scenarios (0.1%).

Of those three left standing.... well... the data speeds for itself.

I honestly could not be more proud of the work our team puts into our product, and the quality research and development we put into our FileCatalyst platform.

To Chris, John, Elton, John E, Marcel, Jack, Greg, and the entire testing and support team, you guys rock!

Congratulations also to Dmitry Kachan and Eduard Siemens on the paper publication, Comparison of Contemporary Solutions for High Speed Data Transport on WAN 10 Gbit/s Connections. Your research was well done, fair, and deserves a round of applause!

FileCatalyst Suite: Compliant with DPP Broadcast Industry Standards

Cara — Fri, 22 Mar 2013 15:42:38 +0000

The Digital Production Partnership (DPP) is an industry-funded non-profit partnership that advocates and promotes the many benefits associated with digital production for broadcast television in the UK. The partnership also leads the standardization of technical and metadata requirements within the UK broadcast industry aiming to simplify and optimize all stages of digital production distributed via multiple platforms. Established in 2009, the DPP is funded and led by the BBC, ITV and Channel 4.

The DPP provides guidance to broadcasters on many topics including HD and SD standards, audio loudness, and file delivery. The partnership provides a list of options that are suggested by the DPP to help producers and broadcasters maximize the potential of digital production. On its official website, the DPP lists the DPP-compliant options available to UK broadcasters for file based delivery, including the FileCatalyst suite of accelerated and managed file transfer solutions.

FileCatalyst Central v3.2: Flexible Views

Greg — Wed, 20 Mar 2013 15:40:14 +0000

As a general development principle at FileCatalyst, we are guided by what our customers need from the product. However, we still occasionally indulge ourselves and include small changes to an application based on the simple question, "What's something *I* would want to see?"

One small feature I've wanted to see since FileCatalyst Central was first published is the ability to configure a page view to suit my own needs. With the upcoming release of FileCatalyst Central 3.2, the team managed to find a bit of time to sneak in a feature that's not likely to be at the center of any press releases. While hovering over the blank space in the title bar of any panel (on desktop computers!), you will see the "crosshairs" cursor appear. Click and drag (or tap and drag for mobile), and you will see a placeholder for where the widget "was", and gray areas representing potential "targets" for dropping the widget. Each page has "half width" areas as well as top and bottom "full width" areas.

Preferences are saved as local preferences (the positions are only saved on the machine being used), and will persist between sessions (for the front-end developers out there, the feature uses the localStorage API, with fallback to cookie storage).

The use case for pages like the "Home"/overview page is fairly straightforward: move the two panels that are most important to you to the top. If you want more width per-column for any individual panel, move it to one of the full-width areas. In a few operations, the application's initial view will give you exactly what you want to see.

The future will hold further improvements to custom views, but the ability to reorder and resize presents a significant first step!

Today’s Media File Sizes – What’s Average?

Cara — Wed, 06 Mar 2013 14:22:15 +0000

Sending media files can be challenging: FTP is slow and unreliable, attachment size limits often prevent sending via email and shipping on physical storage solutions is costly and can take days to deliver.

The issues associated with transferring media files including security, reliability, efficient workflow integration, and many others. But the overriding issue is the speed of the transfer, often slowed by today’s large format media files.

For a better understanding of how much data is being moved when you’re transferring different types of media files, see the chart below for an overview of average sizes of today’s media files, by file type.

Average Media File Sizes

As shown in the chart, richer types of media (ex. videos vs. ebooks) have higher file sizes. Furthermore, newer file types (4K video files vs. HD video files) tend to also have higher file sizes. It’s also important to keep in mind that uncompressed formats are even larger than the file sizes listed here. For example, an average mp3 file is approximately 3.5 mb but an uncompressed song is closer to 50 mb in size.

While compression is useful in most cases, professionals in media typically need to work on uncompressed files during a project. As shown in the chart, richer types of media (ex. videos vs. ebooks) have higher file sizes. Furthermore, newer file types (4K video files vs. HD video files) tend to also have higher file sizes. It’s also important to keep in mind that uncompressed formats are even larger than the file sizes listed here. For example, an average mp3 file is approximately 3.5 mb but an uncompressed song is closer to 50 mb in size. While compression is useful in most cases, professionals in media typically need to work on uncompressed files during a project.

As files grow in size they become challenging to move using traditional file transfer methods. This is where file transfer acceleration comes in, turning transfers that can take hours or even days into transfers only taking a few minutes.

For more on accelerating large file transfers, see our whitepaper: Accelerating File Transfers.

Resources:

http://media.cinemasquid.com/blu-ray/blu…
http://www.makeuseof.com/answers/size-true-1080p-movie-hours-long/
http://blog.threepress.org/2009/11/16/how-big-is-the-average-epub-book/
http://wiki.answers.com/Q/What_is_the_average_size_of_an_mp3_file
http://wiki.answers.com/Q/On_average_how_many_megabytes_are_on_a_album
http://www.microscope-microscope.org/imaging/image-resolution.htm
http://www.techcentral.co.za/the-trouble-with-4k-tv/37344/