Dev’s Corner: Java 8 and the FileCatalyst API

The following article is a bit on the technical side. It is exploring an example of how our current client API meshes with Java 8. It is meant for our customers who integrate our API within their software. I am expecting the reader to be familiar with the Java language.

So after much anticipation, Java 8 is now out and today I am going to cover a pretty cool use case utilizing the new Java 8 Streaming and even a little bit of the new lambda syntax.

The guys over at Oracle realize that data sets are getting bigger and their new Java release really exemplifies that fact with the new streaming additions. Dealing with n-sized data sets, while keeping memory use under control and optimizing for multi-core machines, is not the most straightforward task and Java has streamlined the process amazingly well.

So I am going to walk through a quick bit of code that meshes the new Java 8 power with our current API, it’s basic design is to transfer a large directory of files in parallel.

Since I only want to concentrate on what Java 8 brings to the table I am going to abstract out the things that anyone familiar with our API should already know, namely client configuration, connection, and cleanup. I pass around a BlockingQueue of FCClients that basically act as a pool. I infer all actual data harvesting and monitor of the transfer in the following code snippet

Configurations and Monitoring are abstracted out

       /**
	 * See the API documentation on how to run a transfer once you have a TransferHook
	 * */
	public interface TransferRunner {
		public abstract void runTransfer(Pair<TransferHook,FCClient> clientHookPair);
	}
	/**
	 * See the API documentation for details on how to configure our FCClient objects
	 * */
	private void configConnectAndLogin(FCClient nextClient) {
		//fill in your configuration and connection methods here
	}

	public Pair<TransferHook,FCClient> initTransfer(File aFile,LinkedBlockingQueue<FCClient> clientPool) {
		TransferHook hook = null;
		Pair<TransferHook,FCClient> pair = null;
		try {
			FCClient nextClient = clientPool.take();
			configConnectAndLogin(nextClient);
			hook = nextClient.upload(aFile);
			pair = new Pair<>(hook,nextClient);
		} catch (Exception e) {
		}
		return pair;
	}

Once we have some simple code defined we can get right to transferring files to a single user account in parallel via the new Java 8 Streaming. This very basic example code demonstrates how effortless it is to set up, what would normally be, an extremely complex bit of parallel programming.

The Streaming Method of Transfer

public void simpleUpload(
			Path dir, //This would be the directory you wish to transfer
			FileCatalystFileFilter filter, //only transfer the files you want
			TransferRunner runner, //your concrete implementation of how to handle a running transfer
			LinkedBlockingQueue<FCClient> queue //a set sized pool of FCClient objects
			)	
			throws FCException, IOException {
		//create a DirectoryStream<Path> from a single directory
		DirectoryStream<Path> dirStream = Files.newDirectoryStream(dir);
		//Create a new Java 8 Stream<Path> using the spliterator() command, 
		//the boolean "true" means create this stream in parallel
		StreamSupport.stream(dirStream.spliterator(),true)
		//Since the FCCLient does not accept Path objects, 
		//simply map the Stream<Path> to a Stream<File>
		.map(p -> p.toFile())
		//Filter out any unwanted files from the stream
		.filter(f -> filter.accept(f))
		//map each File to a Pair<TransferHook,FCClient> for monitoring,
		.map(f -> initTransfer(f, queue))
		//harvest the TransferHooks for transfer statistics, 
		//once the transfer is run, return the FCClient to the queue
		.forEach(pair -> {runner.runTransfer(pair);queue.add(pair.getSecond());});
	}

So as you can see, Java 8 gives us three things here. First, the parallelism is very simplified and transparent. Second, the plug and play aspect of streaming work extremely well for different workflows. In this case we are capturing a neat way to apply our filtering although we could have used the same lambda expression to apply a filter directly to the DirectoryStream, I wanted to give a demonstration of the new streaming power. Finally we have the new Java 8 Lambda expressions. Once you get used to the syntax you will use them everywhere. At the very least it makes the code extremely readable.

In Summary

That’s all there is, normally setting up a parallel transfer system is a daunting task, this however just takes a few minutes in a very clean, easy to read way. The new Java 8 makes dealing with big data much less daunting.