Dev’s Corner: Fighting Complexity with Smart Design

by , September 30, 2013
Contributed by Jack Scuncio

You might be hearing about our new Multi-Client feature in FileCatalyst Hotfolder soon, with 3.3 coming out soon. While the description of this wonderfully powerful new product is better left in more capable hands. I speak briefly about what's under the hood. That's what really excites me in the end and as they say, write about what you know.

This new feature is a masterful showcase of two very powerful concepts in computer science. The first is Parallel Computing (PC) and the second is the finite-state machine (FSM).

We are in the age of multi-core processors and as developers, we need to take advantage. All the cores in the world will make no difference if they aren't taken into consideration when coding. If you cannot break up your work to allow those multiple cores to contribute, great potential is wasted. One way to assure those cores are working is Parallel Computing. Parallel computing is, in the most simplest terms, doing multiple things at once to complete a task. In the case of multi-file transfers, this is an obvious paradigm to follow.

Fighting Complexity with Smart Design

Fighting Complexity with Smart Design

The way we decided to achieve this was by simply pooling our FileCatalyst clients directly from our own API. Of course this adds an exponential layer of complexity to a system. To fully understand just how complicated this can make a system you have to understand the minimal impact it will have.

Let's say that an FileCatalyst client has 10 possible code paths it can follow (it is actually much much higher, our FileCatalyst client is quite complex, but 10 is easy and math is hard). If you now have two FileCatalyst clients working together on a single task, you now have 10 x 10 = 100 code paths that can be followed. Add a third client and it is 10 x 10 x 10 = 1000 code paths. We wanted the ability to use up to 100 clients to execute a single task. So in the extremely low-ball estimate of 10 code paths followed with one client, we would have a minimal of a googol of code paths that potentially can be followed with 100 clients (Go ahead, google a "googol", it's a real thing).

If you have not guessed yet, the basic problem is that the increased complexity is basically incomprehensible. The best way to describe it would be ... herding cats, all of them that ever existed.

This is where the finite-state machine comes in. A finite-state machine (FSM) or finite-state automaton (plural: automata), or simply a state machine, is a mathematical model of computation used to design both computer programs and sequential logic circuits. It is conceived as an abstract machine that can be in one of a finite number of states. The machine is in only one state at a time; the state it is in at any given time is called the current state.

This state machine's basic job is to make sure every pooled client is in the exact same state, it is enforced every time a client is put to use. By enforcing this, you take that googol of potential code paths and bring it back down to 10. From incomprehensible to manageable in one easy design pattern.

Simply amazing if you ask me. By choosing the right design for the job, we get the power of Parallel Computing without the complexity. In other words, we get to have our cake and eat it too.

Revision List

#1 on 2013-Sep-30 Mon  09:46+-14400

#2 on 2013-Sep-30 Mon  09:10+-14400

#3 on 2013-Sep-30 Mon  09:29+-14400

#4 on 2013-Sep-30 Mon  09:07+-14400

#5 on 2013-Sep-30 Mon  09:25+-14400

#6 on 2013-Sep-30 Mon  09:43+-14400

#7 on 2013-Sep-30 Mon  09:55+-14400

#8 on 2013-Sep-30 Mon  09:47+-14400

#9 on 2013-Sep-30 Mon  09:46+-14400

#10 on 2013-Sep-30 Mon  09:26+-14400

No Comments Yet

Be the first to respond!

Sorry, comments for this entry are closed at this time.