19 Jan Data Pipes by Actio® – The Back Story
In 2015 Jim and I started an engagement at a hospital. It seemed simple enough, extract data from 3 different systems and report on that data. At the time Dashboards had been around for many years and visualisation was a big topic but we both knew that most of the effort in a “Dashboard” project was below the surface.
When we started the project I asked Jim “how many times do you think people have written custom code for data integration project?” His response was “millions”, “at least”. So we decided to pause and consider a better way. This lead to what we codenamed “Act 1” as every play starts with Act 1 and every software project starts with the data. Act 2 was what you did with the data when you had it, etc. you get the idea.
In time “Act 1” became “Data Pipes” as we realised we were streaming data and from one place to another with 1 or more “transforms” within the stream. We streamed data from one system or cloud or file or one database to another system or file or cloud or database. Our goal with Data Pipes was to make data movement and manipulation faster and less mysterious. We were seeking to create tools that made the data more visible right from the start of any project and enable collaboration between the people wanting the data and the people providing the data.
Making data movement and manipulation faster speaks for itself. The project happens faster, the benefits are realised faster and the customer is happy faster. Making it less mysterious needs some explanation. In most projects involving data, the business person who actually wants to data is often kept at arm’s length, not intentionally but it is a reality. The project usually flows like this. Business person expresses a need (in writing or verbally). Techo interprets the needs, examines the existing data (usually by looking at files or databases or understanding web service or by looking “under the covers” of a system. The techo determines what is or is not possible and then responds to the business person with an estimate. Once the estimate is accepted, the magic happens ***, in reality, the techo head for the coding cave. The techo codes up a solution and like magic or sometimes tragically the data appears in some format the business user can review it. The first pass is almost never what the business person had intended and the process needs to go through a number of iterations until they get what they want.
There are a number of problems with this approach.
- When you create a software program it is brittle. You need to compile it, test it, run it, debug it, etc. Then you need to deploy it. If there is an error you need to change the code, recompile it, redeploy it and start testing again. This all takes a lot of time and does not allow the Business person to see what is going on. So the techo might be on the wrong track and invest a significant portion of their time into retrieving, validating and cleaning data that is not necessary or irrelevant.
- After this has been built it has to run and it may or may not run successfully.
- f the source or destination for the stream changes you need to go and find the techo that coded it and get them to change it as they are usually the only person who actually know what is going on in that program.
- Even if nothing has changed, how do you know all the data you were expecting travelled from the source to the destination. The source or destination may be unavailable.
This is all basic stuff but it hasn’t been solved.
Data Pipes by Actio® makes the process of onboarding, previewing and monitoring data flows faster and more visible. It provides a platform for IT to engage the business within minutes and ensure that the project is headed in the right direction.