Controlled Flexibility

Last Updated: Mon Jul 05 2021

Development of the LogiSense Billing Platform has been guided by a few key principals. A key design criteria has been the concept of "controlled flexibility".

What is “Controlled Flexibility”?

Within the context of a software platform, the concept of "controlled flexibility" is to ensure flexibility at key strategic points within the platform while maintaining a strong, stable operating structure.

Knowing where to the measure of flexibility to apply represents the "art" of building a software platform.

This article is intended to show how controlled flexibility has been woven into the fabric of the LogiSense Billing Platform, and what better place to start than the "birthplace" of usage data within the platform - Mediation.

Mediation and Controlled Flexibility

All usage data is mediated, passing through our Data Importer Service and making its way to our database. In that passage, there are a series of steps and transformations as the initial data is translated into something that can be consumed by our API's, and subsequently, persisted to the database. This series of steps is referred to as a "pipeline" and each step is considered a "pipe". In simplified form, this pipeline looks something like this:

The incoming data is fed through several buffers. The first splits the data into "records", and the second into "layouts". After the data from the layout buffer is passed through a parser, we are now able to distinguish the individual data items (or "fields").

After this, input transforms are then applied. We then apply transformations to move the data from the input side to the target side. This is the responsibility of the "Output Map" pipe. These transformations are those on the visual map that lie between the source and the target. Here's a sample source to target "map":

This particular map transforms the incoming Name data by applying a formula to establish the target “depth” value to put in the API calls.

The final step prepares and passes the data to our API's.

As can be seen from this, considerable flexibility has already been incorporated into the pipeline with a series of available transformations. However, while these transforms can handle the majority of data manipulation needs, there may be situations where the available transforms fall short.

Looking at the pipeline, there are certain important junctures. The most critical of these is the results coming out of our "Output Map" pipe. The data here is prepared for our API calls at this point. However, what if this data needs "tweaking" or requires some complex manipulation not covered by the suite of transforms currently available? The design principal of "controlled flexibility" suggests this a place where more flexibility is warranted.

Output Map Callouts

We have the data in a form that is very close to what we need. How can we add flexibility to do any final "tweaks" of this data prior to the API calls? The answer for us was to support a "callout", a short snippet of JavaScript code that takes the output map data as input and does a final transformation on it as needed.

This makes available the full power and flexibility of JavaScript for those complicated data manipulations that cannot be handled by our standard transforms.

Other Callouts

While the "Output Map" callout is the most critical, we have added other points of interception in our pipeline. For example, the results from our record buffer are available to a callout to do more complicated validations and bypass (or "skip") the data if the validations fail.

Each situation where we have added callouts was guided by the principal of "controlled flexibility", applying extra flexibility at key points. In this way we can preserve overall integrity and stability, while also allowing flexibility in those key, strategic places.