The reality of realtime systems

Uniformity recently deployed a new service for our clients to see their data in ‘realtime’ – or did we? Just what is ‘realtime’ and what is ‘near realtime’?

Realtime systems are generally used in anti-fraud, Industrial and IOT scenarios where discreet data points are monitored for anomolies and an action can be taken. Such scenarios require an in-flight decision to be made based on a discreet piece of data in isolation. Examples of realtime requirements would be;

  • to identify a credit card transaction where the customer’s home country is not the country of transaction and that country of transaction is on a risk register.
  • a door has been opened in an area of a building that should be empty at 3am.
  • the water level of a Dam has breached the safety limit.
  • the temperature of a nuclear centrifuge has exceeded its safe limit
  • a server CPU is at 95%

These are operational scenarios where an alert is required in real time to a system or person that is monitoring so that the appropriate action can be taken. Realtime systems have some draw backs and limitations. You can not summarise or aggregate data in real time – as there is a time and compute cost to persist and aggregate data into a set of existing running aggregations. For example if you wish to add every sale to a ‘sales today’ aggregation – and also allow for items to be removed if a refund is issued – then this becomes impossible to be truly ‘realtime’ as there is an inherent latency to compute those totals and refresh a display or send an alert. True realtime systems are difficult to add any moderate amount of calculations to in order to determine an action. In the example above, if there are 5 servers in a cluster and only one is running at 95% for a short period then it is not worth alarming a human to that. Systems like Splunk do such IT infrastructure monitoring very well in an near realtime fashion but, in general, expecting any degreee of complex calculations from a true realtime system is optimistic and will invariably be very expensive to implement.

With business data however, we often think of ‘realtime’ data as the ability to know what is going within the business ‘now’ or today. Quite often business owners and managers are blind to what is going on during the day – or sometimes the month – until data is gathered, crunched and a report is generated or the data is ingested to a Business Intelligence (BI) system for visual reporting. Such reports and BI have traditionaly been serviced by data warehouse design patterns and procedures and would historically have been capable of showing data for yesterday – or one ‘intra day’ report. Some less efficient systems could have as much as a month latency in reporting data.

In the world of ‘big data’ and AI there is almost an expectation that machines can do magic and tell us what we want to know, when we need to know it – without us asking for it! The reality is that any system needs to be carefully designed and implemented so that it satisfies what the business actually needs at a manageable cost.

Our customers have asked us if we can provide Business Intelligence reports in realtime. We can now offer what is – in technical terms – ‘near’ realtime. We provide aggregated, summary insights to how the business is performing within a latency of less than 10 seconds. Our customers are not complaining. They call this realtime, but we need to be more accurate in how we describe it to our technology friends!

If you are interested to understand how we can improve actionable insights to your business as each day unfolds then just get in touch. hello@uniformity.io