Robust method for running analysis that is dependent on two (or more) inputs having the same timestamp?

Every day we collect samples from a product line. We perform several laboratory analyses on the sample, and the results from these analyses are timestamped at 00:00:00 on the day that the sample was collected by the lab data system, which is then pushed to PI. The result from each recurring analysis is stored in a PI point.

 

The timing on when the result is available and pushed into PI can vary quite a lot, depending on availability of lab equipment etc, and various results from a given sample can arrive to PI in different orders on different days.

 

Often these results need to be combined in various ways to produce useful information. As a simplified example, say we have a wet product that we want to know the water contents of - the lab weighs the collected sample, then dries it in an oven and then weighs it again. The wet weight and dry weight results then make their way into PI into their respective PI points. The water content is then determined via a simple calculation in a PI analysis (in this example the order may be quite fixed, but assume the weights may arrive in any order with quite large delays).

 

The question we struggle with is how to trigger this analysis consistently when both inputs for a given day have arrived.

 

Since we want the information as soon as it is available and the delay is very unpredictable, it is unsatisfactory to schedule it periodically with a large enough offset to guarantee data availability.

 

Since the order of arrival is unknown, we are currently trying to trigger the analysis on both inputs and exiting on the first row if either of the two inputs doesn't have an event on the trigger time (the idea being that it will trigger twice - once when the first of the two show up where it immediately exits, and a second time when the second shows up in which case the analysis will proceed and a result would be output to a third PI point).

 

Currently we've been using something similar to this (but also checking for BadVal) to determine that both results are present:

IF (SecSinceChange('Input1') > 0) or (SecSinceChange('Input2') > 0)
THEN Exit()
ELSE ...

Followed by a line performing the actual calculation and it seems to work. A majority of the time. But quite often we get no result at all stored in the output point. Auto-recalculate is enabled for the analyses, and we've had success using the same approach on other data, so we're not really sure what the conditions are that would make this approach sometimes fail.

 

I've looked into the arrival timings in some of the data and can't see a satisfying explanation there either. If both values arrive pretty much simultaneously (sent to PI at the same second), it seems fine, but if the arrival is staggered it is a bit more inconsistent.

 

Do you have any ideas what might be going on? Is the approach bad, and what other options should we consider? Is the approach good but the choice of function for detecting simultaneous events is bad?

 

Thank you in advance,

John

Parents
  • In a perfect world, all your lab data would be pushed to PI within a fraction of a second of each other. For an event triggered analysis, the first one updated would trigger the analysis, and this again would be mere subseconds after the very first one was updated.

    But the world is not perfect. You PI Server may be busy with lots of other tasks, so it is possible for all the lab data to be pushed into PI over a few seconds, which would throw off the analysis.

    Rather than checking SecsSinceChange, I would suggest checking that either the timestamps are the same (lab data implies stepped tags so this should work), or else check to see that the Day function returns the same value since everything is stamped at midnight.

  • Hi, thank you for your answer!

     

    The duration of the analyses we run can vary quite a lot, so most of the time there will be several hours between arrivals of the input values (and often a day or two after the result timestamp itself). Our current approach works well *most* of the time even then.

     

    The intention with using SecsSinceChange was actually do to just what you suggested "checking ... that the timestamps are the same...". I'm not sure if I'm missing something about what it does (again, it works more often than not, so it's not totally off).

     

    The idea for ensuring timestamp synchronization using SecsSinceChange is like this:

    The analysis is triggered by tags 'A' and 'B'. The trigger time ("*") of the analysis will thus be exactly the timestamp of an event on one of the tags.

    If 'A' triggers the analysis, SecsSinceChange('A') should return 0.

    If 'B' already had an event on the same timestamp, SecsSinceChange('B') should also return 0.

    Thus checking that both are equal to 0 should be a viable method for ensuring that both tags have a value at "*" and that we should compute and output a result.

     

    Am I missing something obvious here? It evidently works most of the time, so the question is really why it doesn't work *all* of the time.

     

     

Reply
  • Hi, thank you for your answer!

     

    The duration of the analyses we run can vary quite a lot, so most of the time there will be several hours between arrivals of the input values (and often a day or two after the result timestamp itself). Our current approach works well *most* of the time even then.

     

    The intention with using SecsSinceChange was actually do to just what you suggested "checking ... that the timestamps are the same...". I'm not sure if I'm missing something about what it does (again, it works more often than not, so it's not totally off).

     

    The idea for ensuring timestamp synchronization using SecsSinceChange is like this:

    The analysis is triggered by tags 'A' and 'B'. The trigger time ("*") of the analysis will thus be exactly the timestamp of an event on one of the tags.

    If 'A' triggers the analysis, SecsSinceChange('A') should return 0.

    If 'B' already had an event on the same timestamp, SecsSinceChange('B') should also return 0.

    Thus checking that both are equal to 0 should be a viable method for ensuring that both tags have a value at "*" and that we should compute and output a result.

     

    Am I missing something obvious here? It evidently works most of the time, so the question is really why it doesn't work *all* of the time.

     

     

Children
No Data