• State Not Answered
  • Replies 5 replies
  • Subscribers 3 subscribers
  • Views 542 views
  • Users 0 members are here
Options
Related

How to remove unnecessary archive points

caffreys_col

I''ve found that one of our PI points was mis-configured (no compression or exception settings configured) and as a result I have data values every second, sometimes two values every second in the archives. When people try to retrieve data for large time periods the query will fail as the number of records is too great (we currently have the limit set to 5 million, which at 1 second scan rates equates to about 56 days of data). 

Is there a way to retrospectively apply compression settings to clean out the unwanted events from the archive? Can you re-process the archive to clean out the second value where there are 2 values in a 1 second period (this shows up as duplicate values for the same timestamp - see screenshot below).

 

Thanks

Col

 


pastedImage_1.png.png

Parents
  • 0

    Hi Colin,

     

    This is a question that actually comes up quite often, unfortunately, there is no good built-in way to "re-compress" archived PI data.  There are a few UserVoice entries regarding this, such as this one, but none have been implemented to this point.

     

    Unfortunately, the only way to truly re-compress data is to export all the tag's data somewhere else (txt file, another PI Server, etc), delete all the existing tag data, update the correct compression settings, and rewrite the data in-order to the tag again via PItoPI, UFL, PowerShell/PICONFIG, etc.

     

    I've also seen some cases where the data is "faux" re-compressed by something like the following steps:

    1. Extract all tag data to .txt file
    2. Remove all existing tag data for the PI tag (make sure to have backups first!)
    3. Have a custom app that filters unnecessary events from the txt file through custom algorithm
    4. Write the remaining values back to the PI tag

     

    I would be interested to hear if others on here have come up with any other solutions for this type of issue.

     

    --

     

    To directly answer your other question:

    "Can you re-process the archive to clean out the second value where there are 2 values in a 1 second period"

     

    Again, unfortunately the answer in this case is no.  Archive reprocessing does not remove duplicate events (same timestamp, same value, or both) or remove events if there are more than 1 in a single second.  In most cases, behavior like this could be considered data loss.  The solution to this question would follow the same recommendation as made above.

     

    Hope this helps at least clear some things up,

    [EDIT] - spelling

     

    - Adam

  • 0 in reply to Adam

    Adam - how would I use PIconfig to re-write the old data with correct data? I'm thinking that I dump the current data, calculate the correct value, then use PIconfig to update the timestamp with the new data. I've done something similar before where I upload a CSV file with timestamps and data, but I'm not sure what the piconfig commands would be to replace existing data using a CSv file.

Reply
  • 0 in reply to Adam

    Adam - how would I use PIconfig to re-write the old data with correct data? I'm thinking that I dump the current data, calculate the correct value, then use PIconfig to update the timestamp with the new data. I've done something similar before where I upload a CSV file with timestamps and data, but I'm not sure what the piconfig commands would be to replace existing data using a CSv file.

Children
No Data