• State Verified Answer
  • Replies 3 replies
  • Subscribers 12 subscribers
  • Views 1222 views
  • Users 0 members are here
Options
Related

System Platform – Merging Two Galaxies into One (Multi-Site Deployment)

Onyinyechi Ndukwe

Hi all,

We’re consolidating two independent System Platform deployments into a single Galaxy spanning two geographically separate sites, connected via routed fibre.

Each site retains its own PLC layer, but object deployment and management will be centralised.

For those who’ve done this in production — what issues should we watch for?
(Deployment stability, latency, failover behaviour, historian strategy, alarms, etc.)


Appreciate any lessons learned.

  • +1

     We had a situation involving multiple geographic sites where the platform was deployed across Site A and Site B.

    In this architecture, an OMI client requires the MXDataProvider service to acquire data. As of today, there is no way to control or restrict which MXDataProvider node an OMI viewer connects to. Each time the user opens or closes the viewer, it may randomly connect to a different available MXDataProvider node.

    Under normal circumstances, if the viewer connects to an MXDataProvider hosted within Site A, everything functions correctly and data updates in real time. However, in some cases the viewer establishes a connection to the MXDataProvider hosted on Site B instead. When this happens, we observe noticeable delays in the data updates within OMI.

    If I recall correctly, the root cause was traced to the firewall between the two sites, which introduced latency and slowed down the data transfer when the OMI viewer was unintentionally routed to the remote MXDataProvider.

  • +1

    I've deployed this several times in the water/wastewater space.  The two BIGGEST things to consider are network robustness, latency, and hardening.

    1) Network robustness - Keep in mind that all platforms talk to each other (minimally) all the time.  I'm sure you've seen the good old "NMX heartbeat timeout of 8000ms" errors in the logger.  You'll want to scale your NMX heartbeat accordingly.  there ARE (or were) backend ways to limit the heartbeats by limiting what platforms talked to each other in the registry, but I don't know if that strategy still works in modern versions, and it constantly needs to be updated when you deploy a new machine or redeploy an old platform.   

    You want to minimize reads and writes to objects and engines across that bridge as well.  Trying to do redundancy across the two sites would probably be spotty at best. I've seen it work, but it's definitely not something I'd ever recommend.  Best bet is to have redundant pairs at each site for the appengine redundancy.  

    Separating the area models and limiting your alarm providers to only provide alarms running in each site will help lower the number of "lost alarm communications" alarms and issues with displaying alarms in clients. 

    Keep in mind that you're going to want to keep a good pipeline on that fiber - don't limit the bandwidth to something small. 

    Make sure you have bidirectional port communications across all the ports listed in the documentation.  There is a specific tech note that provides all the ports needed. 

    Historian can work fine going across the sites, the engines have bandwidth throttling you can configure, but keep in mind if your historian is in site A and the fiber goes down, Site B loses their trends.  it may be worth looking to a second historian license.  

    Keep in mind that your GR is only going to be in one place - this will cause latency in initial user log-in to the systems, and could cause windows to bind slower on the remote site, as the GR is by default used as the main directory point for the galaxy namespace.  The platforms will figure it out on their own if the GR isn't in runtime, but it does cause a slight delay, all depending on the number of attributes on a given screen. 

    That's a start - if I think of other's I'll try to add - hope fhis helps!

  • +1

    Hi, multi site system platform with a single galaxy may sound like a good idea but has several challenges. How good is your router fiber ( is it L2 or L3 ? ) im betting its L3 and there maybe issues in deployment. When its L3 generally a firewall is in place and cause havoc to system platform. Please do think about this and tread carefully