• State Not Answered
  • Replies 13 replies
  • Subscribers 3 subscribers
  • Views 350 views
  • Users 0 members are here
Options
Related

Weird behavior processing IEnumerable<AFEventFrame>

andrew_Inwood

I have some code where I search for some event frames, then try to process the results.

The issue I have is that processing the IEnumerable results is very, very slow. It consistently takes my application time from 5 seconds to 45 seconds.

 

Here is the code:

        public void LoadClosedTasks(CurrentShift currentShift)
        {
            //CodeTimer LoadClosedTasksTimer = new CodeTimer();
            if (currentShift == null) return;
            List<AFSearchTokenBase> tokens = AppliedSearchTokens.ToList();                  //Copy the list
            tokens.RemoveAll(x => x.Filter == AFSearchFilter.InProgress);                   //Remove tokens for "In Progress"
            tokens.Add(new AFSearchValueToken("|COMPLETED_SHIFT_ID", currentShift.ShiftID.ToString()));
            IEnumerable<AFEventFrame> RetrievedFrames = null;
            using (AFEventFrameSearch search = new AFEventFrameSearch(LogDatabase, "baseSearch", tokens))
            {
                //DateTime searchStart = DateTime.Now;
                RetrievedFrames = search.FindObjects(0, true, 0);
            }
            ConcurrentBag<OperatorTask> createdFrames = new ConcurrentBag<OperatorTask>();  //Add to concurrent bag to avoid multi-thread issues.

            return;

            Parallel.ForEach(RetrievedFrames, frame =>
                {
                    //OperatorTask frameInstance = new OperatorTask();
                    //frameInstance.PopulateFromFrame(MainForm, frame, this);
                    //createdFrames.Add(frameInstance);

                });
            }

For info, this routine is called using an await Task.Run.

You can see where I place the "return" statement in the code, to bypass the Parallel.ForEach() block. With return there, the application loads (doing lots of other stuff) in 4.5 seconds. If I remove the return statement, the application takes 45 seconds. And this is with the code body in the Parallel.ForEach block commented out, as it shows above! It is not the code in the block that is slow, because there is no code running and I still get the problem.

I also tried a synchronous ForEach across the collection. Same issue.

This suggests to me that the simple act of enumerating the returned search is taking 40 seconds. (Timing it confirms that).

 

So... how many elements is it iterating?

 

Glad you asked. The search results are returning exactly 1 result!

 

Any idea on what is happening here? Search.FindObjects does have LoadFull set to true, but switching it to false doesn't help. Setting the page size doesn't help. I really am at a loss on how to fix this.

Parents
  • 0

    "The search returns very fast." This is not correct. Rather, the search BLOCK of code that you have executes very fast, but all you have really done is the overhead of merely defining a search.  YOU HAVE NOT ACTUALLY SEARCHED ON ANYTHING. An AFSearch is not being performed until the underlying client AF SDK code triggers an RPC to the AF Server, where the SQL Server hosting the PIFD database will actually search its SQL tables for event frame records. With your original code, no RPC's are issued, thus no search is made.

     

    AFSearch is keenly aware that it may make many RPC calls, which generally are considered expensive in terms of time. Therefore, AFSearch was intentionally designed to be lazy.  It defers on making an RPC until you absolutely request data, e.g with a foreach, a ToList (which itself calls a foreach), or something like search.GetTotalCount(). As is, the AFElementSearch.FindObjects returns a fairly minimal IEnumerable<AFEventFrame> object. Note the only method required by IEnumerable is GetEnumerator. There most likely is an internal page of items included with FindObjects result, but without enumerating over those items, that page is EMPTY. Which is why your code has the illusion of working extremely fast.  No search has actually been performed and you have an empty page of items.

     

    Also, a full load for event frames can be quite HEAVY. If you require client-side filtering, then you absolutely should perform a full load. But loading an event frame into memory does not just load the AFAttributes. It loads other child properties. And if the event frame was created by an analysis, it loads the analyses as well. That's what I mean by HEAVY.

     

    If it were me trying to trouble shoot, the first thing I want to know is how many items pass the server-side filter? Or to put that another way: how many event frames are being sent to me fully-loaded so that client-side filtering can be performed? I may be expecting a final result of 10 items, but what if 20K are being sent for client-side filtering? That's a big performance killer. If you have a large number of items being sent to the client, then you should try to refine the search to perform better filtering server-side to reduce what is sent to the client.

     

    But let's say that only 20 items pass server-side filtering. That should be a quick enough operation and should be fairly fast on the client. If this is what is happening to you, then all I can relate is 1 very rare case from a friend of mine.  

     

    His event frame search was expecting less than 5 items but was taking over a minute. No matter what we did to the code, including using a skinny search, it still was abnormally slow. The problem was completely external to AF and the PIFD. His PIFD used a shared TempDB and another rogue database on that SQL Server had written 30 million records to a temp table. Once his SQL Server admin cleaned up and truncated the TempDB, his AFSearch dropped down to less than 2 seconds. I do not know what steps are required to clean this up on the SQL Server side, so you may want to work with your own SQL Server admin.

     

    Also, I may be wrong but I am under the impression that you can defined a dedicated TempDB (e.g. "PIFD_TempDB") for just your PIFD, so that you are insulated from this happening again in the future.

Reply
  • 0

    "The search returns very fast." This is not correct. Rather, the search BLOCK of code that you have executes very fast, but all you have really done is the overhead of merely defining a search.  YOU HAVE NOT ACTUALLY SEARCHED ON ANYTHING. An AFSearch is not being performed until the underlying client AF SDK code triggers an RPC to the AF Server, where the SQL Server hosting the PIFD database will actually search its SQL tables for event frame records. With your original code, no RPC's are issued, thus no search is made.

     

    AFSearch is keenly aware that it may make many RPC calls, which generally are considered expensive in terms of time. Therefore, AFSearch was intentionally designed to be lazy.  It defers on making an RPC until you absolutely request data, e.g with a foreach, a ToList (which itself calls a foreach), or something like search.GetTotalCount(). As is, the AFElementSearch.FindObjects returns a fairly minimal IEnumerable<AFEventFrame> object. Note the only method required by IEnumerable is GetEnumerator. There most likely is an internal page of items included with FindObjects result, but without enumerating over those items, that page is EMPTY. Which is why your code has the illusion of working extremely fast.  No search has actually been performed and you have an empty page of items.

     

    Also, a full load for event frames can be quite HEAVY. If you require client-side filtering, then you absolutely should perform a full load. But loading an event frame into memory does not just load the AFAttributes. It loads other child properties. And if the event frame was created by an analysis, it loads the analyses as well. That's what I mean by HEAVY.

     

    If it were me trying to trouble shoot, the first thing I want to know is how many items pass the server-side filter? Or to put that another way: how many event frames are being sent to me fully-loaded so that client-side filtering can be performed? I may be expecting a final result of 10 items, but what if 20K are being sent for client-side filtering? That's a big performance killer. If you have a large number of items being sent to the client, then you should try to refine the search to perform better filtering server-side to reduce what is sent to the client.

     

    But let's say that only 20 items pass server-side filtering. That should be a quick enough operation and should be fairly fast on the client. If this is what is happening to you, then all I can relate is 1 very rare case from a friend of mine.  

     

    His event frame search was expecting less than 5 items but was taking over a minute. No matter what we did to the code, including using a skinny search, it still was abnormally slow. The problem was completely external to AF and the PIFD. His PIFD used a shared TempDB and another rogue database on that SQL Server had written 30 million records to a temp table. Once his SQL Server admin cleaned up and truncated the TempDB, his AFSearch dropped down to less than 2 seconds. I do not know what steps are required to clean this up on the SQL Server side, so you may want to work with your own SQL Server admin.

     

    Also, I may be wrong but I am under the impression that you can defined a dedicated TempDB (e.g. "PIFD_TempDB") for just your PIFD, so that you are insulated from this happening again in the future.

Children
No Data