Click or drag to resize
OSIsoft, LLC

List / Bulk Data Methods Overview

This topic provides an overview of how the List Data Methods, also known as Bulk Data Methods, work in the AF SDK.

The result from all of the list data access calls is an IEnumerable; most of the calls return IEnumerable<AFValues>. This return type was chosen, because internally the code can take two different paths that have different return types, and it allows the SDK to provide results as they become available. The AF SDK developers did their best to abstract the partial result handling away from developers consuming the AF SDK. The developer simply iterates the results via a foreach loop, and the loop proceeds as results become available.

First, when a list call is made, the SDK partitions the points in the list by server. Next, it executes the data access call against each PI Data Archive server. If the server version is greater than or equal to 3.4.390 (PI Server 2012), then the SDK is aware that it supports the bulk list data access calls. If the version is less than 3.4.390, then the SDK will internally call the singular data access equivalent in parallel on each PIPoint as an alternative to produce the same results. This topic will only focus on the bulk data access code path.

Note Note

The bulk Summary methods require PI Data Archive version 3.4.390.18 or greater.

Internally the AF SDK constructs a bulk query on the server using the parameters in the PIPagingConfiguration object. The most commonly used parameters are:

The PageType specifies how partial results should be "paged" back to the client. Consumers of the SDK can either page by TagCount or EventCount, but there is a catch for which developers should be aware. The PI Data Archive will never return partial results for a single tag, even if the application pages by EventCount. The server will always finish collecting events for the current tag, even if that threshold has been reached. Assume the consumer of the SDK makes a list data access call for RecordedValues for the last 24 hours, and each tag has 20,000 events for that 24 hour period. If you page by EventCount with a PageSize of 10,000, then the application actually ends up with a page for each tag containing 20,000 events instead of the 10,000 originally desired for each page. Developers need to take this into consideration when determining both PageSize and PageType.

Now that the topic of paging has been discussed, the timeouts need to be reviewed. There are two timeouts on PIPagingConfiguration: OperationTimeoutOverride and KeepAliveTimeout. Developers normally should not be concerned with the KeepAliveTimeout. It represents the amount of time the applications has to request the next page from the server, before it "forgets" about the query. The AF SDK automatically requests the next page, in the background, as soon as the current one arrives.

The OperationTimeoutOverride is more useful to consumers of the AF SDK. It represents the amount of time the server can allocate attempting to collect the current page of events before timing out. For example, if a developer sets their PageType to TagCount and PageSize to 100 and they are using the same tags mentioned earlier, then the server needs to collect 2,000,000 events for each page, which might take more than the default timeout. The operation timeout, also known as the data access timeout, defaults to 60 seconds and can be modified through the PI System Explorer or About PI-SDK Connections dialog. It can be temporarily overridden it for the duration of a list data call instead using the PIPagingConfiguration object's OperationTimeoutOverride property. Timeouts can be avoided by increasing the this property's time span, or by adjusting PageType and PageSize to reduce the size of the pages. The latter is usually a better solution.

Note Note

Even if the BulkPayloadPercentThreshold is used for "proactive paging," remember that the server will still need to collect the results for at least one tag before completing a page. Even if this threshold is lowered, it is still possible to timeout, if the server needs to collect a large number of events for a single tag.

Below is the code example for using RecordedValues. Most bulk / list data access methods follow a similar pattern.

// Holds the results keyed on the associated point
Dictionary<PIPoint, AFValues> resultsMap = new Dictionary<PIPoint, AFValues>();

// Results should be sent back for 100 tags in each page.
PIPagingConfiguration config = new PIPagingConfiguration(PIPageType.TagCount, 100);

    IEnumerable<AFValues> listResults = pointList.RecordedValues(
        timeRange, AFBoundaryType.Inside, null, false, config);

    foreach (AFValues pointResults in listResults)
        // Map the results back to the point
        resultsMap[pointResults.PIPoint] = pointResults;
catch (OperationCanceledException)
    // Errors that occur during bulk calls get trapped here
    // The actual error is stored on the PIPagingConfiguration object
catch (Exception otherEx)
    // Errors that occur in an iterative fall-back method get trapped here

' Holds the results keyed on the associated point
Dim resultsMap As New Dictionary(Of PIPoint, AFValues)

' Results should be sent back for 100 tags in each page.
Dim config As New PIPagingConfiguration(PIPageType.TagCount, 100)

    Dim listResults As IEnumerable(Of AFValues) = _
        pointList.RecordedValues(timeRange, AFBoundaryType.Inside, Nothing, False, config)

    For Each pointResults As AFValues In listResults
        ' Map the results back to the point
        resultsMap(pointResults.PIPoint) = pointResults

Catch canceledEx As OperationCanceledException

    ' Errors that occur during bulk calls get trapped here
    ' The actual error is stored on the PIPagingConfiguration object

Catch otherEx As Exception

    ' Errors that occur in an iterative fall-back method get trapped here
End Try

Developer's will want to have at least two catch blocks to cover both code paths mentioned earlier. When a bulk method fails, an OperationCanceledException is thrown. This is the result of an internal .NET task being canceled. The actual error that caused the cancellation is stored on the configuration's Error property. If the cancellation was the result of multiple errors, then the Error property will be an AggregateException containing all of the exceptions. The other catch block covers the case where an error occurs against a pre 3.4.390 PI Data Archive. Timeout exceptions would be caught in the catch block for the OperationCanceledException.

Next this topic will discuss how partial results are processed. First, developers should understand what a foreach loop is converted to by the C# compiler:


foreach (V v in x)
  // embedded-statement

Becomes this:

    E e = ((C)(x)).GetEnumerator();
        V v;  // Inside the while in C# 5.
        while (e.MoveNext()) 
            v = (V)e.Current;
            // embedded-statement
        // necessary code to dispose e

Essentially, it calls GetEnumerator on the IEnumerable, and then uses a while loop to continuously call MoveNext() and iterate through the IEnumerable. When code consuming the AF SDK calls MoveNext() on the returned enumerator, there are a few things that can happen:

  1. A collection of AFValues is ready for the consuming code to process, so the function returns immediately and your code continues along.
  2. The current page has not been received and converted, so the MoveNext() function blocks until the next page arrives.
  3. The consuming code reached the end of the results, so MoveNext() returns false and the code exits the loop.
  4. An exception occurs, so the task processing the pages internally cancels, and an exception is thrown.

The advantage is that code consuming the AF SDK is kept simple. Developers simply iterate the results in a loop, and the AF SDK makes them available as soon as they are ready via a background task.

Enabling Operational Intelligence