from seeq import spy
import pandas as pd

# Set the compatibility option so that you maximize the chance that SPy will remain compatible with your notebook/script
spy.options.compatibility = 192

# Log into Seeq Server if you're not using Seeq Data Lab:
spy.login(url='http://localhost:34216', credentials_file='../credentials.key', force=False)

Asset Trees 2: Templates

In the first asset trees tutorial notebook, we learned how to use the spy.assets.Tree Python class to define, modify, and push asset trees to Seeq. In this notebook, we will dive deeper into the spy.assets submodule and explore its ability to create asset trees out of customized Python “templates”.

You may have heard of the concept of a Digital Twin, which is a “virtual representation that serves as the real-time digital counterpart of a physical object or process.” The spy.assets submodule provides a framework for defining Digital Twins using the power and relative ease-of-use of the Python programming language. You can leverage object-oriented design principles like encapsulation, inheritance, composition, and mixins to increase reuse and consistency while still accommodating the many exceptions that naturally occur in manufacturing scenarios.

As with other capabilities in SPy, the work you do with spy.assets is (by default) sandboxed to a particular workbook that you own. As a result, you can easily experiment with new asset structures and calculations before publishing to your broader organization.

Simple Trees vs Templates

When should you use the spy.assets.Tree() and when should you use the Template methods described here?

Start with the simpler spy.assets.Tree() functionality unless, as you skim through this documentation, it’s clear that you’d like to try the Template functionality first. You can always “graduate” to this more advanced approach.

Concepts

There are several important concepts to understand as we dive in to the spy.assets submodule:

  • An asset is simply a container (with associated properties) that has children. Those children can be other assets and/or attributes (signals, conditions and scalars).

  • An attribute is a signal, condition or scalar that is a child of an asset and is said to be contained by that asset. This is a similar concept to OSIsoft Asset Framework’s attribute.

  • A property is a named value (with optional units of measure) that captures metadata associated with assets, signals, conditions and scalars. They correspond to the columns in a metadata DataFrame.

  • A component is an asset that is a child of another asset and is said to be contained by the parent asset. For example, a Furnace asset may contain a Blast Air Blower.

  • A template defines the set of attributes and components that comprise a particular Asset. A template corresponds to a class of asset like Furnace, Well, Pump.

  • A reference is a link to a signal, condition or scalar that exists somewhere else. If you are mapping a flat list of tags into an asset hierarchy, the asset hierarchy contains references to those tags.

How Python fits in

A template is specified in Python by defining a class that derives from spy.assets.Asset or spy.assets.Mixin (more about mixins later). These classes can have member functions and data members, just like any other Python class.

Classes can have special member functions that represent attributes. They are decorated with the spy.assets.Asset.Attribute decorator, which tells the SPy framework to use them to define attributes for the asset. Similarly, the spy.assets.Asset.Component decorated indicates to the SPy framework that a component is being defined as a child of the asset, which is usually a different Python class instance that derives from spy.assets.Asset or spy.assets.Mixin.

You’ll see how it works in the examples that follow.

Getting Started

First we need to import the Python modules we will need and log in:

from seeq.spy.assets import Asset, ItemGroup

# Show all data in DataFrame output -- don't truncate it
pd.set_option('display.max_colwidth', None)

Preparing the Ingredients

There are two main steps to creating an asset tree, and they’re similar to cooking in your kitchen: First you find and prepare the “ingredients”, then you use them in a “recipe”.

The “ingredients” are the signals, conditions and/or scalars that already exist in Seeq, either as indexed items from external datasources or as items you’ve imported to the internal Seeq database. You will create a DataFrame that represents the metadata ingredients, and those ingredients will be used by spy.assets.build() to create another DataFrame full of new items to be pushed as an asset tree.

For this example, we’re going to map a flat tag structure into an asset template called HVAC. We will use Seeq’s built-in Example Data.

Let’s search for all the “flat list” example data tags that have the pattern Area <letter>_<sensor name>:

hvac_metadata_df = spy.search({
    'Name': 'Area ?_*'
})

hvac_metadata_df.head()

The hvac_metadata_df metadata DataFrame serves as the “ingredients” for the recipe that we will define a little later.

In order to build the HVAC assets, we must add three important columns to our metadata DataFrame:

  • Build Asset specifies the name of the asset that a row of metadata applies to. In our case, that will be Area X, where X is a letter differentiating the different areas of the plant that an HVAC system serves.

  • Build Path specifies the path through the asset hierarchy where you want the asset to live.

We will create these columns using the power of Pandas DataFrames:

# We can use Pandas' string extraction capabilities to create the Build Asset column
hvac_metadata_df['Build Asset'] = hvac_metadata_df['Name'].str.extract('(Area .)_.*')

# We will specify a simple path in a new tree where we want these to live
hvac_metadata_df['Build Path'] = 'My HVAC Units >> Facility #1'

hvac_metadata_df

Writing the Recipe

The recipe that turns the ingredients into an asset structure is specified using Python classes that derive from Asset or Mixin.

First let’s define our HVAC class:

class HVAC(Asset):

    @Asset.Attribute()
    def Temperature(self, metadata):
        # We use simple Pandas syntax to select for a row in the DataFrame corresponding to our desired tag
        return metadata[metadata['Name'].str.endswith('Temperature')]

    @Asset.Attribute()
    def Relative_Humidity(self, metadata):
        # All Attribute functions must take (self, metadata) as parameters
        return metadata[metadata['Name'].str.contains('Humidity')]

This Python code defines an HVAC asset that has two attributes: Temperature and Relative Humidity. These attributes are represented in Python as functions that can return any of the following:

  1. A single-row DataFrame containing an ID column that identifies an item (signal/condition/scalar/metric) to expose on this asset. (As seen above.)

  2. A dictionary that defines the item. A Type entry is required (whose value must be Signal, Condition, Scalar or Metric). If Name is not supplied, the function name will be used as the Name with any underscores automatically replaced with a space. As you’ll see below, you can specify Formula and Formula Parameters if you are trying to specify a calculated item.

  3. A list of dictionaries if you want to specify multiple items in the format of (2) above.

Now we can feed the “ingredients” into the “recipe” using the spy.assets.build() function. This command will build a new set of asset and signal definitions based on the hvac_metadata_df metadata DataFrame. Each unique combination of Build Path and Build Asset will be treated as a different asset, and the metadata argument that is passed in to the Asset.Attribute() decorated functions will only contain the rows for a particular Build Path / Build Asset pair.

build_df = spy.assets.build(HVAC, hvac_metadata_df)

build_df

# NOTE:
#
# There will be errors in this example, the status table will be colored red. Read more below.

In the progress table above (which should be colored red), if you look at the Build Result column for Area F, you can see that no matching metadata was found. If you then do a search in Seeq Workbench for Area F_, you’ll see that there are no Temperature or Relative Humidity tags for that area. That’s fine! When we push, we simply won’t add signals for Area F.

The new build_df DataFrame contains new metadata that represent all the Seeq items that can be pushed into Seeq to realize this simple asset model.

spy.push(metadata=build_df, workbook='SPy Documentation Examples >> spy.assets')

The asset tree is now available for viewing in Seeq. It is scoped to a workbook that you own called SPy Documentation Examples >> spy.assets, so you won’t see it in any other workbook. If/when you want to publish the tree globally, add the workbook=None parameter to your spy.push() function call.

Calculated Signals, Conditions and Scalars

Now let’s do something a little more advanced. We’ll define a new class that has calculations alongside references to raw signals. This class will derive from our existing HVAC class so that it inherits the references we already defined.

class HVAC_With_Calcs(HVAC):

    @Asset.Attribute()
    def Temperature_Rate_Of_Change(self, metadata):
        return {
            'Type': 'Signal',

            # This formula will give us a nice derivative in F/h
            'Formula': '$temp.lowPassFilter(150min, 3min, 333).derivative() * 3600 s/h',

            'Formula Parameters': {
                # We can reference the base class' Temperature attribute here as a dependency
                '$temp': self.Temperature(),
            }
        }

    @Asset.Attribute()
    def Too_Hot(self, metadata):
        return {
            'Type': 'Condition',
            'Formula': '$temp.valueSearch(isGreaterThan($threshold))',
            'Formula Parameters': {
                '$temp': self.Temperature(),

                # We can also reference other attributes in this derived class
                '$threshold': self.Hot_Threshold()
            }
        }

    @Asset.Attribute()
    def Hot_Threshold(self, metadata):
        return {
            'Type': 'Scalar',
            'Formula': '80F'
        }

We’ll make some adjustments to our metadata DataFrame to use the new template and then push into Seeq:

# Make a copy of our original metadata but change the Build Template
hvac_with_calcs_metadata_df = hvac_metadata_df.copy()

build_with_calcs_df = spy.assets.build(HVAC_With_Calcs, hvac_with_calcs_metadata_df)

pd.set_option('display.max_colwidth', 50)

spy.push(metadata=build_with_calcs_df, workbook='SPy Documentation Examples >> spy.assets')

Now when you look at the asset tree in Seeq, you’ll see Temperature Rate Of Change, Too Hot and Hot Threshold. Each time we push, we are overwriting the definition we pushed previously.

Components

You can create an entire asset hierarchy by defining Asset classes that are composed of other Asset classes. To illustrate, let’s start by preparing a DataFrame with our ingredients. This example will be a little contrived given the example data that is available in Seeq.

We will define a metadata DataFrame where there are two Refrigerators and each will have two Compressors. In the cell below, you will see that we define the Build Asset column with the Refigerator name but also define a Compressor column with the Compressor name. We have assigned Area A-D signals to specific Refrigerators and Compressors.

metadata_df = spy.search({
    'Name': r'/Area [A-D]_(?:Temperature|Compressor Power)/',
    'Datasource Class': 'Time Series CSV Files'
})

metadata_df.loc[metadata_df['Name'] == 'Area A_Temperature', 'Build Asset'] = 'Refrigerator 1'
metadata_df.loc[metadata_df['Name'] == 'Area A_Compressor Power', 'Build Asset'] = 'Refrigerator 1'
metadata_df.loc[metadata_df['Name'] == 'Area A_Compressor Power', 'Compressor'] = 'Compressor 1'
metadata_df.loc[metadata_df['Name'] == 'Area B_Compressor Power', 'Build Asset'] = 'Refrigerator 1'
metadata_df.loc[metadata_df['Name'] == 'Area B_Compressor Power', 'Compressor'] = 'Compressor 2'
metadata_df.loc[metadata_df['Name'] == 'Area C_Temperature', 'Build Asset'] = 'Refrigerator 2'
metadata_df.loc[metadata_df['Name'] == 'Area C_Compressor Power', 'Build Asset'] = 'Refrigerator 2'
metadata_df.loc[metadata_df['Name'] == 'Area C_Compressor Power', 'Compressor'] = 'Compressor 3'
metadata_df.loc[metadata_df['Name'] == 'Area D_Compressor Power', 'Build Asset'] = 'Refrigerator 2'
metadata_df.loc[metadata_df['Name'] == 'Area D_Compressor Power', 'Compressor'] = 'Compressor 4'

metadata_df['Build Path'] = 'Refrigerator Units'

metadata_df[['ID', 'Name', 'Build Path', 'Build Asset', 'Compressor']]

Now we can define our Asset classes. Will be using the Asset.Component() decorator and the self.build_components() function:

class Refrigerator(Asset):
    @Asset.Attribute()
    def Temperature(self, metadata):
        # This signal attribute is assigned to the Refrigerator asset
        return metadata[metadata['Name'].str.endswith('Temperature')]

    # Note the use of Asset.Component here, which allows us to return a list of definitions
    # instead of just a single definition.
    @Asset.Component()
    def Compressors(self, metadata):
        # Using the Compressor template class, we build all of the compressor definitions
        # associated with a particular Refrigerator. The column_name supplied tells the
        # build_components function which metadata column to use for the Compressor names.
        return self.build_components(template=Compressor, metadata=metadata, column_name='Compressor')

    @Asset.Attribute()
    def Compressor_Power_Max(self, metadata):
        # We can refer to the Compressors and "pick" attributes for which to perform a
        # roll up. In this example, we're picking the 'Power' signals that are on each
        # compressor and creating a new signal representing the maximum power across
        # all the compressors.
        return self.Compressors().pick({
            'Name': 'Power'
        }).roll_up('maximum')

    @Asset.Attribute()
    def Compressor_High_Power(self, metadata):
        # Similar to Compressor_Power_Max, we are rolling up a compressor calculation but
        # this time it's a condition. 'High Power' at the Refrigerator level will have
        # capsules if either compressor's 'High Power' condition is present.
        #
        # This time we'll use a different method of picking the child items than we used
        # in Compressor_Power_Max() above. In this case, we're going to select the set of
        # compressors that are owned by this asset from the entire set of assets, and use
        # Python conditional logic to find the "High Power" conditions. What you see
        # below is called a Python "list comprehension" that combines iteration over all
        # assets (the "for/in" construct) with filtering (the "if" statement).
        #
        # Helpful functions:
        #  asset.is_child_of(self)      - Is the asset one of my direct children?
        #  asset.is_parent_of(self)     - Is the asset my direct parent?
        #  asset.is_descendant_of(self) - Is the asset below me in the tree?
        #  asset.is_ancestor_of(self)   - Is the asset above me? (i.e. parent/grandparent/great-grandparent/etc)
        #
        return ItemGroup([
            asset.High_Power() for asset in self.all_assets()
            if asset.is_child_of(self)
        ]).roll_up('union')

class Compressor(Asset):
    @Asset.Attribute()
    def Power(self, metadata):
        # Each compressor has just a single attribute, Power
        return metadata[metadata['Name'].str.endswith('Power')]

    @Asset.Attribute()
    def High_Power(self, metadata):
        return {
            'Type': 'Condition',
            'Formula': '$a.valueSearch(isGreaterThan(20kW))',
            'Formula Parameters': {
                '$a': self.Power()
            }
        }

    @Asset.Attribute()
    def Other_Compressors_Are_High_Power(self, metadata):
        # This is a more complex example of using self.all_assets() where we want to
        # look at sibling assets as opposed to parents/children, and do a roll up.
        # Here the "if" statement selects Compressor assets where our parent and
        # their parent are the same but we exclude ourselves.
        return ItemGroup([
            asset.High_Power() for asset in self.all_assets()
            if isinstance(asset, Compressor) and self.parent == asset.parent and self != asset
        ]).roll_up('union')



build_df = spy.assets.build(Refrigerator, metadata_df, errors='raise')

spy.push(metadata=build_df, workbook='SPy Documentation Examples >> spy.assets')

There should now be a Refrigerator Units asset tree in Seeq with two Refrigerators with two Compressors each.

In this manner, you can make an arbitrarily deep hierarchy composed of various asset types.

Metrics

Metrics (aka Scorecard Metrics / Threshold Metrics) are powerful items in Seeq that allow you to easily specify aggregations, statistics and associated boundaries. Click on Scorecard Metric in the Seeq Workbench Tools pane to experiment with them.

Metrics can be specified as attributes in asset classes and can refer to other attributes. Here are some examples:

class HVAC_With_Metrics(HVAC):
    @Asset.Attribute()
    def Too_Humid(self, metadata):
        return {
            'Type': 'Condition',
            'Name': 'Too Humid',
            'Formula': '$relhumid.valueSearch(isGreaterThan(70%))',
            'Formula Parameters': {
                '$relhumid': self.Relative_Humidity(),
            }
        }

    @Asset.Attribute()
    def Humidity_Upper_Bound(self, metadata):
        return {
            'Type': 'Signal',
            'Name': 'Humidity Upper Bound',
            'Formula': '$relhumid + 10',
            'Formula Parameters': {
                '$relhumid': self.Relative_Humidity(),
            }
        }

    @Asset.Attribute()
    def Humidity_Lower_Bound(self, metadata):
        return {
            'Type': 'Signal',
            'Name': 'Humidity Lower Bound',
            'Formula': '$relhumid - 10',
            'Formula Parameters': {
                '$relhumid': self.Relative_Humidity(),
            }
        }

    @Asset.Attribute()
    def Humidity_Statistic_KPI(self, metadata):
        return {
            'Type': 'Metric',
            'Measured Item': self.Relative_Humidity(),
            'Statistic': 'Range'
        }

    @Asset.Attribute()
    def Humidity_Simple_KPI(self, metadata):
        return {
            'Type': 'Metric',
            'Measured Item': self.Relative_Humidity(),
            'Thresholds': {
                'HiHi': self.Humidity_Upper_Bound(),
                'LoLo': self.Humidity_Lower_Bound()
            }
        }

    @Asset.Attribute()
    def Humidity_Condition_KPI(self, metadata):
        return {
            'Type': 'Metric',
            'Measured Item': self.Relative_Humidity(),
            'Statistic': 'Maximum',
            'Bounding Condition': self.Too_Humid(),
            'Bounding Condition Maximum Duration': '30h'
        }

    @Asset.Attribute()
    def Humidity_Continuous_KPI(self, metadata):
        return {
            'Type': 'Metric',
            'Measured Item': self.Relative_Humidity(),
            'Statistic': 'Minimum',
            'Duration': '6h',
            'Period': '4h',
            'Metric Neutral Color':'#189E4D',
            #hex color codes can be optionally appended to thresholds
            'Thresholds': {
                'HiHiHi#FF0000': 60,
                'HiHi': 40,
                'LoLo#0000ff': 20
            }
        }

Now let’s push a small asset tree with these metrics in it:

metadata_df = spy.search({
    'Name': 'Area A_*',
    'Datasource Class': 'Time Series CSV Files'
})

metadata_df['Build Asset'] = 'Metrics Area A'
metadata_df['Build Path'] = 'Metrics Example'
build_df = spy.assets.build(HVAC_With_Metrics, metadata_df)
push_df = spy.push(metadata=build_df, workbook='SPy Documentation Examples >> spy.assets')

You should now see a Metrics Example tree that you can drill into and bring up metrics.

Troubleshooting / Debugging

Since we are using Python classes to describe our asset tree and the attributes therein, troubleshooting our code is a little bit harder than just working with DataFrames. If the code within your @Asset.Attribute-decorated function isn’t working the way you expect, ideally you would be able to see what’s happening inside of it while the spy.assets.build() function is running.

A useful tool is Python’s built-in command-line debugger called pdb. Let’s try using it in the example below. When you execute the following cell, you’ll notice that an ipdb> prompt appears, showing you the line of code you’re about to execute. You can show the value of the metadata variable just by typing metadata and pressing ENTER. Then type c and hit ENTER to allow execution to continue.

# You must import the IPython breakpoint function, called "set_trace"
from IPython.core.debugger import set_trace

class DebuggingExample(Asset):
    @Asset.Attribute()
    def My_Scalar(self, metadata):
        # Put in a "breakpoint" so that execution stops here and a command line pops up. Notice the
        # if statement here as an example of how to be more precise about when the breakpoint is hit:
        # We're only going to enter the debugger if the asset's name is 'Debugging Area A'
        if self.definition['Name'] == 'Debugging Area A':
            set_trace()

        return {
            'Type': 'Scalar',
            'Formula': '%s' % len(metadata)
        }

debugging_metadata_df = pd.DataFrame([{
    'Build Asset': 'Debugging Area A',
    'Build Path': 'Debugging Example'
}])

build_df = spy.assets.build(DebuggingExample, debugging_metadata_df)

There are lots of great commands to help you navigate through your code as it executes. Read through all the commands at pdb to familiarize yourself. The most important ones are step, next and continue. Anything else you type at the prompt gets evaluated as Python code, so you can do things like metadata['Build Asset'] to show just the Build Asset column of the metadata DataFrame.

Using an Integrated Development Environment (IDE)

If you want to “level up” and start using more powerful development and debugging tools, you can! There are several good choices available to you, including SPyder (free and open source) and PyCharm (Community Edition is free).

If you take this route, you’ll want to move your code into “normal” .py files and execute SPy commands from a main script with the debugger engaged.

Detailed Help

All SPy functions have detailed documentation to help you use them. Just execute help(spy.<func>) like you see below.

.. code:: ipython3

help(spy.assets.build)