from seeq import spy
import pandas as pd

# Set the compatibility option so that you maximize the chance that SPy will remain compatible with your notebook/script
spy.options.compatibility = 193

# Log into Seeq Server if you're not using Seeq Data Lab:
spy.login(url='http://localhost:34216', credentials_file='../credentials.key', force=False, quiet=True)

Asset Trees 1: Introduction

Asset trees are a foundational tool that can be used to wrangle the full analytic capabilities of Seeq’s software. They sort physical locations, pieces of equipment, and data on that equipment into a hierarchical structure. Organizing your data into an asset tree allows you to: - Utilize asset swapping to rapidly create identical visualizations for different pieces of equipment - Write high-value calculations for your components and scale them across all components in your tree - Automatically generate scalable content and custom analyses - Use your tree as a starting point for roll-ups, calculations, displays, dashboards, and reports

In this notebook we will show how to define an asset tree using the SPy library, modify it to your liking, and push the resulting tree to Seeq Server.

Defining an Asset Tree

There are multiple ways to define an asset tree in SPy. All use the same function: spy.assets.Tree(). We summarize each way here before taking a further look in the subsections below. 1. To create an empty tree, give a name for your new tree as input: spy.assets.Tree('My Tree') 2. To create a tree with custom structure defined by a CSV (comma-separated values) file, give the filename as input: spy.assets.Tree('my_folder/my_tree_template.csv') 3. To use an existing asset tree from Seeq as a starting point for your tree, give the name of that asset tree as input: spy.assets.Tree('Facility 12_Asset Tree')

1. Creating an Empty Tree

In many cases, the easiest and most efficient way to build an asset tree is to start with the name of your root asset and insert your items later.

my_tree = spy.assets.Tree('My Tree', workbook='My Workbook')

This tree will contain a single asset named “My Tree” to begin with. It is scoped to a new workbook named “My Workbook” that will be created when the tree is pushed to Seeq. Underneath the root asset, we will be able to freely insert more assets, signals, conditions, and scalars as we need.

In the code example above, spy.assets.Tree is a function. The two pieces of text 'My Tree' and 'My Workbook' inside the parentheses are inputs to that function. In return, the function outputs a tree object, which we assign to a variable called my_tree. This variable allows us to refer to the newly defined tree in future code.

2. Creating a Tree using CSV Files

Defining asset trees using CSV template files is a great option for those seeking to create large asset trees using minimal Python code, or for those who already have pre-defined hierarchies for their items that can be exported to CSV format.

The file spy_tree_example.csv in the SPy Documentation/Support Files directory contains the following data:

Name	Level 1	Level 2	Level 3	Friendly Name
Area A_Temperature	My CSV Tree	Cooling Tower 1	Area A	Temperature
Area A_Relative Humidity				Relative Humidity
Area B_Temperature			Area B	Temperature
Area B_Relative Humidity				Relative Humidity
Area D_Temperature		Cooling Tower 2	Area D	Temperature
Area D_Relative Humidity				Relative Humidity
Area E_Temperature			Area E	Temperature
Area E_Relative Humidity				Relative Humidity

When this file is given as input to spy.assets.Tree, SPy will look at each row in the file, find an item in Seeq corresponding to the row, and place it into a newly created asset tree at the specified location. The columns of the file give SPy information about where to find the item and how to put it in your tree: - The Name column tells SPy what item to pull from Seeq Server. - The Level columns tell SPy where in the tree to put the item. - If a level column is empty for a particular row, then the rows above are referred to. For example, the third row above has levels My CSV Tree, Cooling Tower 1, and Area B. - The Friendly Name column tells SPy what what to call the item after putting it in the tree. - An optional ID column can help SPy find items from Seeq Server that don’t have unique names.

For example, the first row tells SPy to take the data from the existing signal “Area A_Temperature” in Seeq, create a new signal called “Temperature” containing the same data, and insert the new signal into the tree underneath the asset “Area A”. The asset “Area A” will subsequently be underneath “Cooling Tower 1”, which is underneath the root asset “My CSV Tree”.

Below we create a tree from this CSV file, and then use the visualize() function to look at what the end result is.

my_csv_tree = spy.assets.Tree('Support Files/spy_tree_example.csv', workbook='My Workbook')
my_csv_tree.visualize()

3. Working with an Existing Asset Tree

Say you have an existing asset tree in Seeq, perhaps from an external datasource like OSIsoft PI AF, and you wish to clean up the tree or add calculations to it for further analysis. You can define a tree in SPy using this existing tree as a starting point by referring to it by name in the spy.assets.Tree input.

For instance, let’s pull in the asset tree that organizes Seeq’s example data. Its root asset is named 'Example'.

example_data_tree = spy.assets.Tree('Example',
                                    workbook='My Workbook',
                                    description='My custom copy of Example Data')

This creates a tree that is a full copy of the example tree, with an added description so you can tell your new tree apart from the old tree. Therefore, when we make modifications and push them to Seeq, the original tree will remain unaltered. The only time a copy will not be made is if the tree we choose to work with was also created by SPy.

An existing asset tree or subtree can also be pulled into SPy using the ID of its root asset…

example_data_tree = spy.assets.Tree('656B88EC-E71F-44B6-B2A1-D60202B3B0CD')

…or by using spy.search results.

search_results = spy.search({'Name': 'Example',
                             'Type': 'Asset',
                             'Datasource Name': 'Example Data'})
example_data_tree = spy.assets.Tree(search_results)

Inserting Items into the Tree

The next step to building your asset tree is to add more items to it. Inserting the data that makes up your tree can be broken down into roughly three distinct substeps: 1. Insert assets to give the tree structure 2. Add signals, conditions, and scalars from Seeq server into your new asset hierarchy 3. Insert calculations

Each of these steps is done using the insert() function. The insert() function can be called directly on your tree like this: my_tree.insert(...inputs...). Your tree will be updated accordingly.

1. Inserting Assets

Assets are logical groups of signals, conditions, scalars, or even other assets. In most use cases, they represent physical locations, pieces of equipment, or organizational collections of other assets. The assets of our tree will form its backbone, while other items will usually be at the bottom of our tree, farthest from the root asset.

To insert anything into the tree, we need to specify (1) what we are inserting and (2) where we want to insert it. We will always specify where we are inserting using the parent input – for example, write parent='Area A' as input to insert your data directly underneath the asset in your tree with name “Area A”.

To insert assets into the tree, give a name to the children input.

my_tree.insert(children='Cooling Tower 1',
               parent='My Tree')
my_tree.visualize()

You can also give a list of names to create multiple assets at once.

my_tree.insert(children=['Area A', 'Area B', 'Area C'],
               parent='Cooling Tower 1')
my_tree.visualize()

Now we’re ready to organize our data in the tree hierarchy we have created.

2. Inserting Signals, Conditions, and Scalars from Seeq Server

To insert items from the server, we will need to give our tree information on where to find those items. One practical way to find this information is to query items from Seeq by name using the spy.search function.

search_results = spy.search({'Name': 'Area A_Temperature',
                             'Datasource Name': 'Example Data'})

We can pass these search results directly to the children input of the insert() function. Additionally, we can use the friendly_name input to rename the signal as we insert it.

search_results = spy.search({'Name': 'Area A_Temperature',
                             'Datasource Name': 'Example Data'})

my_tree.insert(children=search_results,
               friendly_name='Temperature',
               parent='Area A')
my_tree.visualize()

The new signal “Temperature” we see underneath “Area A” is a copy of the existing signal “Area A_Temperature” from Seeq.

The example notebook for spy.search contains many more examples of how to query the server for the items you want to put in your tree. For instance, we can use wildcards to search for and insert many signals at once underneath Area B.

search_results = spy.search({'Name': 'Area B_*',
                             'Type': 'Signal',
                             'Datasource Name': 'Example Data'})

my_tree.insert(children=search_results,
               parent='Cooling Tower 1 >> Area B')
my_tree.visualize()

Note that we included part of the path of “Area B” in the parent input – if there were multiple assets in the tree named “Area B”, the function would then know to insert under the Area B that lies underneath Cooling Tower 1.

Using spy.search, we can add hundreds or thousands of items to the tree in an organized manner. In addition to visualize(), you can use the function missing_items() to verify that your tree is balanced. For example, it flags to us that the asset “Area F” in the example data is missing signals that other assets in Cooling Tower 2 have.

example_data_tree.missing_items()

Other attributes and functions helpful for understanding your tree include my_tree.size, my_tree.height, my_tree.name, and my_tree.summarize().

An alternative to using spy.search when inserting a single item is to copy the item’s ID in Workbench and pass that ID to the children input.

my_tree.insert(children='219C06CC-67C5-4512-9242-2271596CEF56',
               parent='Area A')

3. Inserting Calculations

At this point our tree gives organization to pre-existing data. When we push it to Seeq, we can navigate around the tree and find the data grouped together by asset. While this functionality alone can enable powerful analytics in Seeq Server, we can further our analysis by creating calculations based upon the data items in our tree.

A calculation requires a name, a formula, and a collection of formula parameters. The formula is written in Seeq Formula Language, and the formula parameters assign variables in the formula to items in your tree.

my_tree.insert(name='Too Hot',
               formula='$temp > 100',
               formula_parameters={'$temp': 'Temperature'},
               parent='Area A')
my_tree.visualize()

It’s often handy to create the same calculation for many assets across your tree that contain similarly named signals. This can be done by using wildcards in the parent input, just like when using spy.search(). Children will be inserted underneath every asset in the tree that matches the query given to parent.

Let’s try adding a “Dew Point” calculation underneath every area in a tree. Note that we’ll use my_csv_tree for this example instead of my_tree because its signals have more uniform names across the tree.

my_csv_tree.insert(name='Dew Point',
                   formula='$t - ((100 - $rh.setUnits(""))/5)', # From https://iridl.ldeo.columbia.edu/dochelp/QA/Basic/dewpoint.html
                   formula_parameters={'$t': 'Temperature', '$rh': 'Relative Humidity'},
                   parent='Area ?')
my_csv_tree.visualize()

Because there are signals “Temperature” and “Relative Humidity” under each parent asset, the calculations can successfully be inserted, with their formula parameters referring to the respective signals under their parent asset.

The parent parameter will also accept spy.search results, an ID, or regular expressions for more advanced querying, though we won’t show examples here.

Removing and Moving Items

After working with your new tree, you may find that you need to make adjustments. You can use the remove() and move() functions to do this. These functions may also be useful for narrowing the focus of a large asset tree you have copied from Seeq.

The remove() function takes a single input that specifies which items in the tree to remove. Let’s start by removing a single signal from my_tree

my_tree.remove('Area B_Compressor Stage')
my_tree.visualize()

Note that if an asset is removed, then all of its children will be removed as well:

my_tree.remove('Area B')
my_tree.visualize()

The input for remove() also supports paths, wildcards, and spy.search results.

example_data_tree.remove('Area A >> Temperature')
example_data_tree.remove('Cooling Tower 2 >> Area ? >> Optimizer')
example_data_tree.remove(spy.search({'Path': 'Example >> Cooling Tower 1', 'Asset': 'Area C'}))

The move() function pops data out of the tree and inserts it back in at another location. It takes two inputs: source and destination. The source is what is being removed, and the destination is the new parent it should have when re-inserted into the tree.

Let’s say that we accidentally insert “Area C_Temperature” into Area A of My Tree.

my_tree.insert(children=spy.search({'Name': 'Area C_Temperature', 'Datasource Name': 'Example Data'}),
               parent='Area A')
my_tree.visualize()

Then we can easily amend our issue without starting over or rerunning commands by using the move() function:

my_tree.move(source='Area A >> Area C_Temperature',
             destination='Area C')
my_tree.visualize()

Pushing a Tree

The last step to working with an asset tree in SPy is to push your changes to Seeq! Up until now, all of our operations have only modified objects in Python — in order to view your tree in Workbench and share it with others, use the push() function to send it to the server.

my_tree.push()
my_csv_tree.push()

Tada! The link displayed in the output will take you to the workbook that now contains your asset tree.

Detailed Help

All SPy functions have detailed documentation to help you use them. Just execute help(spy.<func>) like you see below.

help(spy.assets.Tree)

Advanced Features

Inserting with Custom DataFrames

For even more flexible insertions, you can provide any Pandas DataFrame as input to the children argument. The features of the Pandas library can be harnessed to customize the path, name, formula, and properties of every item you are inserting into your tree.

new_signals = pd.DataFrame([{
    'Name': 'Relative Humidity',
    'ID': spy.search({'Name': 'Area A_Relative Humidity', 'Datasource Name': 'Example Data'}).ID.squeeze(),
    'Parent': 'Area A'
}, {
    'Name': 'Dew Point',
    'Formula': '$t - ((100 - $RH.setUnits(""))/5)',
    'Formula Parameters': {'$t': 'Temperature', '$rh': 'Relative Humidity'},
    'Parent': 'Area A'
}])

my_tree.insert(new_signals)

Inserting Metrics

Once your tree is defined with the data items you’re interested in, you can add metrics to create tabular calculations.

Metrics usually have more inputs than other types of tree items and therefore must be defined using a DataFrame. The required properties are the Name, Type (‘Metric’), and a Measured Item. Like formula parameters for calculations, the Measured Item, Bounding Condition, and Thresholds can refer to other items in your tree.

new_metrics = pd.DataFrame([{
    'Name': 'Dew Point Hourly Average',
    'Type': 'Metric',
    'Parent': 'Area A',
    'Measured Item': 'Dew Point',
    'Statistic': 'Average',
    'Duration': '1h',
    'Period': '1h',
}, {
    'Name': 'Overheating Severity',
    'Type': 'Metric',
    'Parent': 'Area A',
    'Measured Item': 'Dew Point',
    'Aggregation Function': 'percentile(95)',
    'Bounding Condition': 'Too Hot',
    'Bounding Condition Maximum Duration': '48h',
    'Metric Neutral Color': '#FFFFFF',
    'Thresholds': {
        'HiHiHi#FF0000': 95,
        'HiHi': 'Temperature',
        'Hi': 85
    }
}])

my_tree.insert(new_metrics)

Inserting Roll-up Calculations

Roll-up calculations are a great way to evaluate summary statistics across multiple assets in your tree in order to monitor the health and performance of your assets. To insert a roll-up calculation, use the roll_up_statistic and roll_up_parameters inputs to the insert() function.

my_csv_tree.insert(name='Average Temperature of All Areas',
                   roll_up_statistic='Average',
                   roll_up_parameters='Area ? >> Temperature',
                   parent='Cooling Tower ?')
my_csv_tree

The resulting calculation is created by applying the function specified by roll_up_statistic to all parameters that match the string given by roll_up_parameters. In this case, the roll-up Cooling Tower 1 >> Average Temperature of All Areas calculates the average of Area A >> Temperature and Area B >> Temperature, and similarly for Cooling Tower 2, Area D, and Area E respectively.

Inserting with References

When inserting many items into a tree at once, you may want each item you are inserting to have a different friendly name and a different parent. To achieve this quickly, you can pass references to DataFrame columns to the arguments friendly_name and parent of the insert() function.

In the Seeq example data, there are a collection of signals with names of the format 'Area A_Temperature'. Column references allows you to insert all of these signals at once, where the signal 'Area A_Temperature' has parent 'Area A' with friendly name 'Temperature', while the signal 'Area D_Compressor Power' has parent 'Area D' and friendly name 'Compressor Power', and so on. First let’s remove all existing signals from my_tree.

my_tree.remove('Area ? >> *')

Then let’s grab the signals from Seeq:

search_results = spy.search({'Name': 'Area ?_*',
                             'Datasource Name': 'Example Data'}, order_by='Name')
search_results.head(5)

A column value reference has the following syntax: {{ ... } ... }. The substring within the inner braces specifies which column of the above DataFrame to look for data in. If nothing is provided where the second ellipsis is, then the data from that column will be returned. For example the first row seen above would return kW when queried with {{Value Unit Of Measure}}. However, if we only want some of the data in the column, then we can provide an expression with wildcards that matches the data and has parentheses surrounding the substring we wish to extract. For example, {{Name}Area ?_(*)} will return Compressor Power in the first row.

Let’s see it in action:

my_tree.insert(children=search_results,
               friendly_name='{{Name}Area ?_(*)}',
               parent='{{Name}(Area ?)_*}')
my_tree.visualize()

Below are more examples of column value extraction syntax, with the following input children DataFrame:

ID	Name	Unit	Facility Type
B8FA62DA-5C42-4CA3-9C3F-E80E6E5AE990	Site A_Temp	°F	Research Facility
51D52461-A667-413A-AD9E-548FE16E601B	Site B_Flow	gal/s	Factory

Syntax	Syntax Description	Output for first row	Output for second row
{{Name}}	Get the value of the “Name” column for that row.	Site A_Temp	Site B_Flow
{{Name}Site ?_(*)}	From the “Name” column, get the value of the wildcard match after "Site ?_".	Temp	Flow
{{Name}(*_Temp)}	Get the entire value of the “Name” column for that row if it matches "*_Temp". Note: this pattern does not match the second row. If such a pattern is used for parent, then the row would not be inserted under any parent. If such a pattern is used for friendly_name, its original name will be used.	Site A_Temp	N/A
{{Facility Type}} {{Name}Site (?)_*}	Get the value of the “Facility Type” column for that row. Then a literal " " character. Then from the “Name” column, get the value of the single-char wildcard between "Site " and "_".	Research Facility A	Factory B
Average of {{Name}(Site ?)_*} {{Name}Site ?_(*)} ({{Unit}})	A literal "Average of ". Then from the “Name” column, get the value of the substring matching "Site ?" before the "_". Then a literal " " character. Then from the “Name” column, get the value of the wildcard match after "Site ?_". Then a literal " (". Get the value of the “Unit” column. Then a literal ")".	Average of Site A Temp (°F)	Average of Site B Flow (gal/s)

Pushing Large Trees

If you’ve created a large tree, it can take significant time to execute the push() command. You can speed it up by making use of a metadata state file. You just need to specify a unique filename like so:

my_csv_tree.push(metadata_state_file='Output/asset_tree_tutorial_metadata_state_file.pickle.zip')

If the cell above is executed multiple times, you’ll notice that the Push Result column includes Success: Unchanged, which means SPy didn’t bother sending anything to the Seeq Server since nothing had changed.

The spy.assets Submodule and Asset Tree Templates

Read on to the next tutorial page on asset trees in SPy to learn how to use the spy.assets submodule to build asset trees out of templates.

API Reference Links

• seeq.spy.assets.Tree