from seeq import spy
import pandas as pd

# Set the compatibility option so that you maximize the chance that SPy will remain compatible with your notebook/script
spy.options.compatibility = 196

# Log into Seeq Server if you're not using Seeq Data Lab:
spy.login(url='http://localhost:34216', credentials_file='../credentials.key', force=False)

spy.push

Uploads signals, conditions, scalars, and assets to the Seeq Server.

There are two main types of information processed by Seeq: Data and metadata:

  • Data is the time series and time interval information that is either collected or derived from sensor data. It consists of timestamps and values (samples), or time intervals and properties (capsules). This data can be plotted on a trend or used to train a neural network, for example.

  • Metadata is the information about the data, that is independent of a particular point in time or time interval. For example, a signal’s name, description and unit of measure is classified as metadata; or the formula that is used to derive a new signal from one or more source signals; or the asset tree that is used to model similar equipment or industrial processes.

The spy.push() command allows you to upload both types of information to Seeq Server. When you push metadata, you make entries in Seeq’s data index, which allows you or other users to search for and find such entries. If you also push data, then samples or capsules or scalars will appear when the user selects those index entries for inclusion on a trend, scatter plot, scorecard or other visualization.

spy.push(data=None, metadata=None, item_type=None, workbook='Data Lab >> Data Lab Analysis',
         worksheet='From Data Lab', datasource=None, archive=False, type_mismatches='raise', errors='catalog')

Pushing signal data

The simplest activity you can do with the spy.push() command is to read in a CSV file using Pandas and push it into Seeq. It will be stored in Seeq’s internal time series database.

import csv
csv_file = pd.read_csv('Support Files/csv_import_example.csv', parse_dates=['TIME(unitless)'], index_col='TIME(unitless)')
csv_file.head()

When you want to push data, it must have an index with a datetime data type. That’s why we used the parse_dates and index_col arguments for Pandas.read_csv().

Now you can just push it into Seeq:

push_results = spy.push(data=csv_file, workbook='SPy Documentation Examples >> spy.push')
push_results

You can push multiple times, and as long as the names are the same and the workbook hasn’t changed, you’ll just add to the existing signal.

Pushing metadata

Now let’s try pushing just metadata. You can see that the column names from the CSV file contain the unit of measure in parentheses. Let’s use Pandas to extract the name and the unit of measure as separate columns.

better_metadata = push_results.copy()
better_metadata['Original Name'] = better_metadata.index
better_metadata['Name'] = better_metadata['Original Name'].str.extract(r'(.*)\(')
better_metadata['Value Unit Of Measure'] = better_metadata['Original Name'].str.extract(r'.*\((.*)\)')
better_metadata

spy.push(metadata=better_metadata, workbook='SPy Documentation Examples >> spy.push')

Workbook scoping

When you push metadata, by default the items are only available/discoverable within the workbook specified by the workbook argument. This means your activity is isolated from your colleagues by default.

Specify workbook=None to push to globally-accessible items. But make sure this is what you want to do! You wouldn’t want to pollute the global namespace unintentionally. If you push items globally and want to trash them, check out the Archiving / Trashing Items section below.

Round-tripping

One way to change the metadata on items is by round-tripping: First do a spy.search(<search dictionary>, all_properties=True) to retrieve the item metadata in full, then alter the items and push them back. It’s important to specify all_properties=True and then be sure to keep all the columns in the returned DataFrame, otherwise SPy may fall back to default values which is not what you intend. In particular, the ID and Scoped To columns ensure that SPy does not target the wrong item or move an item to a different workbook in a way that you do not intend.

Here’s an example:

# Retrieve the item(s) to be changed
items_to_change_df = spy.search({'Name': 'BITDEP'}, all_properties=True, workbook='SPy Documentation Examples >> spy.push')

# Change the item(s)
items_to_change_df.at[0, 'Value Unit Of Measure'] = 'm'

# Push them back
spy.push(metadata=items_to_change_df, workbook='SPy Documentation Examples >> spy.push')

Pushing condition and capsule data

You can also push capsules, which are time intervals of interest, to Seeq by supplying a DataFrame with Capsule Start and Capsule End columns. Any additional columns will be added as properties of the capsule.

capsule_data = pd.DataFrame([{
    'Capsule Start':    pd.to_datetime('2019-01-10T09:00:00.000Z'),
    'Capsule End':      pd.to_datetime('2019-01-10T17:00:00.000Z'),
    'Operator On Duty': 'Mark'
}, {
    'Capsule Start':    pd.to_datetime('2019-01-11T09:00:00.000Z'),
    'Capsule End':      pd.to_datetime('2019-01-11T17:00:00.000Z'),
    'Operator On Duty': 'Hedwig'
}])

capsule_data

When you push capsule data, you must supply a metadata DataFrame that contains, at minimum, the Name, Type and Maximum Duration columns like the example below.

If your metadata DataFrame includes multiple conditions, the data DataFrame must have a Condition column to indicate which condition will receive that particular capsule. The value of the Condition column must correspond to an index entry of the metadata DataFrame.

spy.push(data=capsule_data,
         metadata=pd.DataFrame([{
             'Name': 'Operator Shifts',
             'Type': 'Condition',
             'Maximum Duration': '2d'
         }]), workbook='SPy Documentation Examples >> spy.push')

Capsule properties that have units of measure can be specified in the metadata. Say you have properties like Height and Mass:

capsule_data = pd.DataFrame([{
    'Capsule Start':    pd.to_datetime('2019-01-10T09:00:00.000Z'),
    'Capsule End':      pd.to_datetime('2019-01-10T17:00:00.000Z'),
    'Height': 5,
    'Mass': 10,
}, {
    'Capsule Start':    pd.to_datetime('2019-01-11T09:00:00.000Z'),
    'Capsule End':      pd.to_datetime('2019-01-11T17:00:00.000Z'),
    'Height': 3,
    'Mass' : 6
}])

capsule_data

Use Capsule Property Units in metadata dataframe to specify that Height has units of meters, m and Mass has units of kg :

spy.push(data=capsule_data,
         metadata=pd.DataFrame([{
             'Name': 'In Production',
             'Type': 'Condition',
             'Maximum Duration': '2d',
             'Capsule Property Units': {'Height': 'm',
                                        'Mass': 'kg'}
         }]), workbook='SPy Documentation Examples >> spy.push')

Updating capsules

To update capsules, you must replace them with new capsules. An easy workflow for this is to pull all the capsules in the range you want to edit, make changes, and then push with the replace argument set to the same range.

First, get the condition using spy.search():

condition = spy.search({'Name': 'In Production'}, workbook='SPy Documentation Examples >> spy.push')
condition

The start and end should be identical for both the spy.pull() and the replace argument to spy.push() so that nothing is accidentally deleted or duplicated. To make this easier, save them as variables.

replace_start=pd.to_datetime('2019-01-10T09:00:00.000Z')
replace_end=pd.to_datetime('2019-01-11T17:00:00.000Z')

Get all the capsules within that range:

new_capsule_data = spy.pull(condition,
                            start=replace_start,
                            end=replace_end)
new_capsule_data

Now we can update specific capsule properies. Let’s say the height of the first capsule was actually 4 meters and the Mass for the second capsule is 7 kg :

new_capsule_data.at[0, 'Height'] = 4
new_capsule_data.at[1, 'Mass'] = 7
new_capsule_data

Additional capsule propeties can also be added at this time. Suppose you want to record what the temperature was for the first capsule. A new Temperature property can be added:

new_capsule_data.at[0, 'Temperature'] = 67
new_capsule_data

You must specify units of measure for existing properties (e.g., Height and Mass) with every spy.push() along with units for new properties (e.g., Temperature measured in F).

The replace parameter takes a dictionary with the keys 'Start' and 'End'. Any capsules that start in the provided time period will be replaced.

Push the new_capsule_data using the replace parameter.

spy.push(data=new_capsule_data,
         metadata=pd.DataFrame([{
             'Name': 'In Production',
             'Type': 'Condition',
             'Maximum Duration': '2d',
             'Capsule Property Units': {'Height':'m',
                                        'Mass':'kg',
                                        'Temperature':'F'}
         }]),
         replace={'Start':   replace_start,
                  'End':     replace_end},
         workbook='SPy Documentation Examples >> spy.push')

Deleting capsules

The replace argument can also be used to delete capsules. All capsules starting within the time range specified by replace are deleted before the new capsules are added, so providing no new capsules to spy.push() functions as a simple deletion.

Caution: any capsules whose Start value is within the replace range will be deleted. The replace Start is inclusive and replace End is exclusive.

To delete the first capsule, we’ll specify a time range that includes its start (and no other capsule’s start):

spy.push(metadata=pd.DataFrame([{
             'Name': 'In Production',
             'Type': 'Condition',
             'Maximum Duration': '2d'}]),
         replace={'Start':    pd.to_datetime('2019-01-10T09:00:00.000Z'),
                  'End':      pd.to_datetime('2019-01-10T10:00:00.000Z')},
         workbook='SPy Documentation Examples >> spy.push')

If there are several capsules that start at the same time or within the same range, you can delete just one by following the update workflow but removing the capsule to be deleted from the pandas DataFrame before using spy.push()

Pushing calculated items

You can push Signals, Conditions and Scalars that are calculated via Seeq’s Formula language by supplying a metadata DataFrame with the following columns:

  • Formula: A string representing the Seeq Formula, written just as you would in the Formula tool within Seeq Workbench.

  • Formula Parameters: A dictionary where the keys represent variable names declared in the formula and the values can be:

    • A string that is the ID of the item.

    • A single-row DataFrame that contains an ID column referring to the item.

    • A (nested) dictionary of column/value pairs that reference another row’s cell(s) in the metadata DataFrame you are pushing. This is a forward reference, and SPy will handle the correct sequencing of item creation/updating. (You do not have to worry about the order in which items appear in the DataFrame.)

Let’s take a look at how all these methods work in practice:

import textwrap

# Search for the items to use as inputs
area_a_power_row = spy.search({
    'Datasource Name': 'Example Data',
    'Type': 'Signal',
    'Name': 'Area A_Compressor Power'
})

area_b_power_series = spy.search({
    'Datasource Name': 'Example Data',
    'Type': 'Signal',
    'Name': 'Area B_Compressor Power'
}).squeeze()

area_b_power_id = area_b_power_series['ID']

# Define a metadata DataFrame with two calculated items
metadata=pd.DataFrame([
    {
        'Name': 'Area A plus B Compressor Power',
        'Type': 'Signal',
        'Formula': textwrap.dedent("""
            $a + $b
        """).strip(),   # We use textwrap.dedent() and strip() on a multiline string to make it look good in the Formula tool
        'Formula Parameters': {
            '$a': area_a_power_row,
            '$b': area_b_power_id
        }
    }, {
        'Name': 'A and B High Inrush',
        'Type': 'Condition',
        'Formula': textwrap.dedent("""
            $AandB > 60
        """).strip(),
        'Formula Parameters': {
            '$AandB': {
                'Name': 'Area A plus B Compressor Power'   # You can also specify 'Path' and 'Asset' in the dictionary
            }
        }
    }])

# Push those calculations to the workbook
pushed_calculations_df = spy.push(metadata=metadata, workbook='SPy Documentation Examples >> spy.push')

As you can see in the results DataFrame, the A and B High Inrush Condition’s Formula Parameters refer (correctly) to the ID of Area A plus B Compressor Power.

Archiving / Trashing Items

If you pushed some items and you decide you want to put them in the trash, use spy.archive(). You just need a DataFrame with an ID column, and any SPy function that returns items will include one.

# Trash the items
spy.archive(pushed_calculations_df)

# Undo it!
spy.archive(pushed_calculations_df, undo=True)

Detailed Help

All SPy functions have detailed documentation to help you use them. Just execute help(spy.<func>) like you see below.

help(spy.push)