from seeq import spy
import pandas as pd
# Set the compatibility option so that you maximize the chance that SPy will remain compatible with your notebook/script
spy.options.compatibility = 196
# Log into Seeq Server if you're not using Seeq Data Lab:
spy.login(url='http://localhost:34216', credentials_file='../credentials.key', force=False)
spy.push
Uploads signals, conditions, scalars, and assets to the Seeq Server.
There are two main types of information processed by Seeq: Data and metadata:
Data is the time series and time interval information that is either collected or derived from sensor data. It consists of timestamps and values (samples), or time intervals and properties (capsules). This data can be plotted on a trend or used to train a neural network, for example.
Metadata is the information about the data, that is independent of a particular point in time or time interval. For example, a signal’s name, description and unit of measure is classified as metadata; or the formula that is used to derive a new signal from one or more source signals; or the asset tree that is used to model similar equipment or industrial processes.
The spy.push()
command allows you to upload both types of
information to Seeq Server. When you push metadata, you make entries
in Seeq’s data index, which allows you or other users to search for and
find such entries. If you also push data, then samples or capsules or
scalars will appear when the user selects those index entries for
inclusion on a trend, scatter plot, scorecard or other visualization.
spy.push(data=None, metadata=None, item_type=None, workbook='Data Lab >> Data Lab Analysis',
worksheet='From Data Lab', datasource=None, archive=False, type_mismatches='raise', errors='catalog')
Pushing signal data
The simplest activity you can do with the spy.push()
command is to
read in a CSV file using Pandas and push it into Seeq. It will be stored
in Seeq’s internal time series database.
import csv
csv_file = pd.read_csv('Support Files/csv_import_example.csv', parse_dates=['TIME(unitless)'], index_col='TIME(unitless)')
csv_file.head()
When you want to push data, it must have an index with a datetime data
type. That’s why we used the parse_dates
and index_col
arguments
for Pandas.read_csv()
.
Now you can just push it into Seeq:
push_results = spy.push(data=csv_file, workbook='SPy Documentation Examples >> spy.push')
push_results
You can push multiple times, and as long as the names are the same and the workbook hasn’t changed, you’ll just add to the existing signal.
Pushing metadata
Now let’s try pushing just metadata. You can see that the column names from the CSV file contain the unit of measure in parentheses. Let’s use Pandas to extract the name and the unit of measure as separate columns.
better_metadata = push_results.copy()
better_metadata['Original Name'] = better_metadata.index
better_metadata['Name'] = better_metadata['Original Name'].str.extract(r'(.*)\(')
better_metadata['Value Unit Of Measure'] = better_metadata['Original Name'].str.extract(r'.*\((.*)\)')
better_metadata
spy.push(metadata=better_metadata, workbook='SPy Documentation Examples >> spy.push')
Workbook scoping
When you push metadata, by default the items are only
available/discoverable within the workbook specified by the workbook
argument. This means your activity is isolated from your colleagues by
default.
Specify workbook=None
to push to globally-accessible items. But make
sure this is what you want to do! You wouldn’t want to pollute the
global namespace unintentionally. If you push items globally and want to
trash them, check out the Archiving / Trashing Items section below.
Round-tripping
One way to change the metadata on items is by round-tripping: First do
a spy.search(<search dictionary>, all_properties=True)
to retrieve
the item metadata in full, then alter the items and push them back. It’s
important to specify all_properties=True
and then be sure to keep
all the columns in the returned DataFrame, otherwise SPy may fall back
to default values which is not what you intend. In particular, the
ID
and Scoped To
columns ensure that SPy does not target the
wrong item or move an item to a different workbook in a way that you do
not intend.
Here’s an example:
# Retrieve the item(s) to be changed
items_to_change_df = spy.search({'Name': 'BITDEP'}, all_properties=True, workbook='SPy Documentation Examples >> spy.push')
# Change the item(s)
items_to_change_df.at[0, 'Value Unit Of Measure'] = 'm'
# Push them back
spy.push(metadata=items_to_change_df, workbook='SPy Documentation Examples >> spy.push')
Pushing condition and capsule data
You can also push capsules, which are time intervals of interest, to
Seeq by supplying a DataFrame with Capsule Start
and Capsule End
columns. Any additional columns will be added as properties of the
capsule.
capsule_data = pd.DataFrame([{
'Capsule Start': pd.to_datetime('2019-01-10T09:00:00.000Z'),
'Capsule End': pd.to_datetime('2019-01-10T17:00:00.000Z'),
'Operator On Duty': 'Mark'
}, {
'Capsule Start': pd.to_datetime('2019-01-11T09:00:00.000Z'),
'Capsule End': pd.to_datetime('2019-01-11T17:00:00.000Z'),
'Operator On Duty': 'Hedwig'
}])
capsule_data
When you push capsule data, you must supply a metadata
DataFrame
that contains, at minimum, the Name
, Type
and
Maximum Duration
columns like the example below.
If your metadata
DataFrame includes multiple conditions, the
data
DataFrame must have a Condition
column to indicate which
condition will receive that particular capsule. The value of the
Condition
column must correspond to an index entry of the
metadata
DataFrame.
spy.push(data=capsule_data,
metadata=pd.DataFrame([{
'Name': 'Operator Shifts',
'Type': 'Condition',
'Maximum Duration': '2d'
}]), workbook='SPy Documentation Examples >> spy.push')
Capsule properties that have units of measure can be specified in the
metadata. Say you have properties like Height
and Mass
:
capsule_data = pd.DataFrame([{
'Capsule Start': pd.to_datetime('2019-01-10T09:00:00.000Z'),
'Capsule End': pd.to_datetime('2019-01-10T17:00:00.000Z'),
'Height': 5,
'Mass': 10,
}, {
'Capsule Start': pd.to_datetime('2019-01-11T09:00:00.000Z'),
'Capsule End': pd.to_datetime('2019-01-11T17:00:00.000Z'),
'Height': 3,
'Mass' : 6
}])
capsule_data
Use Capsule Property Units
in metadata dataframe to specify that
Height
has units of meters, m
and Mass has units of kg
:
spy.push(data=capsule_data,
metadata=pd.DataFrame([{
'Name': 'In Production',
'Type': 'Condition',
'Maximum Duration': '2d',
'Capsule Property Units': {'Height': 'm',
'Mass': 'kg'}
}]), workbook='SPy Documentation Examples >> spy.push')
Updating capsules
To update capsules, you must replace them with new capsules. An easy
workflow for this is to pull all the capsules in the range you want to
edit, make changes, and then push with the replace
argument set to
the same range.
First, get the condition using spy.search()
:
condition = spy.search({'Name': 'In Production'}, workbook='SPy Documentation Examples >> spy.push')
condition
The start and end should be identical for both the spy.pull()
and
the replace
argument to spy.push()
so that nothing is
accidentally deleted or duplicated. To make this easier, save them as
variables.
replace_start=pd.to_datetime('2019-01-10T09:00:00.000Z')
replace_end=pd.to_datetime('2019-01-11T17:00:00.000Z')
Get all the capsules within that range:
new_capsule_data = spy.pull(condition,
start=replace_start,
end=replace_end)
new_capsule_data
Now we can update specific capsule properies. Let’s say the height of the first capsule was actually 4 meters and the Mass for the second capsule is 7 kg :
new_capsule_data.at[0, 'Height'] = 4
new_capsule_data.at[1, 'Mass'] = 7
new_capsule_data
Additional capsule propeties can also be added at this time. Suppose you
want to record what the temperature was for the first capsule. A new
Temperature
property can be added:
new_capsule_data.at[0, 'Temperature'] = 67
new_capsule_data
You must specify units of measure for existing properties (e.g.,
Height
and Mass
) with every spy.push()
along with units for
new properties (e.g., Temperature
measured in F
).
The replace
parameter takes a dictionary with the keys 'Start'
and 'End'
. Any capsules that start in the provided time period will
be replaced.
Push the new_capsule_data
using the replace
parameter.
spy.push(data=new_capsule_data,
metadata=pd.DataFrame([{
'Name': 'In Production',
'Type': 'Condition',
'Maximum Duration': '2d',
'Capsule Property Units': {'Height':'m',
'Mass':'kg',
'Temperature':'F'}
}]),
replace={'Start': replace_start,
'End': replace_end},
workbook='SPy Documentation Examples >> spy.push')
Deleting capsules
The replace
argument can also be used to delete capsules. All
capsules starting within the time range specified by replace
are
deleted before the new capsules are added, so providing no new capsules
to spy.push()
functions as a simple deletion.
Caution: any capsules whose Start
value is within the replace
range will be deleted. The replace
Start
is inclusive and
replace
End
is exclusive.
To delete the first capsule, we’ll specify a time range that includes its start (and no other capsule’s start):
spy.push(metadata=pd.DataFrame([{
'Name': 'In Production',
'Type': 'Condition',
'Maximum Duration': '2d'}]),
replace={'Start': pd.to_datetime('2019-01-10T09:00:00.000Z'),
'End': pd.to_datetime('2019-01-10T10:00:00.000Z')},
workbook='SPy Documentation Examples >> spy.push')
If there are several capsules that start at the same time or within the
same range, you can delete just one by following the update workflow but
removing the capsule to be deleted from the pandas
DataFrame before
using spy.push()
Pushing calculated items
You can push Signals, Conditions and Scalars that are calculated via Seeq’s Formula language by supplying a metadata DataFrame with the following columns:
Formula
: A string representing the Seeq Formula, written just as you would in the Formula tool within Seeq Workbench.Formula Parameters
: A dictionary where the keys represent variable names declared in the formula and the values can be:A string that is the
ID
of the item.A single-row DataFrame that contains an
ID
column referring to the item.A (nested) dictionary of column/value pairs that reference another row’s cell(s) in the
metadata
DataFrame you are pushing. This is a forward reference, and SPy will handle the correct sequencing of item creation/updating. (You do not have to worry about the order in which items appear in the DataFrame.)
Let’s take a look at how all these methods work in practice:
import textwrap
# Search for the items to use as inputs
area_a_power_row = spy.search({
'Datasource Name': 'Example Data',
'Type': 'Signal',
'Name': 'Area A_Compressor Power'
})
area_b_power_series = spy.search({
'Datasource Name': 'Example Data',
'Type': 'Signal',
'Name': 'Area B_Compressor Power'
}).squeeze()
area_b_power_id = area_b_power_series['ID']
# Define a metadata DataFrame with two calculated items
metadata=pd.DataFrame([
{
'Name': 'Area A plus B Compressor Power',
'Type': 'Signal',
'Formula': textwrap.dedent("""
$a + $b
""").strip(), # We use textwrap.dedent() and strip() on a multiline string to make it look good in the Formula tool
'Formula Parameters': {
'$a': area_a_power_row,
'$b': area_b_power_id
}
}, {
'Name': 'A and B High Inrush',
'Type': 'Condition',
'Formula': textwrap.dedent("""
$AandB > 60
""").strip(),
'Formula Parameters': {
'$AandB': {
'Name': 'Area A plus B Compressor Power' # You can also specify 'Path' and 'Asset' in the dictionary
}
}
}])
# Push those calculations to the workbook
pushed_calculations_df = spy.push(metadata=metadata, workbook='SPy Documentation Examples >> spy.push')
As you can see in the results DataFrame, the A and B High Inrush
Condition’s Formula Parameters
refer (correctly) to the ID
of
Area A plus B Compressor Power
.
Archiving / Trashing Items
If you pushed some items and you decide you want to put them in the
trash, use spy.archive()
. You just need a DataFrame with an
ID
column, and any SPy function that returns items will include one.
# Trash the items
spy.archive(pushed_calculations_df)
# Undo it!
spy.archive(pushed_calculations_df, undo=True)
Detailed Help
All SPy functions have detailed documentation to help you use them. Just
execute help(spy.<func>)
like you see below.
help(spy.push)