from seeq import spy
import pandas as pd
# Set the compatibility option so that you maximize the chance that SPy will remain compatible with your notebook/script
spy.options.compatibility = 192
# Log into Seeq Server if you're not using Seeq Data Lab:
spy.login(url='http://localhost:34216', credentials_file='../credentials.key', force=False)
spy.push
Uploads signals, conditions, scalars, and assets to the Seeq Server.
There are two main types of information processed by Seeq: Data and metadata:
Data is the time series and time interval information that is either collected or derived from sensor data. It consists of timestamps and values (samples), or time intervals and properties (capsules). This data can be plotted on a trend or used to train a neural network, for example.
Metadata is the information about the data, that is independent of a particular point in time or time interval. For example, a signal’s name, description and unit of measure is classified as metadata; or the formula that is used to derive a new signal from one or more source signals; or the asset tree that is used to model similar equipment or industrial processes.
The spy.push() command allows you to upload both types of
information to Seeq Server. When you push metadata, you make entries
in Seeq’s data index, which allows you or other users to search for and
find such entries. If you also push data, then samples or capsules or
scalars will appear when the user selects those index entries for
inclusion on a trend, scatter plot, scorecard or other visualization.
spy.push(data=None, metadata=None, item_type=None, workbook='Data Lab >> Data Lab Analysis',
worksheet='From Data Lab', datasource=None, archive=False, type_mismatches='raise', errors='catalog')
Workbook scoping
When you push any type of data, it is only available/discoverable within
the workbook specified by the workbook argument. This allows you to
sandbox your activity by default, and only publish to your colleagues
later when your experimentation is largely over.
Pushing signal data
The simplest activity you can do with the spy.push() command is to
read in a CSV file using Pandas and push it into Seeq. It will be stored
in Seeq’s internal time series database.
import csv
csv_file = pd.read_csv('Support Files/csv_import_example.csv', parse_dates=['TIME(unitless)'], index_col='TIME(unitless)')
csv_file.head()
When you want to push data, it must have an index with a datetime data
type. That’s why we used the parse_dates and index_col arguments
for Pandas.read_csv().
Now you can just push it into Seeq:
push_results = spy.push(data=csv_file, workbook='SPy Documentation Examples >> spy.push')
push_results
NOTE: Pushing data can be relatively slow. This is an area that Seeq will be optimizing in future versions.
You can push multiple times, and as long as the names are the same and the workbook hasn’t changed, you’ll just add to the existing signal.
Pushing metadata
Now let’s try pushing just metadata. You can see that the column names from the CSV file contain the unit of measure in parentheses. Let’s use Pandas to extract the name and the unit of measure as separate columns.
better_metadata = push_results.copy()
better_metadata['Original Name'] = better_metadata.index
better_metadata['Name'] = better_metadata['Original Name'].str.extract(r'(.*)\(')
better_metadata['Value Unit Of Measure'] = better_metadata['Original Name'].str.extract(r'.*\((.*)\)')
better_metadata
spy.push(metadata=better_metadata, workbook='SPy Documentation Examples >> spy.push')
Pushing condition and capsule data
You can also push capsules, which are time intervals of interest, to
Seeq by supplying a DataFrame with Capsule Start and Capsule End
columns. Any additional columns will be added as properties of the
capsule.
capsule_data = pd.DataFrame([{
'Capsule Start': pd.to_datetime('2019-01-10T09:00:00.000Z'),
'Capsule End': pd.to_datetime('2019-01-10T17:00:00.000Z'),
'Operator On Duty': 'Mark'
}, {
'Capsule Start': pd.to_datetime('2019-01-11T09:00:00.000Z'),
'Capsule End': pd.to_datetime('2019-01-11T17:00:00.000Z'),
'Operator On Duty': 'Hedwig'
}])
capsule_data
When you push capsule data, you must supply a metadata DataFrame
that contains, at minimum, the Name, Type and
Maximum Duration columns like the example below.
If your metadata DataFrame includes multiple conditions, the
data DataFrame must have a Condition column to indicate which
condition will receive that particular capsule. The value of the
Condition column must correspond to an index entry of the
metadata DataFrame.
spy.push(data=capsule_data,
metadata=pd.DataFrame([{
'Name': 'Operator Shifts',
'Type': 'Condition',
'Maximum Duration': '2d'
}]), workbook='SPy Documentation Examples >> spy.push')
Capsule properties that have units of measure can be specified in the
metadata. Say you have properties like Height and Mass:
capsule_data = pd.DataFrame([{
'Capsule Start': pd.to_datetime('2019-01-10T09:00:00.000Z'),
'Capsule End': pd.to_datetime('2019-01-10T17:00:00.000Z'),
'Height': 5,
'Mass': 10,
}, {
'Capsule Start': pd.to_datetime('2019-01-11T09:00:00.000Z'),
'Capsule End': pd.to_datetime('2019-01-11T17:00:00.000Z'),
'Height': 3,
'Mass' : 6
}])
capsule_data
Use Capsule Property Units in metadata dataframe to specify that
Height has units of meters, m and Mass has units of kg :
spy.push(data=capsule_data,
metadata=pd.DataFrame([{
'Name': 'In Production',
'Type': 'Condition',
'Maximum Duration': '2d',
'Capsule Property Units': {'Height': 'm',
'Mass': 'kg'}
}]), workbook='SPy Documentation Examples >> spy.push')
Updating capsules
To update capsules, you must replace them with new capsules. An easy
workflow for this is to pull all the capsules in the range you want to
edit, make changes, and then push with the replace argument set to
the same range.
First, get the condition using spy.search():
condition = spy.search({'Name': 'In Production'}, workbook='SPy Documentation Examples >> spy.push')
condition
The start and end should be identical for both the spy.pull() and
the replace argument to spy.push() so that nothing is
accidentally deleted or duplicated. To make this easier, save them as
variables.
replace_start=pd.to_datetime('2019-01-10T09:00:00.000Z')
replace_end=pd.to_datetime('2019-01-11T17:00:00.000Z')
Get all the capsules within that range:
new_capsule_data = spy.pull(condition,
start=replace_start,
end=replace_end)
new_capsule_data
Now we can update specific capsule properies. Let’s say the height of the first capsule was actually 4 meters and the Mass for the second capsule is 7 kg :
new_capsule_data.at[0, 'Height'] = 4
new_capsule_data.at[1, 'Mass'] = 7
new_capsule_data
Additional capsule propeties can also be added at this time. Suppose you
want to record what the temperature was for the first capsule. A new
Temperature property can be added:
new_capsule_data.at[0, 'Temperature'] = 67
new_capsule_data
You must specify units of measure for existing properties (e.g.,
Height and Mass) with every spy.push() along with units for
new properties (e.g., Temperature measured in F).
The replace parameter takes a dictionary with the keys 'Start'
and 'End'. Any capsules that start in the provided time period will
be replaced.
Push the new_capsule_data using the replace parameter.
spy.push(data=new_capsule_data,
metadata=pd.DataFrame([{
'Name': 'In Production',
'Type': 'Condition',
'Maximum Duration': '2d',
'Capsule Property Units': {'Height':'m',
'Mass':'kg',
'Temperature':'F'}
}]),
replace={'Start': replace_start,
'End': replace_end},
workbook='SPy Documentation Examples >> spy.push')
Deleting capsules
The replace argument can also be used to delete capsules. All
capsules starting within the time range specified by replace are
deleted before the new capsules are added, so providing no new capsules
to spy.push() functions as a simple deletion.
Caution: any capsules whose Start value is within the replace
range will be deleted. The replace Start is inclusive and
replace End is exclusive.
To delete the first capsule, we’ll specify a time range that includes its start (and no other capsule’s start):
spy.push(metadata=pd.DataFrame([{
'Name': 'In Production',
'Type': 'Condition',
'Maximum Duration': '2d'}]),
replace={'Start': pd.to_datetime('2019-01-10T09:00:00.000Z'),
'End': pd.to_datetime('2019-01-10T10:00:00.000Z')},
workbook='SPy Documentation Examples >> spy.push')
If there are several capsules that start at the same time or within the
same range, you can delete just one by following the update workflow but
removing the capsule to be deleted from the pandas DataFrame before
using spy.push()
Detailed Help
All SPy functions have detailed documentation to help you use them. Just
execute help(spy.<func>) like you see below.
help(spy.push)