.. code:: ipython3 from seeq import spy import pandas as pd # Set the compatibility option so that you maximize the chance that SPy will remain compatible with your notebook/script spy.options.compatibility = 196 .. code:: ipython3 # Log into Seeq Server if you're not using Seeq Data Lab: spy.login(url='http://localhost:34216', credentials_file='../credentials.key', force=False) spy.push ======== Uploads signals, conditions, scalars, and assets to the Seeq Server. There are two main types of information processed by Seeq: *Data* and *metadata*: - **Data** is the time series and time interval information that is either collected or derived from sensor data. It consists of timestamps and values (samples), or time intervals and properties (capsules). This data can be plotted on a trend or used to train a neural network, for example. - **Metadata** is the information about the data, that is independent of a particular point in time or time interval. For example, a signal’s *name*, *description* and *unit of measure* is classified as metadata; or the formula that is used to derive a new signal from one or more source signals; or the asset tree that is used to model similar equipment or industrial processes. The ``spy.push()`` command allows you to upload both types of information to Seeq Server. When you push *metadata*, you make entries in Seeq’s data index, which allows you or other users to search for and find such entries. If you also push *data*, then samples or capsules or scalars will appear when the user selects those index entries for inclusion on a trend, scatter plot, scorecard or other visualization. :: spy.push(data=None, metadata=None, item_type=None, workbook='Data Lab >> Data Lab Analysis', worksheet='From Data Lab', datasource=None, archive=False, type_mismatches='raise', errors='catalog') Pushing signal data ------------------- The simplest activity you can do with the ``spy.push()`` command is to read in a CSV file using Pandas and push it into Seeq. It will be stored in Seeq’s internal time series database. .. code:: ipython3 import csv csv_file = pd.read_csv('Support Files/csv_import_example.csv', parse_dates=['TIME(unitless)'], index_col='TIME(unitless)') csv_file.head() When you want to push data, it must have an index with a datetime data type. That’s why we used the ``parse_dates`` and ``index_col`` arguments for ``Pandas.read_csv()``. Now you can just push it into Seeq: .. code:: ipython3 push_results = spy.push(data=csv_file, workbook='SPy Documentation Examples >> spy.push') push_results You can push multiple times, and as long as the names are the same and the workbook hasn’t changed, you’ll just add to the existing signal. Pushing metadata ---------------- Now let’s try pushing just metadata. You can see that the column names from the CSV file contain the unit of measure in parentheses. Let’s use Pandas to extract the name and the unit of measure as separate columns. .. code:: ipython3 better_metadata = push_results.copy() better_metadata['Original Name'] = better_metadata.index better_metadata['Name'] = better_metadata['Original Name'].str.extract(r'(.*)\(') better_metadata['Value Unit Of Measure'] = better_metadata['Original Name'].str.extract(r'.*\((.*)\)') better_metadata .. code:: ipython3 spy.push(metadata=better_metadata, workbook='SPy Documentation Examples >> spy.push') Workbook scoping ---------------- When you push metadata, by default the items are only available/discoverable within the workbook specified by the ``workbook`` argument. This means your activity is isolated from your colleagues by default. Specify ``workbook=None`` to push to globally-accessible items. But make sure this is what you want to do! You wouldn’t want to pollute the global namespace unintentionally. If you push items globally and want to trash them, check out the **Archiving / Trashing Items** section below. Round-tripping -------------- One way to change the metadata on items is by *round-tripping*: First do a ``spy.search(, all_properties=True)`` to retrieve the item metadata in full, then alter the items and push them back. It’s important to specify ``all_properties=True`` and then be sure to keep all the columns in the returned DataFrame, otherwise SPy may fall back to default values which is not what you intend. In particular, the ``ID`` and ``Scoped To`` columns ensure that SPy does not target the wrong item or move an item to a different workbook in a way that you do not intend. Here’s an example: .. code:: ipython3 # Retrieve the item(s) to be changed items_to_change_df = spy.search({'Name': 'BITDEP'}, all_properties=True, workbook='SPy Documentation Examples >> spy.push') # Change the item(s) items_to_change_df.at[0, 'Value Unit Of Measure'] = 'm' # Push them back spy.push(metadata=items_to_change_df, workbook='SPy Documentation Examples >> spy.push') Pushing condition and capsule data ---------------------------------- You can also push capsules, which are time intervals of interest, to Seeq by supplying a DataFrame with ``Capsule Start`` and ``Capsule End`` columns. Any additional columns will be added as properties of the capsule. .. code:: ipython3 capsule_data = pd.DataFrame([{ 'Capsule Start': pd.to_datetime('2019-01-10T09:00:00.000Z'), 'Capsule End': pd.to_datetime('2019-01-10T17:00:00.000Z'), 'Operator On Duty': 'Mark' }, { 'Capsule Start': pd.to_datetime('2019-01-11T09:00:00.000Z'), 'Capsule End': pd.to_datetime('2019-01-11T17:00:00.000Z'), 'Operator On Duty': 'Hedwig' }]) capsule_data When you push capsule data, you must supply a ``metadata`` DataFrame that contains, at minimum, the ``Name``, ``Type`` and ``Maximum Duration`` columns like the example below. If your ``metadata`` DataFrame includes multiple conditions, the ``data`` DataFrame must have a ``Condition`` column to indicate which condition will receive that particular capsule. The value of the ``Condition`` column must correspond to an index entry of the ``metadata`` DataFrame. .. code:: ipython3 spy.push(data=capsule_data, metadata=pd.DataFrame([{ 'Name': 'Operator Shifts', 'Type': 'Condition', 'Maximum Duration': '2d' }]), workbook='SPy Documentation Examples >> spy.push') Capsule properties that have units of measure can be specified in the metadata. Say you have properties like ``Height`` and ``Mass``: .. code:: ipython3 capsule_data = pd.DataFrame([{ 'Capsule Start': pd.to_datetime('2019-01-10T09:00:00.000Z'), 'Capsule End': pd.to_datetime('2019-01-10T17:00:00.000Z'), 'Height': 5, 'Mass': 10, }, { 'Capsule Start': pd.to_datetime('2019-01-11T09:00:00.000Z'), 'Capsule End': pd.to_datetime('2019-01-11T17:00:00.000Z'), 'Height': 3, 'Mass' : 6 }]) capsule_data Use ``Capsule Property Units`` in metadata dataframe to specify that ``Height`` has units of meters, ``m`` and Mass has units of ``kg`` : .. code:: ipython3 spy.push(data=capsule_data, metadata=pd.DataFrame([{ 'Name': 'In Production', 'Type': 'Condition', 'Maximum Duration': '2d', 'Capsule Property Units': {'Height': 'm', 'Mass': 'kg'} }]), workbook='SPy Documentation Examples >> spy.push') Updating capsules ----------------- To update capsules, you must replace them with new capsules. An easy workflow for this is to pull all the capsules in the range you want to edit, make changes, and then push with the ``replace`` argument set to the same range. First, get the condition using ``spy.search()``: .. code:: ipython3 condition = spy.search({'Name': 'In Production'}, workbook='SPy Documentation Examples >> spy.push') condition The start and end should be identical for both the ``spy.pull()`` and the ``replace`` argument to ``spy.push()`` so that nothing is accidentally deleted or duplicated. To make this easier, save them as variables. .. code:: ipython3 replace_start=pd.to_datetime('2019-01-10T09:00:00.000Z') replace_end=pd.to_datetime('2019-01-11T17:00:00.000Z') Get all the capsules within that range: .. code:: ipython3 new_capsule_data = spy.pull(condition, start=replace_start, end=replace_end) new_capsule_data Now we can update specific capsule properies. Let’s say the height of the first capsule was actually 4 meters and the Mass for the second capsule is 7 kg : .. code:: ipython3 new_capsule_data.at[0, 'Height'] = 4 new_capsule_data.at[1, 'Mass'] = 7 new_capsule_data Additional capsule propeties can also be added at this time. Suppose you want to record what the temperature was for the first capsule. A new ``Temperature`` property can be added: .. code:: ipython3 new_capsule_data.at[0, 'Temperature'] = 67 new_capsule_data You must specify units of measure for existing properties (e.g., ``Height`` and ``Mass``) with every ``spy.push()`` along with units for new properties (e.g., ``Temperature`` measured in ``F``). The ``replace`` parameter takes a dictionary with the keys ``'Start'`` and ``'End'``. Any capsules that start in the provided time period will be replaced. Push the ``new_capsule_data`` using the ``replace`` parameter. .. code:: ipython3 spy.push(data=new_capsule_data, metadata=pd.DataFrame([{ 'Name': 'In Production', 'Type': 'Condition', 'Maximum Duration': '2d', 'Capsule Property Units': {'Height':'m', 'Mass':'kg', 'Temperature':'F'} }]), replace={'Start': replace_start, 'End': replace_end}, workbook='SPy Documentation Examples >> spy.push') Deleting capsules ----------------- The ``replace`` argument can also be used to delete capsules. All capsules starting within the time range specified by ``replace`` are deleted before the new capsules are added, so providing no new capsules to ``spy.push()`` functions as a simple deletion. Caution: any capsules whose ``Start`` value is within the ``replace`` range will be deleted. The ``replace`` ``Start`` is inclusive and ``replace`` ``End`` is exclusive. To delete the first capsule, we’ll specify a time range that includes its start (and no other capsule’s start): .. code:: ipython3 spy.push(metadata=pd.DataFrame([{ 'Name': 'In Production', 'Type': 'Condition', 'Maximum Duration': '2d'}]), replace={'Start': pd.to_datetime('2019-01-10T09:00:00.000Z'), 'End': pd.to_datetime('2019-01-10T10:00:00.000Z')}, workbook='SPy Documentation Examples >> spy.push') If there are several capsules that start at the same time or within the same range, you can delete just one by following the update workflow but removing the capsule to be deleted from the ``pandas`` DataFrame before using ``spy.push()`` Pushing calculated items ------------------------ You can push Signals, Conditions and Scalars that are calculated via Seeq’s *Formula* language by supplying a metadata DataFrame with the following columns: - ``Formula``: A string representing the Seeq Formula, written just as you would in the *Formula* tool within Seeq Workbench. - ``Formula Parameters``: A dictionary where the keys represent variable names declared in the formula and the values can be: - A string that is the ``ID`` of the item. - A single-row DataFrame that contains an ``ID`` column referring to the item. - A (nested) dictionary of column/value pairs that reference another row’s cell(s) in the ``metadata`` DataFrame you are pushing. This is a *forward reference*, and SPy will handle the correct sequencing of item creation/updating. (You do not have to worry about the order in which items appear in the DataFrame.) Let’s take a look at how all these methods work in practice: .. code:: ipython3 import textwrap # Search for the items to use as inputs area_a_power_row = spy.search({ 'Datasource Name': 'Example Data', 'Type': 'Signal', 'Name': 'Area A_Compressor Power' }) area_b_power_series = spy.search({ 'Datasource Name': 'Example Data', 'Type': 'Signal', 'Name': 'Area B_Compressor Power' }).squeeze() area_b_power_id = area_b_power_series['ID'] # Define a metadata DataFrame with two calculated items metadata=pd.DataFrame([ { 'Name': 'Area A plus B Compressor Power', 'Type': 'Signal', 'Formula': textwrap.dedent(""" $a + $b """).strip(), # We use textwrap.dedent() and strip() on a multiline string to make it look good in the Formula tool 'Formula Parameters': { '$a': area_a_power_row, '$b': area_b_power_id } }, { 'Name': 'A and B High Inrush', 'Type': 'Condition', 'Formula': textwrap.dedent(""" $AandB > 60 """).strip(), 'Formula Parameters': { '$AandB': { 'Name': 'Area A plus B Compressor Power' # You can also specify 'Path' and 'Asset' in the dictionary } } }]) # Push those calculations to the workbook pushed_calculations_df = spy.push(metadata=metadata, workbook='SPy Documentation Examples >> spy.push') As you can see in the results DataFrame, the ``A and B High Inrush`` Condition’s ``Formula Parameters`` refer (correctly) to the ``ID`` of ``Area A plus B Compressor Power``. Archiving / Trashing Items -------------------------- If you pushed some items and you decide you want to put them in the trash, use ``spy.archive()``. You just need a DataFrame with an ``ID`` column, and any SPy function that returns items will include one. .. code:: ipython3 # Trash the items spy.archive(pushed_calculations_df) # Undo it! spy.archive(pushed_calculations_df, undo=True) Detailed Help ------------- All SPy functions have detailed documentation to help you use them. Just execute ``help(spy.)`` like you see below. .. code:: ipython3 help(spy.push) API Reference Links ------------------- - :py:func:`seeq.spy.push`