.. code:: ipython3 from seeq import spy import pandas as pd # Set the compatibility option so that you maximize the chance that SPy will remain compatible with your notebook/script spy.options.compatibility = 192 .. code:: ipython3 # Log into Seeq Server if you're not using Seeq Data Lab: spy.login(url='http://localhost:34216', credentials_file='../credentials.key', force=False) spy.push ======== Uploads signals, conditions, scalars, and assets to the Seeq Server. There are two main types of information processed by Seeq: *Data* and *metadata*: - **Data** is the time series and time interval information that is either collected or derived from sensor data. It consists of timestamps and values (samples), or time intervals and properties (capsules). This data can be plotted on a trend or used to train a neural network, for example. - **Metadata** is the information about the data, that is independent of a particular point in time or time interval. For example, a signal’s *name*, *description* and *unit of measure* is classified as metadata; or the formula that is used to derive a new signal from one or more source signals; or the asset tree that is used to model similar equipment or industrial processes. The ``spy.push()`` command allows you to upload both types of information to Seeq Server. When you push *metadata*, you make entries in Seeq’s data index, which allows you or other users to search for and find such entries. If you also push *data*, then samples or capsules or scalars will appear when the user selects those index entries for inclusion on a trend, scatter plot, scorecard or other visualization. :: spy.push(data=None, metadata=None, item_type=None, workbook='Data Lab >> Data Lab Analysis', worksheet='From Data Lab', datasource=None, archive=False, type_mismatches='raise', errors='catalog') Workbook scoping ---------------- When you push any type of data, it is only available/discoverable within the workbook specified by the ``workbook`` argument. This allows you to *sandbox* your activity by default, and only publish to your colleagues later when your experimentation is largely over. Pushing signal data ------------------- The simplest activity you can do with the ``spy.push()`` command is to read in a CSV file using Pandas and push it into Seeq. It will be stored in Seeq’s internal time series database. .. code:: ipython3 import csv csv_file = pd.read_csv('Support Files/csv_import_example.csv', parse_dates=['TIME(unitless)'], index_col='TIME(unitless)') csv_file.head() When you want to push data, it must have an index with a datetime data type. That’s why we used the ``parse_dates`` and ``index_col`` arguments for ``Pandas.read_csv()``. Now you can just push it into Seeq: .. code:: ipython3 push_results = spy.push(data=csv_file, workbook='SPy Documentation Examples >> spy.push') push_results NOTE: Pushing data can be relatively slow. This is an area that Seeq will be optimizing in future versions. You can push multiple times, and as long as the names are the same and the workbook hasn’t changed, you’ll just add to the existing signal. Pushing metadata ---------------- Now let’s try pushing just metadata. You can see that the column names from the CSV file contain the unit of measure in parentheses. Let’s use Pandas to extract the name and the unit of measure as separate columns. .. code:: ipython3 better_metadata = push_results.copy() better_metadata['Original Name'] = better_metadata.index better_metadata['Name'] = better_metadata['Original Name'].str.extract(r'(.*)\(') better_metadata['Value Unit Of Measure'] = better_metadata['Original Name'].str.extract(r'.*\((.*)\)') better_metadata .. code:: ipython3 spy.push(metadata=better_metadata, workbook='SPy Documentation Examples >> spy.push') Pushing condition and capsule data ---------------------------------- You can also push capsules, which are time intervals of interest, to Seeq by supplying a DataFrame with ``Capsule Start`` and ``Capsule End`` columns. Any additional columns will be added as properties of the capsule. .. code:: ipython3 capsule_data = pd.DataFrame([{ 'Capsule Start': pd.to_datetime('2019-01-10T09:00:00.000Z'), 'Capsule End': pd.to_datetime('2019-01-10T17:00:00.000Z'), 'Operator On Duty': 'Mark' }, { 'Capsule Start': pd.to_datetime('2019-01-11T09:00:00.000Z'), 'Capsule End': pd.to_datetime('2019-01-11T17:00:00.000Z'), 'Operator On Duty': 'Hedwig' }]) capsule_data When you push capsule data, you must supply a ``metadata`` DataFrame that contains, at minimum, the ``Name``, ``Type`` and ``Maximum Duration`` columns like the example below. If your ``metadata`` DataFrame includes multiple conditions, the ``data`` DataFrame must have a ``Condition`` column to indicate which condition will receive that particular capsule. The value of the ``Condition`` column must correspond to an index entry of the ``metadata`` DataFrame. .. code:: ipython3 spy.push(data=capsule_data, metadata=pd.DataFrame([{ 'Name': 'Operator Shifts', 'Type': 'Condition', 'Maximum Duration': '2d' }]), workbook='SPy Documentation Examples >> spy.push') Capsule properties that have units of measure can be specified in the metadata. Say you have properties like ``Height`` and ``Mass``: .. code:: ipython3 capsule_data = pd.DataFrame([{ 'Capsule Start': pd.to_datetime('2019-01-10T09:00:00.000Z'), 'Capsule End': pd.to_datetime('2019-01-10T17:00:00.000Z'), 'Height': 5, 'Mass': 10, }, { 'Capsule Start': pd.to_datetime('2019-01-11T09:00:00.000Z'), 'Capsule End': pd.to_datetime('2019-01-11T17:00:00.000Z'), 'Height': 3, 'Mass' : 6 }]) capsule_data Use ``Capsule Property Units`` in metadata dataframe to specify that ``Height`` has units of meters, ``m`` and Mass has units of ``kg`` : .. code:: ipython3 spy.push(data=capsule_data, metadata=pd.DataFrame([{ 'Name': 'In Production', 'Type': 'Condition', 'Maximum Duration': '2d', 'Capsule Property Units': {'Height': 'm', 'Mass': 'kg'} }]), workbook='SPy Documentation Examples >> spy.push') Updating capsules ----------------- To update capsules, you must replace them with new capsules. An easy workflow for this is to pull all the capsules in the range you want to edit, make changes, and then push with the ``replace`` argument set to the same range. First, get the condition using ``spy.search()``: .. code:: ipython3 condition = spy.search({'Name': 'In Production'}, workbook='SPy Documentation Examples >> spy.push') condition The start and end should be identical for both the ``spy.pull()`` and the ``replace`` argument to ``spy.push()`` so that nothing is accidentally deleted or duplicated. To make this easier, save them as variables. .. code:: ipython3 replace_start=pd.to_datetime('2019-01-10T09:00:00.000Z') replace_end=pd.to_datetime('2019-01-11T17:00:00.000Z') Get all the capsules within that range: .. code:: ipython3 new_capsule_data = spy.pull(condition, start=replace_start, end=replace_end) new_capsule_data Now we can update specific capsule properies. Let’s say the height of the first capsule was actually 4 meters and the Mass for the second capsule is 7 kg : .. code:: ipython3 new_capsule_data.at[0, 'Height'] = 4 new_capsule_data.at[1, 'Mass'] = 7 new_capsule_data Additional capsule propeties can also be added at this time. Suppose you want to record what the temperature was for the first capsule. A new ``Temperature`` property can be added: .. code:: ipython3 new_capsule_data.at[0, 'Temperature'] = 67 new_capsule_data You must specify units of measure for existing properties (e.g., ``Height`` and ``Mass``) with every ``spy.push()`` along with units for new properties (e.g., ``Temperature`` measured in ``F``). The ``replace`` parameter takes a dictionary with the keys ``'Start'`` and ``'End'``. Any capsules that start in the provided time period will be replaced. Push the ``new_capsule_data`` using the ``replace`` parameter. .. code:: ipython3 spy.push(data=new_capsule_data, metadata=pd.DataFrame([{ 'Name': 'In Production', 'Type': 'Condition', 'Maximum Duration': '2d', 'Capsule Property Units': {'Height':'m', 'Mass':'kg', 'Temperature':'F'} }]), replace={'Start': replace_start, 'End': replace_end}, workbook='SPy Documentation Examples >> spy.push') Deleting capsules ----------------- The ``replace`` argument can also be used to delete capsules. All capsules starting within the time range specified by ``replace`` are deleted before the new capsules are added, so providing no new capsules to ``spy.push()`` functions as a simple deletion. Caution: any capsules whose ``Start`` value is within the ``replace`` range will be deleted. The ``replace`` ``Start`` is inclusive and ``replace`` ``End`` is exclusive. To delete the first capsule, we’ll specify a time range that includes its start (and no other capsule’s start): .. code:: ipython3 spy.push(metadata=pd.DataFrame([{ 'Name': 'In Production', 'Type': 'Condition', 'Maximum Duration': '2d'}]), replace={'Start': pd.to_datetime('2019-01-10T09:00:00.000Z'), 'End': pd.to_datetime('2019-01-10T10:00:00.000Z')}, workbook='SPy Documentation Examples >> spy.push') If there are several capsules that start at the same time or within the same range, you can delete just one by following the update workflow but removing the capsule to be deleted from the ``pandas`` DataFrame before using ``spy.push()`` Detailed Help ------------- All SPy functions have detailed documentation to help you use them. Just execute ``help(spy.)`` like you see below. .. code:: ipython3 help(spy.push) API Reference Links ------------------- - :py:func:`seeq.spy.push`