# Data Stores

Data stores are key-value databases to easily set and get any JSON-serializable data and maintain state across workflow executions.

It's useful for counting values, summing up data between run, or tracking unique data points like email addresses.

Data stores are:

  • Persisted between workflow runs
  • Shareable between workflows
  • Quick to get started, since they don't require any setup or external connections

You can also use native pre-built actions to store, update, and clear data without code.

# Using pre-built Data Store actions

Pipedream includes several pre-built actions to interact with your Data Stores for the most common operations.

# Inserting data

To insert data into a data store, first search for the Data Stores app in a new action within your workflow.

Then you can select the Add or update a single record pre-built action.

Insert a single row into a data store

After selecting this action, you'll be presented with the 3 options to finish setup:

Configure the action

  1. Select or create a Data Store - you can create a brand new empty Data Store or choose another Data Store from your account
  2. Key - this is the unique identifier to look up this data in the future
  3. Value - the data that should be stored under the Key

For example, to store when the workflow was initially triggered, pass the timestamp path to the Value field, and assign the name Triggered as the Key:

{{ steps.trigger.context.ts }}

The Key should always evaluate to a string.

However, you can use dynamic keys as well by passing a path to another step's exports. For instance, retrieving an id entry in the body of an HTTP Webhook trigger:

{{ steps.trigger.event.body.id }}

Workflow trigger example

TIP

Need to store multiple data points in one action? Use the Add or update multiple records action instead.

# Retrieving Data

The Get record action will retrieve the latest value of a data point in one of your Data Stores.

Search for the Data Stores app in a new code step, and then select the Get record action:

Create a get record action

This action has 3 properties:

  1. Select a Data Store - select the Data Store to retrieve data from
  2. Key - where to query the data from
  3. Create new record if key is not found - if the specified key isn't found, you can create a new record

Get record action

# Deleting Data

To delete a single record from your Data Store, use the Delete record action in a step:

Select the delete record step

This action requires 2 properties:

  1. Select a Data Store - select the Data Store that contains the record to be deleted
  2. Key - the key that identifies the individual record

This is an example of deleting the Triggered At key that we've created in the steps above:

Delete a record example

Deleting a record does not delete the entire Data Store. To delete an entire Data Store, use the Pipedream Data Stores Dashboard.

# Managing Data Stores

You can view the contents of your Data Stores at any time in the Pipedream Dashboard (opens new window).

On the left had navigation menu, there's a dedicated Data Stores section.

From here you can open individual Data Stores by clicking on them.

# Editing Data Store values manually

You can also edit Data Store values manually from this view. To edit a particular Data Stores values, click the pencil icon on the far right of the data store.

This will open a text box that will allow you to edit the contents of the value. When you're finished with your edits, save by clicking the checkmark icon.

Editing a data store's value manually

# Deleting Data Stores manually

You can delete a specific Data Store from this dashboard as well. On the far right in the Data Store entry, click the trash can icon to prompt the deletion of the Data Store.

Delete a Data Store

Deleting a Data Store is irreversible. Please take care when using this feature.

WARNING

If the delete icon is greyed out and unclickable, it means that you have dependent workflows using the Data Store.

In order to free up this Data Store to be deleted manually, you'll need to remove those steps from the workflow consuming that Data Store, or switch them to use a different Data Store to allow it to be deleted.

# Using with Data Stores in Code Steps

Refer to the Node.js and Python Data Store documentation to learn how to use Data Stores in code steps in your workflows.

# Data store limitations

Pipedream Data Stores are currently in Beta and are subject to change.

Data Stores are only currently available in Node.js and Python code steps. They are not yet available in other languages like Bash or Go.

# Supported data types

Data stores can hold any JSON-serializable data within the storage limits. This includes data types including:

  • Strings
  • Objects
  • Arrays
  • Dates
  • Integers
  • Floats

But you cannot serialize Functions, Classes, or other more complex objects.

# Retrieving a large number of keys

You can retrieve up to 1,024 keys from a data store in a single query.

If you're using a pre-built action or code to retrieve all records or keys, and your Data Store contains more than 1,024 records, you'll receive a 426 error.

← Overview →