Data lineage

info

Data lineage in Platform requires Nextflow 25.04 or later, AWS S3 object storage, and Amazon Simple Notification Service (SNS). For best results, use Nextflow 26.04 or later.

Data lineage tracks the full provenance of every pipeline run at both the task and workflow level, including what executed, what data it consumed, and what outputs it produced. Use it to audit results, verify reproducibility, and trace file provenance.

Why use data lineage

Production pipelines generate results that teams need to trust, audit, and reproduce. Data lineage provides a precise, immutable record of how each result was produced.

• Reproducibility: Every run, task, and output file receives a unique lineage ID (LID), a traversable URI that points to a structured record of what ran. Verify that two runs produced identical results, or identify where they diverged.
• Auditing and compliance: For teams in regulated industries such as pharma, clinical genomics, and contract research organizations (CROs), lineage provides the audit trail needed for regulatory compliance. Each record captures inputs, outputs, parameters, compute environment, and the user who launched the run.
• Debugging: When a cached task re-executes, or a pipeline produces an unexpected result, lineage traces backward from any output to all contributing tasks and parameters. Compare two task runs to isolate what changed.
• Broader team access: Exploring Nextflow lineage previously required CLI access and the ability to read raw JSON. Platform now surfaces lineage data on pipeline run detail pages and in Data Explorer.
• Pipeline output visibility: When lineage is enabled and a pipeline uses the Nextflow workflow output syntax (Nextflow 24.10.0 or later), all published output files appear in the Pipeline outputs sub-tab on the run details page. Each file entry includes its lineage ID, lineage labels, and a direct link to Data Explorer, so any team member can locate and open a result without navigating cloud storage.
• Discovery across runs: Workflow output labels make output files discoverable across runs. Navigate lineage records by label to find all matching outputs workspace-wide, without knowing which specific run produced a file.

Lineage records and event delivery

When lineage is enabled, Nextflow generates a structured JSON record for each entity in your pipeline:

Record type	Description
WorkflowRun	Full pipeline execution: repository, commit ID, parameters, compute environment, session ID, and Platform context (user, workspace, pipeline)
TaskRun	Individual task execution: script, code checksum, inputs, outputs, container, and dependencies
FileOutput	Output file: path, checksum, size, timestamp, and links back to the task and workflow that produced it

Each record gets a lineage ID (LID), a lid:// URI that uniquely identifies the entity. Every LID and lineage label renders as a link you can follow to related entities across your organization.

Functional flow

1. Nextflow appends lineage record objects (*.data.json) to the defined object storage bucket.
2. The bucket is configured to filter for objects matching .data.json and publishes s3:ObjectCreated:* events to an SNS topic.
3. The SNS topic pushes each event to a per-workspace Platform webhook over HTTPS.
4. Platform verifies each delivery, buffers it, then reads the lineage object from the bucket and indexes it in the database.
5. The index enriches the run details and the display of workflow-generated objects in Data Explorer, adding links to the origin pipeline run and task, the sources of each object, and any lineage labels.

The webhook URL is unique to the workspace. It appears on the workspace lineage settings page once you save the configuration, alongside an Event delivery status that shows whether events are arriving.

Enable data lineage

To start collecting data lineage for all pipeline runs in your workspace:

1. Open Settings > Workspace settings.
2. Select Lineage. If you don't see Lineage listed, contact your system administrator.
3. Toggle Enable lineage by default on to collect data lineage for all pipeline runs in the workspace, or off to require per-pipeline launch configuration. Choose either a Manual or an Automatic configuration for lineage resources:
   • Manual: Use your own pre-provisioned bucket and SNS topic. Define the credentials, region, bucket name, and SNS topic ARN. After saving, subscribe the webhook URL shown on the settings page to your topic. See Configure lineage manually.
   • Automatic: Define the credentials and region. Platform creates the bucket, the SNS topic, the topic policies, the webhook subscription, and the bucket notification rule. This is the default setting.
4. Once set and enabled, all pipeline runs in the workspace generate data lineage. See Lineage for more information about the settings.

danger

Updating the lineage settings after pipelines have generated lineage data will result in historical data loss. The lineage index is tied to the lineage storage bucket and path. Changing it makes existing records inaccessible. To avoid data loss when updating the storage location, first copy all existing lineage data to the new bucket and path (for example, aws s3 cp --recursive s3://old-bucket/path s3://new-bucket/path), then update the workspace setting.

When launching a pipeline in a data-lineage enabled workspace, the Enable lineage toggle in the pipeline Run setup reflects the Enable lineage by default workspace setting. Turn it off to explicitly exclude data lineage for the pipeline run.

tip

Maintain role users and above can toggle lineage on or off when launching a specific pipeline run.

IAM permissions required

Data lineage requires additional AWS IAM permissions. The permissions required depend on the role:

• Platform integration credentials (IAM user): see AWS Batch — Data lineage or AWS Cloud — Data lineage
• EC2 instance role / head job role (manually managed): see Manual AWS Batch configuration

Lineage credentials grant no queue permissions. Your bucket publishes events outward to an SNS topic, which pushes them to Platform.

In Manual mode, Platform makes no control-plane calls other than confirming its own webhook subscription. The credentials need only:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ReadLineageBucket",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::<your-lineage-bucket>",
                "arn:aws:s3:::<your-lineage-bucket>/*"
            ]
        },
        {
            "Sid": "ConfirmLineageWebhook",
            "Effect": "Allow",
            "Action": [
                "sns:ConfirmSubscription"
            ],
            "Resource": "arn:aws:sns:<region>:<account>:<your-lineage-topic>"
        }
    ]
}

Configure lineage manually

In Manual mode you own the bucket, the topic, and the subscription. Before saving the workspace settings:

1. Create the S3 bucket and the SNS topic.

2. Attach a topic access policy that allows the bucket to publish to the topic:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "AllowBucketToPublishEvents",
         "Effect": "Allow",
         "Principal": { "Service": "s3.amazonaws.com" },
         "Action": "sns:Publish",
         "Resource": "arn:aws:sns:<region>:<account>:<your-lineage-topic>",
         "Condition": {
           "ArnEquals": {
             "aws:SourceArn": "arn:aws:s3:::<your-lineage-bucket>"
           }
         }
       }
     ]
   }

3. Configure a bucket notification rule that sends s3:ObjectCreated:* events for the .data.json suffix to the topic:

   {
     "TopicConfigurations": [
       {
         "Id": "LineageRecordCreated",
         "TopicArn": "arn:aws:sns:<region>:<account>:<your-lineage-topic>",
         "Events": ["s3:ObjectCreated:*"],
         "Filter": {
           "Key": {
             "FilterRules": [
               { "Name": "suffix", "Value": ".data.json" }
             ]
           }
         }
       }
     ]
   }

4. Grant the compute environment's IAM role read/write access to the bucket. See Manual AWS Batch configuration.

Then save the workspace lineage settings, copy the Webhook URL shown on the settings page, and subscribe it to your topic:

aws sns subscribe \
  --topic-arn arn:aws:sns:<region>:<account>:<your-lineage-topic> \
  --protocol https \
  --notification-endpoint '<webhook URL from the lineage settings page>'

SNS immediately posts a subscription confirmation to the endpoint, which Platform verifies and confirms with the workspace's lineage credentials. The Event delivery badge on the settings page moves from Awaiting confirmation to Active.

note

The .data.json suffix filter is recommended to reduce cost and delivery volume, but it is not required. Platform discards any event whose object key does not end in .data.json.

Lineage labels

Assign lineage labels to output files using the label directive in your Nextflow process definitions.

Both Platform labels and Nextflow lineage labels propagate to lineage records. Platform excludes resource labels because they relate to underlying compute resources, not the data itself.

info

Nextflow sets lineage labels at execution time, and they cannot be changed. Platform labels are mutable and can change after a run launches. Changing Platform labels after launch produces a mismatch between Platform run labels and Nextflow lineage labels.

Change or disable lineage

If data lineage is changed from automatically provisioned to manually provisioned:

• New object storage bucket: The bucket notification rule is cleared, and the Platform-managed SNS topic and its subscription are deleted. Some events may be missed. The bucket and its data are preserved.
• Same object storage bucket, different SNS topic: The bucket notification rule is redirected to the new topic ARN, and the old Platform-managed topic and subscription are deleted. Some events may be missed. The bucket and its data are preserved.
• Same object storage bucket, same SNS topic: No cloud provider resources change. All events, the bucket, and its data are preserved.

If data lineage is changed from manually provisioned to automatically provisioned, Platform creates a new object storage bucket, SNS topic, subscription, and bucket notification rule. Your previously defined bucket and data, topic, and notification rule are preserved.

If data lineage is deactivated with Disable lineage:

• Automatically provisioned: The notification rule is cleared on the bucket, and the Platform-managed topic and subscription are deleted. Bucket and data are preserved.
• Manually provisioned: No change to cloud resources. Bucket, topic, and data are preserved.

In both cases you can configure lineage again at any time. Records already written to the bucket are re-indexed once delivery is restored.

Lineage in the Platform UI

Platform surfaces lineage data on the run details page and in Data Explorer.

Workflow run details

For a run executed with lineage enabled, the run details page displays lineage data across the following tabs:

• Run Info: Shows the lineage ID, lineage labels, and the full Platform context captured at execution time, including user, workspace, compute environment, pipeline name, revision, and commit ID.
• Tasks: Displays the lineage ID and lineage labels for each TaskRun alongside existing task data. You can trace any task back to its lineage record. All task file inputs and outputs, and upstream and downstream tasks linked by lineage records, are displayed.
• Inputs: Lists all input datasets and parameters with file paths, types, and lineage IDs and lineage labels where available.
• Outputs: Lists all FileOutput records linked to the workflow run, including output name, file path, type, lineage ID, and lineage labels. Files link directly to Data Explorer.

tip

All LIDs and lineage labels are clickable links. Select any LID to open lineage search pre-filled with that identifier.

note

If more than one Nextflow run publishes a file to the same destination, there are two lineage records. The FileOutput records for published files are saved under the lineage ID of the workflow run and can be used to differentiate them.

Data Explorer

Output objects from a lineage-enabled run display their LID and any lineage labels when you preview the object in Data Explorer. You can trace any file back to the pipeline run that produced it.

Search data lineage records

Use the search bar in the top navigation to find workflow runs, tasks, pipelines, and output files across every workspace you can access. To open it, select Search or press Cmd+K (macOS) or Ctrl+K (Windows and Linux). Search covers only workspaces that have data lineage enabled and in which you are a participant. Results include only records you have permission to view.

Results are ordered by most recently indexed and load as you scroll. An empty query returns the most recent records across all accessible workspaces. As you type, the field suggests keywords and, where supported, values.

Search syntax

A query is a series of space-separated tokens. Each token is either a qualifier:value pair or free text. Three rules apply to every qualifier:

• A space between tokens is AND: type:file label:qc returns output files that carry the qc label.
• A comma inside a value is OR: type:workflow,task returns workflow runs and tasks.
• Repeating a qualifier is AND: label:qc label:validated returns records carrying both labels.

Qualifier names and free text are case-insensitive. Free text matches any substring of the record value. For example, salmon matches any record whose value contains salmon.

caution

A record has exactly one type and lives in exactly one workspace. Repeating type: or workspace: returns an empty list because no record can match both values. For example, type:workflow type:file requires a record to be both a workflow run and a file. Use the comma form type:workflow,file to match either type.

Qualifiers

Qualifier	Accepts	Description
type:	workflow, task, file	Restrict results to a record type. Also accepts the internal names WorkflowRun, TaskRun, and FileOutput.
label:	Any label	Records tagged with the label. Covers both Platform labels and Nextflow lineage labels.
workspace:	organization/workspace	Scope the search to one or more workspaces by fully qualified name.
workspaceId:	Numeric workspace ID	Numeric alias for workspace:.
workflow:	A WorkflowRun LID	Scope the search to a single run. Results include the run itself, its tasks, and its published output files.
pipeline:	A pipeline name	Scope the search to a pipeline name. Results include the pipeline itself and the output files in its work directory.
task:	A TaskRun LID	Scope the search to a single task. Results include the task itself and the output files in its work directory.
Free text	Any string	Case-insensitive substring match on the record value.

The field suggests workspace:, type:, and label: as you type. Enter the remaining qualifiers manually. Selecting a suggested type: or workspace: value replaces the current value for that qualifier. Selecting a suggested label: value adds another label: term to the query.

workspace: and workspaceId: set the scope of a search rather than filter its results. A query that contains only a workspace still returns that workspace's most recent records. Omit both to search every workspace available to you. Referencing a workspace you do not participate in returns an error rather than an empty list.

caution

Renaming pipelines after execution can cause data lineage consistency issues. Pipeline names are mutable by design (can be edited). Data lineage records are immutable. If you run a pipeline, generate data lineage records, and then rename the pipeline, the indexed data lineage records will not be associated with the new pipeline name.

Examples

Query	Returns
type:workflow,task	Workflow run or task records
label:qc,validated	Records labeled qc or validated
label:qc label:validated	Records labeled both qc and validated
label:qc,draft label:validated	Records labeled validated and either qc or draft
type:file salmon	Output files whose value contains salmon
type:file multiqc pipeline:rnaseq	Output files whose value contains multiqc and associated with the rnaseq pipeline
workspace:acme/dev label:qc	Records labeled qc in the acme/dev workspace
workspace:acme/dev,acme/prod	Records in the acme/dev or acme/prod workspace
workspace:acme/dev workspace:acme/prod	Nothing, because a record lives in one workspace. Use the comma form instead.
workflow:lid://abc123	The run lid://abc123, its tasks, and its published output files
workflow:lid://abc123 type:task	The tasks of run lid://abc123
task:lid://abc123 type:file	The output files of task lid://abc123

tip

Lineage search is also available through the Platform API. The GET /lineage/search endpoint accepts the same query syntax in its q parameter and returns paginated results. See the Platform API reference for the full set of lineage endpoints.

Test lineage for a single pipeline or run

To test or troubleshoot data lineage for a specific pipeline, add the following to your Nextflow config file under Advanced options when adding a pipeline to the Launchpad.

lineage.enabled = true
lineage.store.location = '<PATH_TO_STORAGE>'

To test for a single pipeline run, add the same code to your Nextflow config file under Advanced options when launching the pipeline run.

warning

If data lineage is defined for a workspace, only that data is displayed in Platform. Any unique specific pipeline or single pipeline run lineage data is only accessible via the AWS S3 console and other related services (such as Amazon Athena).

Lineage costs

Monthly S3 object storage and SNS notification costs scale based on the number of pipeline runs launched with lineage enabled. Each lineage record written to the bucket produces one SNS notification delivery.

Filtering bucket notifications to the .data.json suffix keeps delivery volume (and therefore cost) proportional to the lineage records themselves rather than to all bucket activity.