Running Pipelines¶

Execute Nextflow and WDL workflows using the iFlow CLI.

Overview¶

The iFlow platform supports running bioinformatics pipelines on Google Cloud Batch. Pipelines can be:

Nextflow workflows - Standard nf-core and custom Nextflow pipelines
WDL workflows - WDL/Cromwell-based workflows

Prerequisites¶

Installed CLI - See Installation
Authenticated - Run iflow login
Project access - You need access to a project with GCP credentials configured

Available Pipelines¶

List available pipelines:

iflow pipelines list

Output:

SLUG                      VERSION    NAME                      MODE
---------------------------------------------------------------------------------
hereditary-mock           1.0.0      Hereditary Mock Pipeline  direct_wdl
wdl-minimal               1.0.0      WDL Minimal Pipeline      direct_wdl
nextflow-minimal          1.0.0      nextflow-minimal          container_entrypoint

Get details about a specific pipeline:

iflow pipelines info hereditary-mock
# Get specific version:
iflow pipelines info hereditary-mock -V 1.0.0

List all versions of a pipeline:

iflow pipelines versions hereditary-mock

View pipeline inputs and outputs:

iflow pipelines describe hereditary-mock

Pipeline Types¶

Type	Description	Engine
`direct_nextflow`	Runs standard Nextflow pipeline (e.g., nf-core workflows)	Nextflow
`direct_wdl`	Runs WDL workflow via Cromwell	Cromwell
`container_entrypoint`	Runs container with custom entrypoint (e.g., nextflow-minimal)	Custom

Submitting a Run¶

Basic Submission¶

iflow analyses submit \
  --project PROJECT_ID \
  --pipeline nextflow-minimal \
  -P analysis_name=my_analysis

# With specific pipeline version
iflow analyses submit \
  --project PROJECT_ID \
  --pipeline hereditary-mock \
  -V 1.0.0 \
  -P case_id=case-001

With Parameters¶

Pass parameters using -P KEY=VALUE:

iflow analyses submit \
  --project PROJECT_ID \
  --pipeline nextflow-minimal \
  -P nf_profile_name=client_config_test \
  -P analysis_name=my_analysis \
  -P sample_id=sample1 \
  -P fastq_tumor=gs://bucket/R1.fastq.gz \
  -P fastq_tumor=gs://bucket/R2.fastq.gz

Array Parameters

For parameters that accept multiple values (like paired FASTQ files), use -P key=value multiple times with the same key.

With Watch Mode¶

Automatically monitor the run after submission:

iflow analyses submit \
  --project PROJECT_ID \
  --pipeline nextflow-minimal \
  -P analysis_name=test \
  --watch

Monitoring Runs¶

Check Status¶

iflow analyses status RUN_ID

Output includes: - Current status (queued, running, succeeded, failed, cancelled) - Timestamps (created, started, finished) - Parameters used - Output path (on success) - Error message (on failure)

Watch Until Completion¶

iflow analyses watch RUN_ID --interval 15

The watch command polls the status and displays updates:

[12:00:15] Status: queued
[12:00:30] Status: running
           Started: 2026-01-08T12:00:25Z
[12:05:30] Status: succeeded

Analysis completed successfully!
Output: gs://bucket/outputs/run-abc123/

Press Ctrl+C to stop watching (the run continues in the background).

List Runs¶

View recent runs for a project:

iflow analyses list --project PROJECT_ID --limit 10

Cancelling Runs¶

Cancel a running or queued run:

iflow analyses cancel RUN_ID

You'll be prompted to confirm the cancellation.

Example: nextflow-minimal Pipeline¶

The nextflow-minimal pipeline processes FASTQ files through a minimal Nextflow workflow.

Required Parameters¶

Parameter	Description
`nf_profile_name`	Nextflow profile (e.g., `client_config_test`)
`analysis_name`	Name for the analysis
`sample_id`	Sample identifier
`fastq_tumor`	Path(s) to FASTQ files (use twice for paired)

Full Example¶

# Login
iflow login -e user@example.com -p yourpassword

# Verify pipeline exists
iflow pipelines info nextflow-minimal

# Submit run
iflow analyses submit \
  --project 01234567-89ab-cdef-0123-456789abcdef \
  --pipeline nextflow-minimal \
  -P nf_profile_name=client_config_test \
  -P analysis_name=test_$(date +%Y%m%d_%H%M) \
  -P sample_id=sample1 \
  -P fastq_tumor=gs://e2e-e2edev/test-data/sample_small_R1.fastq.gz \
  -P fastq_tumor=gs://e2e-e2edev/test-data/sample_small_R2.fastq.gz \
  --watch

Expected Output¶

On successful completion:

Analysis submitted successfully!
  ID: abc123-def456-...
  Name: nextflow-minimal-1736339200
  Status: queued

Watching run status (Ctrl+C to stop)...
[12:00:15] Status: queued
[12:00:30] Status: running
           Started: 2026-01-08T12:00:25Z
[12:05:30] Status: succeeded

Analysis completed successfully!
Output: gs://bucket/outputs/nextflow-minimal-1736339200/

Example: WDL Minimal Pipeline¶

The wdl-minimal pipeline demonstrates WDL/Cromwell execution on Flow.

WDL Pipeline Features¶

Cromwell execution - WDL workflows run via Cromwell
No inputs required - The minimal pipeline runs without parameters
Metadata output - Results include Cromwell metadata JSON

Submitting WDL Run¶

# Check pipeline info
iflow pipelines info wdl-minimal

# Submit WDL run (no parameters needed for minimal pipeline)
iflow analyses submit \
  --project PROJECT_ID \
  --pipeline wdl-minimal \
  --watch

Example: Hereditary Mock Pipeline¶

The hereditary-mock pipeline demonstrates WDL execution with actual inputs (FASTQ files).

Pipeline Overview¶

Type: WDL/Cromwell (direct_wdl)
Purpose: Mock hereditary analysis for testing input handling
Required inputs: case_id, child_fastq files

Required Parameters¶

Parameter	Type	Description
`case_id`	String	Case identifier for the analysis
`child_fastq`	Array[File]	Child/proband FASTQ files (R1, R2)

Optional inputs include mother_fastq, father_fastq, patient metadata, HPO terms.

Submitting Hereditary Mock Run¶

# Check pipeline info
iflow pipelines info hereditary-mock

# Submit with FASTQ inputs
iflow analyses submit \
  --project PROJECT_ID \
  --pipeline hereditary-mock \
  -P case_id=case-001 \
  -P child_fastq=gs://bucket/sample_R1.fastq.gz \
  -P child_fastq=gs://bucket/sample_R2.fastq.gz \
  --watch

Complete Walkthrough¶

1. Login to iFlow

iflow login -e your@email.com -p yourpassword

2. Check pipeline details

iflow pipelines info hereditary-mock

Output:

Pipeline: Hereditary Mock Pipeline
  Slug: hereditary-mock
  Version: 1.0.0
  ID: abc123-def456-...
  Description: Mock hereditary analysis pipeline for WDL/Cromwell testing
  Source Type: gitlab
  Execution Mode: direct_wdl
  Default Profile: local
  Active: Yes

View inputs/outputs:

iflow pipelines describe hereditary-mock

3. Submit the run with FASTQ inputs

iflow analyses submit \
  --project 01c965b9-3366-4a09-81f0-dcc88375e76a \
  --pipeline hereditary-mock \
  -P case_id=test-case-001 \
  -P child_fastq=gs://e2e-e2edev/test-data/regions_R1.fastq.gz \
  -P child_fastq=gs://e2e-e2edev/test-data/regions_R2.fastq.gz \
  --watch

Output:

Analysis submitted successfully!
  ID: hereditary-mock-20260112-134500
  Status: queued

Watching run status (Ctrl+C to stop)...
[13:45:15] Status: queued
[13:46:00] Status: running
           Started: 2026-01-12T13:45:50Z
[13:48:00] Status: succeeded
           Finished: 2026-01-12T13:47:45Z

Analysis completed successfully!
Duration: 2m 30s

4. Check run status

iflow analyses status hereditary-mock-20260112-134500

WDL Pipeline Output¶

Analysis submitted successfully!
  ID: wdl-minimal-pipeline-20260110-...
  Status: queued

Watching run status (Ctrl+C to stop)...
[12:00:15] Status: queued
[12:01:30] Status: running
[12:03:30] Status: succeeded

Analysis completed successfully!
Output: gs://bucket/outputs/wdl-minimal-pipeline-.../

WDL with Parameters¶

For WDL pipelines that require inputs:

iflow analyses submit \
  --project PROJECT_ID \
  --pipeline germline-pipeline \
  -P sample_id=sample1 \
  -P input_bam=gs://bucket/sample.bam

Complete WDL Walkthrough (Minimal Pipeline)¶

Step-by-step guide to run the WDL minimal pipeline:

1. Login to iFlow

iflow login -e your@email.com -p yourpassword

2. List available pipelines

iflow pipelines list

Output:

SLUG                      VERSION    NAME                      MODE
---------------------------------------------------------------------------------
hereditary-mock           1.0.0      Hereditary Mock Pipeline  direct_wdl
wdl-minimal               1.0.0      WDL Minimal Pipeline      direct_wdl
nextflow-minimal          1.0.0      nextflow-minimal          container_entrypoint
nfcore-demo               1.0.0      nf-core Demo Pipeline     direct_nextflow

3. Get WDL pipeline details

iflow pipelines info wdl-minimal

Output:

Pipeline: WDL Minimal Pipeline
  Slug: wdl-minimal
  Version: 1.0.0
  ID: abc123-def456-...
  Description: Minimal WDL pipeline for testing Cromwell execution
  Source Type: gitlab
  Execution Mode: direct_wdl
  Default Profile: local
  Active: Yes

4. Submit the WDL run

iflow analyses submit \
  --project 01c965b9-3366-4a09-81f0-dcc88375e76a \
  --pipeline wdl-minimal \
  --watch

Output:

Analysis submitted successfully!
  ID: wdl-minimal-pipeline-20260110-210636
  Status: queued

Watching run status (Ctrl+C to stop)...
[21:06:51] Status: queued
[21:07:06] Status: queued
[21:08:06] Status: running
           Started: 2026-01-10T21:08:00Z
[21:09:06] Status: running
[21:09:36] Status: succeeded
           Finished: 2026-01-10T21:09:30Z

Analysis completed successfully!
Duration: 1m 30s

5. Check run details

iflow analyses status wdl-minimal-pipeline-20260110-210636

Output:

Analysis: wdl-minimal-pipeline-20260110-210636
  Pipeline: WDL Minimal Pipeline
  Status: succeeded
  Created: 2026-01-10T21:06:36Z
  Started: 2026-01-10T21:08:00Z
  Finished: 2026-01-10T21:09:30Z
  Duration: 1m 30s

What happens during execution:

GCP Batch creates a VM with the Cromwell container
The WDL file is downloaded from workflows.intelliseq.com
Cromwell parses and validates the workflow
Tasks are executed locally in the container
Results are captured in Cromwell metadata
Job completes and status is updated

Order-Run Association¶

Runs can be associated with clinical orders from the Miner service. This enables tracking which pipeline runs were executed for which clinical cases.

Submitting a Run with Order ID¶

Use --order-id (or -o) to associate a run with an order:

iflow analyses submit \
  --project PROJECT_ID \
  --pipeline hereditary-mock \
  --order-id ORDER_ID \
  -P case_id=case-001 \
  -P child_fastq=gs://bucket/R1.fastq.gz \
  -P child_fastq=gs://bucket/R2.fastq.gz \
  --watch

Listing Runs by Order¶

View all runs associated with a specific order:

iflow analyses list -p PROJECT_ID --order-id ORDER_ID

Output:

ID                                       NAME                                STATUS
------------------------------------------------------------------------------------------
8e9beaa2-3be1-482a-a313-75ea64876ec3     hereditary-mock-20260113-000500     succeeded

Checking Order ID in Run Status¶

When viewing run details, the associated order ID is displayed:

iflow analyses status RUN_ID

Output:

Analysis: hereditary-mock-20260113-000500
  ID: 8e9beaa2-3be1-482a-a313-75ea64876ec3
  Status: succeeded
  Pipeline ID: 302ca72c-df9c-4733-8b3d-174ab82e5fc4
  Order ID: ORDER_ID
  Profile: local
  Created: 2026-01-13T00:05:00Z
  Started: 2026-01-13T00:06:00Z
  Finished: 2026-01-13T00:08:00Z
  Output: gs://bucket/nf-results/hereditary-mock-20260113-000500

Complete Order-Run Workflow¶

This example shows the full workflow from login to run completion with order association:

1. Login

iflow login --token YOUR_PAT_TOKEN
# Or: iflow login -e email@example.com -p password

2. Submit run with order association

iflow analyses submit \
  --project 01c965b9-3366-4a09-81f0-dcc88375e76a \
  --pipeline hereditary-mock \
  --order-id my-clinical-order-001 \
  -P case_id=patient-12345 \
  -P child_fastq=gs://e2e-e2edev/test-data/regions_R1.fastq.gz \
  -P child_fastq=gs://e2e-e2edev/test-data/regions_R2.fastq.gz \
  -t clinical \
  -t hereditary \
  --watch

3. Check run status

iflow analyses status hereditary-mock-20260113-...

4. List all runs for the order

iflow analyses list -p PROJECT_ID --order-id my-clinical-order-001

API Access¶

For programmatic access, the order-run association is also available via the API:

Submit run with order_id:

curl -X POST https://compute.flow.labpgx.com/api/v1/runs \
  -H "Authorization: Bearer TOKEN" \
  -H "X-Project-ID: PROJECT_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "pipeline_id": "PIPELINE_ID",
    "order_id": "ORDER_ID",
    "params": {"case_id": "test"},
    "tags": ["clinical"]
  }'

Get runs by order:

curl https://compute.flow.labpgx.com/api/v1/runs/by-order/ORDER_ID

Callback Integration (LIS)¶

For automated LIS integration, you can specify a callback URL that receives a webhook when the run completes or fails.

Submitting with Callback URL¶

iflow analyses submit \
  --project PROJECT_ID \
  --pipeline hereditary-panel \
  -P vcf_file=gs://bucket/sample.vcf.gz \
  --callback-url https://your-lis.example.com/api/callback \
  --watch

When the run completes, iFlow sends a POST request to your callback URL with:

On success: Run details + signed URLs for output files (valid 1 hour)
On failure: Run details + error code and message

Example Callback Payload¶

{
  "run_id": "run-abc123",
  "status": "succeeded",
  "outputs": {
    "report_pdf": "https://storage.googleapis.com/...?signature=..."
  }
}

See LIS Integration Guide for complete payload structure and implementation details.

Troubleshooting¶

"Not authenticated"¶

Run iflow login to authenticate:

iflow login -e your@email.com -p yourpassword

"Resource not found" for Pipeline¶

The pipeline may not be configured. Check available pipelines:

iflow pipelines list

Run Stuck in "queued"¶

Possible causes: - GCP Batch quota limits - Service account permission issues - Network connectivity problems

Check with your administrator.

"credentials" Error¶

The project needs GCP credentials configured: 1. Open the Admin Console 2. Navigate to your project 3. Add GCP credentials under Settings

Run Failed¶

Check the error message:

iflow analyses status RUN_ID

Common issues: - Invalid input file paths - Insufficient compute resources - Pipeline-specific parameter errors

Next Steps¶

Commands Reference - Full command documentation
Compute Service - Web interface for pipelines