Azure Data Factory Plugin ReRun Failed Pipeline

[ConnorF]

I am trying to restart a failed Pipeline in Azure from Autosys via the ADF Plugin (Plugin.Extension.Azure.DataFactory_PE.Azure.DataFactory_1_0_15+build.1700579352485). In this example, Run 1 has Pipeline 1 succeed but Pipeline 2 fail. Restarting the Autosys Job would start from Pipeline 2 however this restarts the Pipeline from the beginning. 

The Broadcom website states as follows:
The following method will rerun a pipeline with parameters using OAuth 2.0 authentication. The failed pipeline run ID should be provided. It is not possible to change any parameters from the previous run.
Class: com.broadcom.pojo.adf.AzureDataFactory
Method: runPipeline
https://techdocs.broadcom.com/us/en/ca-enterprise-software/intelligent-automation/workload-automation-plugin-extensions/GA/workload-automation-agent-plugin-extension/azure-data-factory-plugin-extension/azure-data-factory-plugin-extension---methods.html#concept.dita_d3303fde-e786-4fd4-b0b6-e3a28fd60a82_8


Assuming we would need to pass in the run Id of the failed pipeline which we get when the initial job is submitted as the last J2EE parameter however not sure how we would get to that?

[Richard Kao]

You would need to retrieve the spool file for the failed job.  In the spool output you will see a line similar to this...

2024-11-13T06:01:18.237-05:00: Run ID: c02c4d3e-b34a-4bc8-97d7-9d61c0018e39

[ConnorF]

Hi Richard,

Thanks for this. We are re-directing the spool to a log file on the scheduler and have found the Run ID. Is it possible to automate the extraction of this and populate "adf.PipelineRunID" so that the job can restart automatically as we have two retries defined?

Thanks,

Connor

[Richard Kao]

Currently there is no direct way to do it.  However you could run a script which runs the following to extract the Run ID from the spool file and set a global variable.

sendevent -E SET_GLOBAL -G RUNID=`autosyslog -t S -J adf1_pipeline_parameters_FAIL | grep "Run ID: " | awk '{printf $4}'`

The above example will extract the Run ID from the spool file for the adf1_pipeline_parameters_FAIL job and use sendevent to set a global variable RUNID in AutoSys.  In the successor job you would use the global variable RUNID as the parameter value.  The example will work with UNIX and Linux machines.

Since this use case is not handled directly in the product, can you open a enhancement request with support.

[Sudheer Putta]

i think you have got this resolved, please see below link the latest from Broadcom
Azure Data Factory Job

restart_pipeline_run_id