Page History

When running scheduled jobs from WhereScape RED, it's possible to configure a job to run commands when the job finishes successfully or when it a task in the job fails. These commands can be set to send trigger email notifications or , maintenance tasks, create a log file about failure or success status, or other actions. 

Success and Failure commands can use variables and tokens, this allows them to use host scripts stored on the metadata repository instead of setting up scripts for each existing scheduler.

Success Commands

Failure Commands

Along with supporting the existing RED 9 behavior we have also added the capability to use the token format for referencing scripts from the metadata within the success/failure commands, which means you don't need to store a static script somewhere for each scheduler and instead pick it up from a host script stored in the metadata.

The version of Azkaban we ship has a known issue in that even for successful conditional execution the job in Azkaban is marked as cancelled, because in either execution path, success or failure, there are items in the flow that are skipped (i.e the failure step is skipped for success and the success is skipped on failure).

image-20251027-230347.png
 

Due to the issue above the proposed solution in the original description below might not be acceptable. In later version of Azkaban this behavior was apparently changed so that skipped tasks due to conditional workflows would not mark the job as KILLED.

If the failure command is populated execute the Failure Command as a system command when a job is being marked as failed by task thread.

If the job is being failed due to a parent job dependency failure then conditionally run the command based on the setting ‘Execute Failure Command in event of dependency failure’.

Ensure the command is only executed once for the job, instead of each failing thread which executing tasks in parallel.

If the success command is populated execute the Success Command as a system command when a job is being set to successful completion in the final stage of execution.

Expansion of existing documented tokens in the success/failure command should be supported.

The success (exit code 0) or failure (exit code non-zero) of the command should be reported in the job log. i.e.

Running Success Command [succeeded|failed]

Running Failure Command [succeeded|failed]

Set the usual script environment variables such as WSL_META*, job, task, sequence and scheduler api prior to running the command.

Allow the existing token support for script sourcing by looking for tokens in the form  $WSL_SCRIPT_<script_name_from_metadata>_CODE$ and replacing them with the full path of the metadata script which should be written to disk at that location.

Enable execution of success/failure commands directly in our plugin RED-12369: RED 10 success and failure command support
In QE
 

Create a single task at the end that takes both success and failure commands as args and run one or the other after first determining the execution's state via the REST API. This would work well with the RED-12091: Add a new method to schedulerUtils.jar such that an individual script can be run
Closed
 enhancement where we could run any script from the metadata. This would require changes to the wsl_sceduler_publish script. These parameters could be passed to scripts: https://submitteddenied.github.io/azkaban2/documents/2.1/jobconf.html

Success and Failure commands in Jobs are not yet catered for out of the box in RED 10.4
Reqs:

Any placeholder tokens in the success/failure commands should be converted to the equivalent tokens available in Azkaban.

wsl_scheduler_publish script should be updated to include success failure commands

We will soon be releasing a RED version 10.6.0.2 that adds success/failure commands directly to the RED Scheduler Plugin for Azkaban so that the existing RED 9 behavior works directly in RED 10 without the need to change the standard publish scripts to get it working. 

...

The commands can be entered in the Job Definition screen as shown below. It is important to make sure the commands you enter reflect the operating system (Linux or Windows) where the job will run or be portable between environments, as the commands entered are simply run as a system command where the job is being executed.

Image AddedJob Definition screen

Success Commands

Success commands run when a job is a marked as successful in the final stage of Job execution and the Success Command field is populated. The example used in the Job definition screen above simply outputs the values of the special tokens to a log file which also uses the tokens to form the log file name, thus creating a unique log for each run.

echo "Job $JOB_KEY$, Seq $JOB_SEQ$, Name $JOB_NAME$" >"c:\temp\$JOB_SEQ$_success_$JOB_KEY$_$JOB_NAME$.log"

Failure Commands

Failure commands run when a task marks a job as failed and the Failure Command field is populated. This command will only run once for the job even if more than one task fails in multi-threaded jobs.

If the Execute Failure Command in event of dependency failure option is set the failure command will also be run when the following conditions are met:

• The job has a Frequency set other than 'None'
• The job has at least one parent job dependency defined
• One of the parent jobs has failed

Success and Failure Command Parameters

Special Variables or "Tokens" 

• $JOB_KEY$ - replaced with the Job ID
• $JOB_SEQ$ - replaced with the Job Sequence Number
• $JOB_NAME$ - replaced with the Job Name
• $WSL_SCRIPT_<host script name in metadata>_CODE$ - Script Sourcing Tokens

RED Parameter tokens are not currently supported:

• $P<parameter_name>$ - RED Parameter tokens will be supported in a later release, instead please use the parameter read and write functions within your script if required.

Environment Variables

The following RED environment variables are set prior to running the command

• WSL_META_<variable name> - all the usual metadata environment variables available to scripts
• WSL_<JOB|TASK>_<KEY|NAME> - Job and Task environment variables
• WSL_<SEQUENCE|WORKDIR|BINDIR> - Sequence, Work Directory and Bin Directory environment variables
• WSL_SCH_RESTAPI_<URL|USER|PWD> - The Scheduler REST API URL and credentials

If the Job is executed via Azkaban Scheduler then you will also have the usual Azkaban Environment variables available, the most useful being JOB_NAME

• JOB_NAME - this equates to the name of the task in the job at the time the Success/Failure command was run, useful to determine the task in the job that has failed.  

Other Credentials

If you need to pass other credentials securely to your scripts or commands such as email server credentials then this can be achieved by utilizing one or more of the Advanced Authentication Parameters on the metadata connection. These fields will be available to the script or command as environment variables:

• WSL_META_<AUTHTKN_1|AUTHTKN_2|AUTHTKN_3|AUTHCFG>

Success and Failure Command Script Sourcing

RED can reference other scripts form the metadata at run-time, this Script Sourcing feature of RED is also available to be used in the Success and Failure Commands, this allows referencing a Host Script from the RED metadata rather than a script that already exists on the operating system.

In the following example command, you would have a python Host Script in the metadata named 'my_python_notify_script' which performs a notification based on two args, the job status and the email to notify. In the Success command you would pass an arg of 'success' and in the Failure command you can pass an arg of 'failed' so that the script can determine what the status is. The script itself will have access to the appropriate environment variables to construct a suitable email that informs of the failure or success.

python "WSL_SCRIPT_my_python_notify_script_CODE" "success" "job-admin@mydomain.com"

When a Success or Failure Command is triggered RED will do the following:

• Parse the command for RED Tokens
  • Write referenced metadata Host Scripts found in any Script Sourcing Tokens to the work directory  
  • Expand the RED Tokens in the command line, Script Sourcing Tokens are replaced with the full file path of the script
• Set up the environment variables
• Execute the expanded command
• Report if the command ran successfully or not (this has no bearing on the Job Status)

Success and Failure Command Output Protocol

Only the exit code of the Success or Failure command is considered by the Scheduler, all output streams are ignored.

• An exit code of '0' signals a successful execution of the command, this is reported in the Scheduler Job Log in RED
• A non-zero exit code signals a failure executing the command, this is also reported in the Scheduler Job Log in RED
• All output streams are ignored, therefore your script or command should perform and persist it's own logging if required.