Prerequisites

1. Java Runtime Environment installed and added to the PATH variable - only Open JDK 11 is supported and tested with Azkaban and RED.

2. PuTTY or similar tools for remote installation and interactive task execution from Windows.

Installation

Currently, WhereScape Azkaban installation on Linux is performed using the azkaban-installer.jar file located at <RED_install_dir>\Azkaban\. You need to transfer this file to your Linux installation and run it with Java.

You also need to supply a pre-configured response file containing all the parameters for the installation. You can use the provided web example response file, and executor example response file remember to modify the file with your own settings.

Azkaban-installer command line help

The same installer can be run from Windows or Linux using the following command to output the installer command line help:

java -jar azkaban-installer.jar --help

Which outputs:

Usage: azkaban-installer [-hV] [--log-file=FILE] [COMMAND]
   --log-file=FILE   File name (including path) for the log file (default: C:
                          \Users\Red1\AppData\Local\Temp\WsSchedulerInstaller_2023050
                          2_201952_803.log)
  -h, --help            Show this help message and exit.
  -V, --version         Print version information and exit.
Commands:
  example-response-file    Generates an example well-formed XML response file
                             to standard output
  install-server           Installs or upgrades an Azkaban server as described
                             in the response file
  mandatory-executor-tags  Outputs the executor tags that should always be
                             added to executors running on the current machine.
  response-file-schema     Writes the XML schema for the response file to
                             standard output
  test-connection          Tests that a JDBC connection to the specified
                             database can be established.
  upgrade-schema           Upgrades or installs the Scheduler metadata schema.

Azkaban Installation

1. Install the Azkaban Metadata

2. Install the Azkaban Web Server

3. Install one or more Azkaban Executor Servers

4. Add the Scheduler configuration to RED

1. Install the Azkaban Metadata

To install the Azkaban metadata, perform the following steps in the command shell.

• First, set the environment variables for your Azkaban metadata user and pwd:
  AZ_DB_USER=myuser
AZ_DB_PWD=mypwd
• Then, run the azkaban-installer.jar with the upgrade-schema command, change AZ_REPO to the scheduler database and change PG_HOST to the server that runs Postgres:
  

  java -jar "azkaban-installer.jar" upgrade-schema --database=AZ_REPO --host=PG_HOST --port=5432 --schema=white --username-var=AZ_DB_USER --password-var=AZ_DB_PWD

2. Install the Azkaban Web Server

On the Linux machine that will host the main Web Server, open a command shell and navigate to the folder that contains both azkaban-installer.jar and web_server_response_file.xml. Next, run the following command:

java -jar azkaban-installer.jar --log-file=./install.log install-server --response-file=web_server_response_file.xml

3. Install one or more Azkaban Executor Servers

On the Linux machine that will host the Executor Web Server, open a command shell and navigate to the folder that contains both azkaban-installer.jar and executor_server_response_file.xml. Next, run the following command:

java -jar azkaban-installer.jar --log-file=./install.log install-server --response-file=executor_server_response_file.xml

4. Add the scheduler configuration to RED

To start using this scheduler in RED first, the Scheduler Configuration needs to be added to RED via the Scheduler tab.

Once in RED, navigate to the Scheduler tab and select the menu item Scheduler→Configuration, this will open the configuration for the attached scheduler.

• Enter your URLs for the Azkaban Web Server you setup earlier into both the Dashboard and Rest API URL fields.
• Provide the credentials for Dashboard User into the Dashboard section
• Provide the credentials for Publish User in the Rest API section
• Add your default scheduler tags which you setup on your Executor servers (these are case sensitive so ensure you have the same case as in your response file)
• Set the default scripts as shown in the example below

Example response files:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<installServerResponse>
    <serverType>WEB_SERVER</serverType>
    <serverName>WEB_SERVER</serverName>
    <destination>
        <destinationDirectory>/home/AzkabanSchedulerUser/WhereScape/</destinationDirectory>
    </destination>
    <metadataDatabase>
        <databaseName>schedmeta</databaseName>
        <hostName>pgserver</hostName>
        <password>schedpassword</password>
        <port>5432</port>
        <schema>white</schema>
        <userName>scheduser</userName>
    </metadataDatabase>
    <redMetadataDatabase>
        <databaseName>redmeta</databaseName>
        <hostName>pgserver</hostName>
        <password>redpassword</password>
        <port>5432</port>
        <schema>red</schema>
        <userName>reduser</userName>
    </redMetadataDatabase>
    <serverConfig>
        <executorTags></executorTags>
        <port>8001</port>
        <timeZone>Pacific/Auckland</timeZone>
        <webServerExternalHostName>AzkabanServer</webServerExternalHostName>
        <webServerExternalPort>8001</webServerExternalPort>
        <adminUserName></adminUserName>
        <adminPassword></adminPassword>
        <dashboardUserName>wsdashboarduser</dashboardUserName>
        <dashboardPassword>wsdashboardpass</dashboardPassword>
        <publishUserName>wsapiuser</publishUserName>
        <publishPassword>wsapipass</publishPassword>
    </serverConfig>
    <emailConfig>
        <host>emailhost</host>
        <port>587</port>
        <user>emailuser</user>
        <password>emailpassword</password>
        <sender>senderemail</sender>
        <tls>true</tls>
        <useAuth>true</useAuth>
    </emailConfig>
    <redBindir>/home/AzkabanSchedulerUser/WhereScape/</redBindir>
	<azkabanPasswordEncryption>NONE</azkabanPasswordEncryption>
	<azkabanWalletCmd></azkabanWalletCmd>
	<redProfilePassword>redprofilepass</redProfilePassword>
	<wherescapeJobWorkdir>/tmp</wherescapeJobWorkdir>
</installServerResponse>

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<installServerResponse>
    <serverType>EXEC_SERVER</serverType>
    <serverName>EXEC_SERVER</serverName>
    <destination>
        <destinationDirectory>/home/AzkabanSchedulerUser/WhereScape/</destinationDirectory>
    </destination>
    <metadataDatabase>
        <databaseName>schedmeta</databaseName>
        <hostName>pgserver</hostName>
        <password>schedpassword</password>
        <port>5432</port>
        <schema>white</schema>
        <userName>scheduser</userName>
    </metadataDatabase>
    <redMetadataDatabase>
        <databaseName>redmeta</databaseName>
        <hostName>pgserver</hostName>
        <password>redpassword</password>
        <port>5432</port>
        <schema>red</schema>
        <userName>reduser</userName>
    </redMetadataDatabase>
    <serverConfig>
        <executorTags>
			<tag>linux</tag>
		</executorTags>
        <port>8002</port>
        <timeZone>Pacific/Auckland</timeZone>
        <webServerExternalHostName>AzkabanServer</webServerExternalHostName>
        <webServerExternalPort>8001</webServerExternalPort>
        <adminUserName></adminUserName>
        <adminPassword></adminPassword>
        <dashboardUserName></dashboardUserName>
        <dashboardPassword></dashboardPassword>
        <publishUserName></publishUserName>
        <publishPassword></publishPassword>
    </serverConfig>
    <emailConfig>
        <host>emailhost</host>
        <port>587</port>
        <user>emailuser</user>
        <password>emailpassword</password>
        <sender>senderemail</sender>
        <tls>true</tls>
        <useAuth>true</useAuth>
    </emailConfig>
    <redBindir>\home\AzkabanSchedulerUser\WhereScape\</redBindir>
 	<azkabanPasswordEncryption>NONE</azkabanPasswordEncryption>
	<azkabanWalletCmd></azkabanWalletCmd>
	<redProfilePassword>redprofilepass</redProfilePassword>
	<wherescapeJobWorkdir>/tmp</wherescapeJobWorkdir> 
</installServerResponse>

For more information about the wherescapeJobWorkdir property, refer to the Work Directory section in RED Scheduler Configuration.

RED has implemented two password values for the azkabanPasswordEncryption property for Linux: WALLET and NONE

• Use WALLET and set the value of azkaban.walletCmd to the system command that retrieves the password from the wallet. 
  For more details on the WALLET option refer to the Using WALLET Encryption section on RED Scheduler Configuration.
  For more information on using a standard Linux password manager, visit https://www.passwordstore.org/
• If you set the value to NONE you need to enter all passwords as plain text, they are also stored as plaintext on the config files.

Example Install Command:

Typical installation command assuming the azkaban-installer.jar has been copied to the home directory for the running user on Linux:

java -jar ~/azkaban-installer.jar --log-file=~/install.log install-server --response-file=~/response.xml

Starting and Stopping the Azkaban Servers

On Linux, the Azkaban Servers are not started automatically after installation to run the startup script.

• <azkaban_install_dir>/<web_server_name>/azkaban-web-server/bin/start-web.sh

• <azkaban_install_dir>/<exec_server_name>/azkaban-exec-server/bin/start-exec.sh

There are various ways to have these servers start when Linux starts. The simplest is to add it via crontab with the @reboot keyword.

For example:

crontab -e Opens crontab editor for the current user

Add a line for each server, save, and close the editor:

• @reboot <azkaban_install_dir>/<web_server_name>/azkaban-web-server/bin/start-web.sh

• @reboot <azkaban_install_dir>/<exec_server_name>/azkaban-exec-server/bin/start-exec.sh

crontab -l Lists the current users' cron jobs.