Linux Prerequisites

Before you begin, ensure the following prerequisites are met:

Docker requires a 64-bit installation with a kernel version of 3.10 or higher. You can check your kernel version using:

uname -r

Docker Installation

Follow the official Docker installation guide:

Docker Engine Installation - https://docs.docker.com/engine/install

Docker Compose Installation

Follow the official Docker Compose installation guide:

Docker Compose Installation - https://docs.docker.com/compose/install

Steps to Install

  1. Locate and update the .env file: Update the properties in the .env file with the correct values, including RED repository details such as DSN, database name, host, etc., along with the target connection details (DSN, server name, and database name).
  2. Copy the Azkaban Installer: Navigate to the Azkaban folder and copy the azkaban-installer.jar file found in the Azkaban folder within RED's root directory (C:\Program Files\WhereScape\RED). Ensure that the correct version of the RED .jar file is used.
  3. Update XML Files: In the same Azkaban folder, update the web_server_response_file.xml and executor_server_response_file.xml. If you have pre-configured files, replace them here with the same names. 

Installation Steps Using Docker Compose

  1. Install the Azkaban Metadata: Inside the Azkaban Docker Compose package directory, install the metadata by running the following command:
    docker compose build upgrade-schema
  2. Install the Azkaban Web Server: Build the Azkaban Web Server by specifying the target warehouse:
    docker compose build --build-arg TARGET_WAREHOUSE=SqlServer web
  3. Install Azkaban Executor Servers: Build the Azkaban Executor servers, also specifying the target warehouse:
    docker compose build --build-arg TARGET_WAREHOUSE=SqlServer exec
  4. Start the Azkaban Web Server in the Background 
    docker compose up web -d
  5. Start the Azkaban Executor Server 
    docker compose up exec -d
  6. Start Additional Azkaban Executor Servers
    Scale up the executor servers if needed:
    docker compose scale exec=2

Final Steps

Run wsl_scheduler_profile_maintenance Script: After setup is completed, run the wsl_scheduler_profile_maintenance script from the RED host script. Ensure the profile encryption secret passed in the script matches the redProfilePassword in both the web_server_response_file.xml and executor_server_response_file.xml XML files.

Useful Docker Compose Commands

  1. Show Azkaban Web Server Logs
    • To view logs:
      docker compose logs web
    • To follow logs:
      docker compose logs --follow web
    • To show the last 100 lines and follow the logs:
      docker compose logs --follow --tail=100 web
  2. Bash into Azkaban Web Server
    docker compose exec web bash
  3. Stop the Azkaban Web Server
    docker compose stop web
  4. Stop Azkaban Executor Server
    docker compose stop exec

Update

Rebuild Web and Executor Servers

For example, when there is a newer version of azkaban-installer.jar, copy it into the azkaban folder inside the Docker Compose package and run the following commands:

Stop and remove the running containers:

docker compose down web
docker compose down exec

Rebuild

docker compose build web
docker compose build exec
  • No labels