Contributing plugins for Uptime Infrastructure Monitor

Introduction

With the introduction of the Extension Manager in up.time 7.3, plugins and gadgets are now even easier to install. As part of this change, we have also posted the source code for all plugins on GitHub. Use these examples as starting templates for building your own plugins.

What's a plugin?

Plugins are custom service monitors that allow Uptime Infrastructure Monitor to monitor and collect metrics from a variety of different sources. In most cases they're a combination of XML which defines the input and output variables, and some simple shell scripts that handle the actual task of monitoring. These monitors usually consists of some sort of login or connectivity check, as well as exposing key metrics in a format that can be understood by UIM.

The specifics of the scripts themselves depend on what protocol and libraries are required to the talk to device/system/service on the other end. But this topic explains how to use element details from UIM to control where these scripts are targeted as well as how to define your own input variables that are stored as service monitor settings in UIM.

Input variables

UIM uses environment variables as the main way of exposing information to all custom scripts. The most important of these environment variables is UPTIME_HOSTNAME which provides the hostname of the element the service monitor is running against. Next comes the various input variables from the monitor's xml file, which provide the specific details needed for the script itself. The best way to understand how this works is to take a look at an example of the xml definition a monitor, how that looks as service monitor settings in the GUI, and finally how all this information ends up as environment variables in the script.

A good example is the Custom Remote Monitor as this plugin also provides a great starting point for custom monitors that also rely on an agent-side script. But for now we're just going to focus on the XML details and the script that runs directly on the monitoring station itself. The full contents of the xml file are available on GitHub at this link.

XML - Grid Info

The first section of the XML file contains details about the name of the plugin/service monitor and which category its shows up under on the Add Service Monitor screen.

<uptime xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.uptimesoftware.com/erdc/erdc erdc" xmlns="http://www.uptimesoftware.com/erdc/erdc" xmlns:upt="http://www.uptimesoftware.com/erdc/erdc" version="4.0" target_grouping="other">
     <class>Custom Remote Monitor</class>
     <exec>com.uptimesoftware.uptime.erdc.MonitorCustomScriptRunner</exec>
     <erdc_version>3.01</erdc_version>
     <category>Advanced and Script Monitors</category>
     <category_related>Applications - General</category_related>
     <grid_info>
          <supported_versions>7.3</supported_versions>
          <supported_platforms>windows, linux</supported_platforms>
          <upgrade_in_place>true</upgrade_in_place>
          <requires_agent_scripts>false</requires_agent_scripts>
     </grid_info>

Let's talk about each of these tags, and what they represent:

• <class>. This is the service monitor's name as seen on the Add Service Monitor screen in the UIM GUI.
• <exec>. This tells UIM that our plugin is a MonitorScriptRunner. There are also java-type plugins, but this doc only focuses on custom scripts.
• <erdc_version>. This is the version number of the plugin which is used to identify when new versions are available.
• <category>. This is the primary category on the Add Service Monitor screen where you'll be able to find this monitor.
• <category_related>. This is a secondary category where the monitor is displayed in the Related Monitors list.
• The <grid_info> section outlines details about which OS platforms this monitor can run on, as well as which versions of UIM. It is used by the Extension Manager to filter which plugins to show as available for installation.
  • <supported_versions>. This is the version of UIM monitoring station this plugin is compatible with.
  • <supported_platforms>. This is a list of which monitoring station platform a monitor can run on. This can be either Windows or Linux, or both like in the example above.
  • <upgrade_in_place>. This tells UIM that you can update the plugin without having to re-define the services monitors assigned to it. In most cases, you'll want to set this to true.

Next comes the elements section of the XML file. This is a list of element tags which define the input and output variables you see when editing a service monitor in the Uptime Infrastructure Monitor GUI. The contents of these variables are stored in the Uptime Infrastructure Monitor datastore just like regular service monitors.

XML - Elements

The first of these elements are the process_linux & process_windows which control which script actually gets run on the monitoring station when a service monitor is triggered.

<elements>
     <element name="process_linux" parameter_type="input" data_type="string" gui_basic="1" range_type="0" hidden="true">
            <control_options> size:40 </control_options>
            <default_value>plugins/scripts/monitor-custom-remote/monitor-custom-remote.sh</default_value>
            <short_description>Script Name</short_description>
            <long_description>Name of the script/binary/process to be executed by up.time </long_description>
            <validation_rule>
                  <alphanumeric/>
            </validation_rule>
            <error_message>
                  <alphanumeric>required field</alphanumeric>
            </error_message>
            <gui_type>
                  <string/>
            </gui_type>
      </element>
      <element name="process_windows" parameter_type="input" data_type="string" gui_basic="1" range_type="0" hidden="true">
            <control_options> size:40 </control_options>
            <default_value>plugins/scripts/monitor-custom-remote/monitor-custom-remote.bat</default_value>
            <short_description>Script Name</short_description>
            <long_description>Name of the script/binary/process to be executed by up.time </long_description>
            <validation_rule>
                  <alphanumeric/>
            </validation_rule>
            <error_message>
                  <alphanumeric>required field</alphanumeric>
            </error_message>
            <gui_type>
                  <string/>
            </gui_type>
      </element>

Full details about what the various attributes of the XML file's element tag control are available in the UIM Plugin Guide. But for now, we'll just cover the most important details from the example above:

• name. This is the name of variable itself. In the examples above, you can see this is either set to process_windows or process_linux. You need to have at least one of these defined for every plugin.
• parameter_type. This can be either input or output. Input parameters are set as an environment variable at run time. We'll talk more about output type parameters below.
• data_type. This can be either string or integer. This usually more important on the output variable side of things, as it also controls what WARN/CRIT threshold comparisons can made against it.
• hidden. This can either be true or false . This controls whether a parameter is displayed on the Edit Service Monitor Screen. A hidden variable cannot be edited by a user in the GUI, but can still be updated in the database directly.
• <default_value>. This provides the default value for a variable, which is especially important for the examples above, as both are set as hidden and are not editable from the GUI.

Now that we've looked at the process_windows or process_linux fields, let's take a look at some input variables for this monitor. If you are familiar with the Custom Remote Monitor in UIM, you've likely seen this combination of Agent Port, Agent Password, and Remote Script fields before, as these are commonly used to trigger custom agent scripts. A few notes:

• We can see the Agent Port parameter is set up as type integer, and has the default port 9998 already set for us.
• The password parameter is setup as a data_type of string, but also has the gui_type set to <password/> which as mentioned above, tells UIM to encrypt this value when it's stored in the database, as well as blank out the field itself when displayed in the GUI.
• The remote_script and args fields are both set up as basic string fields with nothing special about them. 

<element name='port' parameter_type='input' data_type='integer' 
gui_basic='1' range_type='0' units=''>
     <control_options>size:8</control_options>
      <default_value>9998</default_value>
      <short_description>Agent Port</short_description>
      <long_description>up.time Agent Port (default is 9998)</long_description>
      <validation_rule/>
      <error_message/>
      <gui_type>
            <integer/>
      </gui_type>
</element>
<element name='password' parameter_type='input' data_type='string' 
gui_basic='1' range_type='0' units=''>
      <control_options>size:8</control_options>
      <default_value></default_value>
      <short_description>up.time Agent Password</short_description>
      <long_description>Password setup on the agent system</long_description>
      <validation_rule/>
      <error_message/>
      <gui_type>
            <password/>
      </gui_type>
</element>
<element name='remote_script' parameter_type='input' data_type='string' 
gui_basic='1' range_type='0' units=''>
      <control_options>size:8</control_options>
      <default_value></default_value>
      <short_description>Remote Script / Command</short_description>
      <long_description>Script name (posix) or command (win) setup on the agent</long_description>
      <validation_rule/>
      <error_message/>
      <gui_type>
            <string/>
      </gui_type>
</element>
<element name='args' parameter_type='input' data_type='string' 
gui_basic='1' range_type='0' units=''>
      <control_options>size:8</control_options>
      <default_value></default_value>
      <short_description>Arguments</short_description>
      <long_description>Arguments that will be sent to the remote script</long_description>
      <validation_rule/>
      <error_message/>
      <gui_type>
            <string/>
      </gui_type>
</element>

Editing the Service Monitor view

Now that we've talked about the details that go into a plugin's XML file, let's take a look at how these input variables are displayed in the UIM GUI when creating an instance of the Custom Remote Monitor:

• The first thing you notice is that we don't see either the process_windows or process_linux fields, but that's to be expected here, as those fields where set as hidden.
• The contents of the password field are obscured, as expected. 
• The Remote Script and Arguments fields are basic string fields.
• You can also see the two output values for this monitor, but we'll talk about those more later. 

Environment variables

Next let's take a look at how all these parameters and input fields come together as the environment variables that are used by the script itself. Environment variables aren't normally displayed or retained by the service monitor when it runs, but they're easy to capture for testing and debugging purposes by adding the appropriate command to your shell/bat script:

• Linux: printenv > /var/tmp/envvars.txt
• Windows: set > C:\envvars.txt

There's normally plenty of other environment variables that are set, but we've filtered those out to only look at the uptime values that are coming from our service monitor's XML definition.

UPTIME_ARGS=test-args 
UPTIME_TIMEOUT=60 
UPTIME_SYSTEM_TYPE=NODE 
UPTIME_PORT=9998
UPTIME_PROCESS_WINDOWS=plugins/scripts/monitor-custom-remote/monitor-custom-remote.bat 
UPTIME_PROCESS_LINUX=plugins/scripts/monitor-custom-remote/monitor-custom-remote.sh 
UPTIME_REMOTE_SCRIPT=test-script
UPTIME_HOSTNAME=cat1.uptimesoftware.com 
UPTIME_PASSWORD=test-pass

Here's a few things you should notice about these values: 

• They all start with UPTIME_ and the contents of the name attribute from the element tag. 
• The UPTIME_HOSTNAME has the hostname of the element the monitor is running against.
• The remaining input variables show up as expected with the same contents as we set in the GUI example above. 

Shell/Bat wrapper scripts

Now let's take a quick look at what the custom_remote.bat and custom_remote.sh scripts actually do with these environment variables. As mentioned above, most of the UIM-provided plugins rely on a common PHP script to perform the actual task of monitoring. As such, the bat and sh scripts are just responsible for telling UIM's bundled php executable to run the required script, and then pass along the environment variables in the correct order. The specifics of how this done obviously varies between both Windows and Linux, so let's take a look at both methods.

For more information about this feature, see monitor-custom-remote.bat on GitHub.

@echo off
..\..\..\apache\php\php.exe rcs.php -h=%UPTIME_HOSTNAME% -p=%UPTIME_PORT% -s=%UPTIME_PASSWORD% -c=% UPTIME_REMOTE_SCRIPT% -a=%UPTIME_ARGS%

From the example here, you can see that windows allows you access environment variables by wrapping the name in % characters, i.e. %UPTIME_HOSTNAME%. So we've used that to arrange the input variables from xml into the order expected by the rcs.php script. You may also notice the ..\..\..\apache\php\php.exe path used to start php. Whenever a script monitor is triggered the current working directory is set to the location of the script itself, which is in this case is C:\Program Files\uptime software\uptime\plugins\scripts\monitor-custom-remote\monitor-custom-remote.bat. But the location of the UIM directory can vary depending on where it was initially installed. So we use a relative path from the plugin's location to find <uptime_dir>\apache\php\php.exe. When making your own plugins, you can usually simplify this by using the hard-coded path of your UIM directory.

For more information about this feature, see monitor-custom-remote.sh on GitHub.

#!/bin/sh 

inst=`grep ^inst /etc/init.d/uptime_httpd | cut -d= -f2`
MIBDIRS=$inst/mibs
export MIBDIRS

/usr/local/uptime/apache/bin/php rcs.php -h=$UPTIME_HOSTNAME -p=$UPTIME_PORT -s=$UPTIME_PASSWORD -c=$UPTIME_REMOTE_SCRIPT -a=$UPTIME_ARGS

In the case of the Linux shell script, we can access the contents of our environment variables by prefacing the name with the $ symbol. We then pass the same input variables along to php in the same order as in the bat script above. Just like on Windows, the current working directly is set to the location of plugin script itself, but in this case we can use the contents of the /etc/init.d/uptime_httpd script to find the location the UIM install directory and use that to set the location of an extra MIBDIRS environment. This is only required for PHP scripts on Linux, so that the php_snmp library doesn't complain about being unable to find it's mibdirs. If you are not using php in your own script, you can ignore this completely. The other thing worth noticing here is that we've hardcoded the path of UIM's bundled php executable, as the Apache folder is always installed into the default location of /usr/local/uptime/apache even if UIM was installed elsewhere. This prevents us from using the same relative path method as we did on Windows. 

We won't get into the details of the rcs.php script itself as this typically changes depending on the details of the plugin itself. But the full contents of this file on GitHub here if you are interested.

Output variables

We've looked at all the details that go into a plugin scripts input variables all the way from the xml file, the Edit Service Monitor Screen, and how they finally get exposed to the custom script as environment variables. Now let's take a look at how a custom monitor can return it's output back to UIM, and the different things we can do it with.

XML - Output variables

Just like input variables, output variables are also defined in the xml file via the same element tags. They're covered by the same basic rules outlined above, and full details for their XML definitions are available in the Plugin Guide.

Below are the two output variables that are defined in the Custom Remote Monitor's xml file. For reference the full version of this file is available on GitHub here.

            <element name='custom1' parameter_type='output' data_type='string' gui_basic='1' range_type='0' units=''>
                 <control_options>size:8</control_options>
                 <default_value></default_value>
                  <short_description>Output</short_description>
                  <long_description>Output from the script</long_description>
                  <validation_rule/>
                  <error_message/>
                  <gui_type>
                        <string/>
                  </gui_type>
            </element>
            <element name="timer" parameter_type="output" data_type="integer" gui_basic="0" range_type="0" units="ms">
                  <control_options>size:8</control_options>
                  <default_value/>
                  <short_description>Response time</short_description>
                  <long_description>Response time in ms</long_description>
                  <validation_rule/>
                  <error_message/>
                  <gui_type>
                        <integer/>
                  </gui_type>
            </element>
      </elements>
      <help>Execute remote scripts via the up.time agent. Output from the remote string is returned as String Data.</help>
</uptime>

Here's some things you notice about these examples:

• In both cases, the parameter_type is set as output . This tells UIM that this is output we expect to see back form the script.
• There's an optional unit attribute in the element tag, which is used to show that the response time is being stored in milliseconds. This unit is also displayed when graphing these metrics within UIM.
• We have an example of both the integer and string data_types, which in the case of output variables controls what type of threshold comparisons can be used to set the monitors WARN/CRIT status. Integer type values allow for greater/less than numerical comparisons. Where-as strings have contains/does not contains and regex-style comparisons.

Now that we've see what the output variables look like in the xml itself, lets take a look at what the script needs to return so that UIM knows which output variable is which. This is based on the element tag's name attribute, which is used to look for a line of output from the script that starts with that name, which in the case above is 'custom1'. The rest of that line is then pulled into UIM as the variable and can used for threshold comparisons. In our example here, this custom1 variable is treated as string. In monitors with multiple output variables, the output from the script itself can come any order, but any lines that don't start with the name of a variable are ignored.

The 'timer' output variable is a special case that always contains the time it takes the service monitor to run in milliseconds. This doesn't need to be provided as part of the output from your custom script.

The best way to understand how this works it to look at the output on the Test Service Monitor screen, which shows you the raw output from the service monitor, and then a list of any output variables that UIM was able to parse out based on their names. For an example, we'll use the custom remote monitor with an agent side script called check-uptime which returns output similar to:

uptime_core is stopped 
 MON Role

When this is triggered via the Test Service Monitor button, you would see something like this:

The first thing you'll likely notice here is that the monitor has set its Status to OK, which is the default state for a service monitor that's run successfully, as long as none of its threshold comparisons are triggered. The Message line shows that the process returned a valid status, which is then followed by the output from the plugin monitor, i.e. custom1 uptime_core is stopped MON Role.

You may wonder where the 'custom1' string is coming from as it's not in the example output of the agent side script above. This is inserted by the rcs.php and used to mark that the output from the agent script should link up with the 'custom1' variable from the XML. Which you can see on the next line Output where 'custom1' is replaced with that variable's short description also from the xml. The next line is an example of UIM attempting parse the second line of output from our agent script into a variable called 'MON' but because this isn't defined in our xml file, it won't be available for comparisons within the GUI. The final line here is the Response time value, which is the special timer mentioned above.

The general takeaway from this test service monitor example is that when UIM is able to parse and understand the output from a script, the matched up variables are listed line by line after the message. With the name of the variable replaced by its short description. If you don't see your output variable listed like this, then double check the names of your output variables and how the script itself is returning its output.

Installing plugins

Now that we've seen how plugin monitors are made of a combination an XML file that defines the various inputs/outputs and a script performs the actual task of monitoring, let's take a look how these pieces are packaged together, and how UIM knows which plugins are available for installation.

Packaging plugins

The best way to understand how UIM plugins are packaged is to take a look at the zip file for a plugin itself. We'll use the same Custom Remote Monitor from above as our example again. The actual zip file that is used when installing the plugin is located within this repo's dist folder, along with some .upk files (The legacy plugin format used by older versions of Uptime Infrastructure Monitor).

You can easily download the monitor-custom-remote.zip yourself, and look at the contents directly, but the contents of that archive are just a zipped-up version of the repo's pkg directory, which you can easily view on GitHub here. 

The contents of the zip/pkg folder are made up of the xml file that we've already discussed in detail above, and a mix of one or more folders. These files folders contain the actual scripts that will be extracted into place on the UIM monitoring station. The reason for the files-win and files-posix folders are that plugins may require different files or scripts depending which OS (ie. Windows or Linux) the UIM monitoring station is running. As such, the contents of the files-win folder is extracted only on Windows monitoring station, and the files-posix folder is extracted only on a Linux monitoring station.

The mechanism for how these three folders are extracted during the plugin's installation are the same general process whether the monitoring station is running Windows or Linux. Which is that each of the files folders is extracted into the directory where UIM is installed on the monitoring station. Any sub-folders within the plugin's files folders is created along the way if they don't already exist. Though in most cases, you'll want to place all your plugin's scripts/files within its own sub-folder in uptime_dir/plugins/scripts/.

So in the case of the Custom Remote Monitor's 'files' folder, you'll notice it contains some sub folders (ie. plugins/scripts/monitor-custom-remote ) which eventually contain a single rcs.php file.

When installed on a Linux monitoring station, those files end up in a location similar to below. In this example, UIM was installed in the default /user/local/uptime_location. The 'monitor-custom-remote.sh' file comes from a similar sub-folder within the 'files-posix' folder.

Similarly, on a Windows monitoring station that file would end up in the location similar to below. In this example, UIM was installed into the default C:\Program Files\Uptime Softwarup.time\[location for Windows]. The 'monitor-custom-remote.bat' file comes from a similar sub- folder within the plugin's 'files-win' folder.

Extension.json

Now that we've seen how a plugin monitor is packaged into it's own zip file, the last piece we need to look at the extension.json file that the monitoring station uses to know which plugins are available for install and where to get the associated zip files. By default, UIM attempts to get these details from our website at this location: http://the-grid.uptimesoftware.com/extension.json.

In order to install your own plugins for testing, you'll need to create your own copy of this file that includes details about where to locate your own plugin files. The general process is outlined in How to install plugins without direct internet access. Though that article focuses on how to internally mirror the currently available plugins for environments where the UIM server doesn't have direct access to the external internet. In our situation, we instead need to use the same general process to create our own extension.json file that lists details about our own plugins.

Each plugin listed needs a entry in the json file similar to the below example of the 'Custom Remote Monitor':

{
     "category": "Advanced and Script Monitors", 
     "supported_versions": [
          "7.3"
     ],
     "name": "Custom Remote Monitor",
     "git_file_target": "https://github.com/uptimesoftware/monitor-custom-remote/raw/master/dist/monitor-custom-remote.zip",
     "icon_url": "http://the-grid.uptimesoftware.com/img/posts/placeholder-logo.png,
     "upgrade_in_place": true,
     "requires_additional_setup": false, 
     "requires_agent_scripts": false, 
     "version": "3.01", 
     "supported_platforms": [
          "windows", 
          "linux"
     ],
     "grid_url_target": ["http://the-grid.uptimesoftware.com/plugin/monitor-custom-remote.html",
     "type": "plugin",
     "description": "Execute remote scripts via the up.time agent. Output from the remote script is returned as String Data."
}

Most of these fields are pretty self explanatory and match up with similar fields from the monitor's XML file. The most important of these fields is git_file_target which tells UIM's extension manager where to find the zip file it needs to download to install the plugin. Most of the plugins provided by UIM have this field pointing to a location on GitHub.com, though that isn't required, and you can place these files anywhere that is accessible from your monitoring station over http.

The next most important fields for a plugin are the 'name' and 'version' as these are used to figure out which plugins are currently installed within UIM and if they're running the current version. A combination of these two fields controls whether a user sees the Installed/Install/Upgrade buttons within the extension manager. The contents of the name field in the JSON needs to exactly match the XML's class tag.

Contributing a plugin

By now you've hopefully been able to create your own plugin monitor for use with Uptime Infrastructure Monitor and are asking yourself how you can share your creation with other UIM users.

Choosing a license

One thing you may not of thought about much more when creating software/scripts for internal project is, "What copyright license should I release this under?"

If you are not aware, there's a variety of different copyright licensing models available for open source software that outlines what changes someone can make to your code and how can they share it among others. We recommend using one of the Creative Commons Licenses as they provide easy to understand descriptions of what the license covers. Details about these licenses are available at: https://creativecommons.org/licenses/.

Of the licenses listed on the Creative Commons website, we recommend using the Attribution-ShareAlike with plugins as it provides the best fit for how plugins are typically created and expanded upon. Once you've chosen the appropriate license for your plugin, you'll need to mention it somewhere within the plugin itself, typically either as part of the readme itself or by adding a license.txt file to your plugin repo. See this page on GitHub for some common ways people handle this.

Putting your plugin on GitHub

If you haven't been using git or some other version control tool while creating your plugin then it's time to create a repository for it on GitHub, as that's the easiest way to share your creation with us and make it available for download via the Extension Manager. Here's some documentation on GitHub.com that covers the basics of howto create your own repository and a general introduction for working with git.

Once your plugin is available on GitHub it's time to contact uptime-support@idera.com and let us know about your creation. From there we'll take a look at what your plugin does, and how it works, and do some of the remaining tasks involved in making it available for download via Uptime Infrastructure Monitor's extension manager.