The Uptime Infrastructure Monitor Linux agent collects the following performance metrics from the systems on which it is installed:
Each set of performance metrics is averaged over an interval of one second.
CPU
The Uptime Infrastructure Monitor agent uses the sar -urWqR 1 command to compare the system counters during a one-second interval. The statistics returned by the agent are averaged for all CPUs on the system.
| Metric | Description | Source |
|---|---|---|
% USR | Percentage of time that the processor spends in user mode (a processing mode for applications and subsystems). | /proc/cpuinfo |
% SYS | Percentage of time that the kernel spends processing system calls. | /proc/cpuinfo |
% WIO | Amount of waiting time that a runable process for a device takes to perform an I/O operation. | /proc/cpuinfo |
% Total | Total amount of User %, System %, and Wait I/O % | /proc/cpuinfo |
Run Queue Length | Percentage of time that one or more services or processes are waiting to be served by the CPU. | /proc/cpuinfo |
Multi-CPU
The Uptime Infrastructure Monitor agent uses the sar and mpstat utilities on a Linux system to collect the metrics in the table below from Linux systems with multiple CPUs. The agent averages the statistics from each CPU using the sar -x SELF -I SUM -P ALL -wu 1 command, which compares the system counters during a one-second interval. The statistics that the agent returns are for the entire system, per CPU.
| Metric | Description |
|---|---|
User % | Percentage of CPU user processes that are in use. |
System % | Percentage of CPU kernel processes that are in use. |
Wait I/O % | Percentage of time that a process which can be run must wait for a device to perform an I/O operation. |
SMTX | Number of read or write locks that a thread was not able to acquire on the first attempt, as reported by the |
XCAL | Number of interprocess cross-calls. In a multi-processor environment, one processor sends cross-calls to another processor to get that processor to do work. Cross-calls can also be used to ensure consistency in virtual memory. Heavy file system activity (such as NFS) can result in a high number of cross-calls. |
Interrupts | Number of CPU interrupts. |
Total % | Total amount of User %, System %, and Wait I/O%. |
Memory
The UIM agent uses the free command to collect the Free Memory metric from a Linux system. The rest of the memory related metrics are gathered by the sar -urWqR 1 command which compares the system counters during a one-second interval. The statistics that the agent returns are for the entire system.
| Metric | Description | Source |
|---|---|---|
Free Memory | Amount of physical memory available to the operating system, system library files, and applications. | /proc/meminfo |
Cache Hit Rate | How often the system accesses the CPU cache. | /proc/meminfo |
PageOut per Second | Rate at which pages were written to disk. | /proc/meminfo |
PageIn per Second | Rate at which pages were read from or written to the disk. | /proc/meminfo |
PageFree per Second | Number of pages that are freed from memory each second. | /proc/meminfo |
PageScan per Second | Average number of pages that are scanned each second. | /proc/meminfo |
Free Swap | Amount of available free swap space, as a percentage of total available free swap space. | /proc/meminfo |
Disk
The UIM agent gathers file system statistics for each file system using the df -lk command. Disk statistics (e.g. %busy, reads per second, and writes per second) are output per disk and compared between polling intervals using the iostat -d -x 1 2 command.
| Metric | Description | Source |
|---|---|---|
Disk (Spindle) Name | Names of each disk on the system. | /proc/diskstats |
Usage (% Busy) | Percentage of time during which the disk drive is handling read or write requests. | /proc/diskstats |
Throughput (Blk/s) | Number of read and write operations on the disk that occur each second. | /proc/diskstats |
Read/Writes/s | Average number of bytes that have been transferred to or from the disk during write or read operations. | /proc/diskstats |
Average Queue Length | Number of threads that are waiting for processor time. | /proc/diskstats |
Average Service Time | Average amount of time, in milliseconds, that is required for a request to be carried out. | /proc/diskstats |
Average Wait Time | Average time, in milliseconds, that a transaction is waiting in a queue. The wait time is directly proportional to the length of the queue. | /proc/diskstats |
Network
The UIM agent uses the netstat -s command to retrieve a combined total of TCP Retransmits for all network interfaces. Other network statistics (e.g. kbps, errors and collisions) are averaged, per interface, using the sar -n DEV -n EDEV 1 command, which compares the system counters during a one-second interval.
| Metric | Description | Source |
|---|---|---|
In Kbps | Rate, in kilobytes per seconds, at which data is received over a specific network adapter. | /proc/net |
Out Kbps | Rate, in kilobytes per seconds, at which data is sent over a specific network adapter. | /proc/net |
In Errors | Number of inbound packets that contained errors, which preventing those packets from being delivered to a higher- layer protocol. | /proc/net |
Out Errors | Number of outbound packets that could not be transmitted because of errors. | /proc/net |
Collisions | Number of signals from two separate nodes on the network that have collided. | /proc/net |
TCP Retransmits | Number of packets that have been re-sent over a network interface. The agent returns a combined total for all interfaces. | /proc/net |
Process
The UIM agent uses the ps -eo command to collect the process information listed in the table below from a Linux system. By default, the agent gathers the top 20 processes and sorts them by the highest CPU usage.
| Metric | Description | Source |
|---|---|---|
PID | Unique identifier of a specific process. | /proc/stat |
PPID | Identifier of the process that the process that is currently running. | /proc/stat |
UID | Value that identifies the current user. | /proc/stat |
GID | Value that identifies a group of users. | /proc/stat |
Memory Consumed | Amount of memory that is being used by a process. | /proc/stat |
RSS | Amount of physical memory that is being used by a process. | /proc/stat |
CPU % Utilization by Process | Percentage of CPU time that is being used by individual processes. | /proc/stat |
Memory % Utilization by Process | Amount of physical memory that is being used by individual processes. | /proc/stat |
Process Start Time | Time at which the process started. | /proc/stat |
Process Run Time | Time at which the process started. | /proc/stat |
Number of Processes Running | Total number of processes that are currently running on the system. | /proc/stat |
Number of Blocked Processes | Total number of processes that are blocking resources. | /proc/stat |
Number of Waiting Processes | Total number of processes that are waiting to be executed by the CPU. | /proc/stat |
Execs per Second | Total number of system calls that are executed each second. | /proc/stat |
Process Creation Rate | Total number of processes that are being spawned over a specified time period. | /proc/stat |
Workload
The UIM agent uses the ps utility to collect workload information from a Linux system. Workload statistics (based on the same 20 processes that were gathered from the Process method) are sorted within UIM's core. The workload processes that the agent gathers include the user/group/process name and their individual statistics, which can be sorted based on the user's desired graph presentation (e.g. user, group or process name).
| Metric | Description | Source |
|---|---|---|
Workload by Process | Demand that network and local services are putting on a system, based on the processes that are running. | /proc/load |
Workload by User | Demand that network and local services are putting on the system, based on the IDs of the users who are logged into a system. | /proc/load |
Workload by Group | Demand that network and local services are putting on the system, based on the IDs of the user groups that are logged into a system. | /proc/load |
Workload Top 10 by Process | The 10 processes that are consuming the most CPU resources. | /proc/load |
Workload Top 10 by User | The 10 processes the are consuming the most CPU resources, based on user ID. | /proc/load |
Workload Top 10 by Group | The 10 processes the are consuming the most CPU resources, based on group ID. | /proc/load |
User
The UIM agent uses the following commands to collect user statistics from a system:
ps -eolast | head 10(login history for the last 10 users on the system)who(lists who is currently logged into the system)
| Metric | Description |
|---|---|
Login History | Number of times or frequency at which a user has logged into a system during any 30 minute time interval. |
Sessions | Number of sessions or number of distinct users who are logged into a system during any 30 minute time interval. |