# PagerDuty Agent Integration Guide Install and configure the PagerDuty Agent, a command-line daemon that forwards monitoring events from on-premise Linux hosts. The PagerDuty Agent is a helper program for your infrastructure that integrates your monitoring tools with PagerDuty. PagerDuty Agent also includes command line tools to accept events from your monitoring system and a daemon process that handles sending events to PagerDuty. The daemon process manages buffering, retrying, and event throttling. Read more about supported event in the developer documentation [Events API v2 Overview](/developer/events-api-v2-overview). > **Looking for PagerDuty AI Agents?:** PagerDuty Agent is an application that integrates with traditional monitoring software. If you’re looking for our AI-driven agents, please read [PagerDuty Advance](/ai-automation/advance#ai-agents). ## System Requirements The PagerDuty Agent is currently available in `deb` and `rpm` package formats and supported on the following operating systems: - Ubuntu 12.04, 14.04, and 16.04 - CentOS 6 The Agent does not run on CentOS 5 or lower, as it requires a newer version of Python than the version included with CentOS 5. Please use the [Perl-based integration for Nagios](https://www.pagerduty.com/docs/guides/nagios-perl-integration-guide/) or older [Python-based integration for Zabbix](https://pagerduty.com/docs/guides/zabbix-integration-guide-agentless/) on older CentOS systems. The PagerDuty Agent requires Python 2.7 or higher, but Python 3 is recommended. Regardless of version, it is necessary that the Python binary be accessible via `python`; to verify this, run the following command: ```shell which python ``` If no Python is found, run the following command to create a symlink: ```shell sudo ln -fs $(which python3) /usr/bin/python ``` ## Integration Walkthrough ### Create a Generic API Service in PagerDuty PagerDuty Agent can send events to any technical service via a generic Events API integration. Please read [Create a Generic Events API Integration](/incident-management/services/services-and-integrations#create-a-generic-events-api-integration) for more information about creating a service with a generic Events API integration. > **Integration Key Requirements:** Not all integration types and integration keys are compatible with the Events API. When selecting an integration type, you must select **Events API V2** to guarantee compatibility with our API for testing your integration or when using the `pd-send` command. ### Agent Source, Installation and Updates PagerDuty Agent and its integrations are open source, and can be found on [GitHub](https://github.com/PagerDuty/pdagent). We recommend installing PagerDuty Agent from our `apt` or `yum` repositories as described below. After installing PagerDuty Agent, please remember to update agent packages regularly to ensure that you have the latest fixes. Depending on your Linux distribution, you can keep your agent packages up to date with the following steps: - [Ubuntu](#ubuntu) - [CentOS](#centos) #### Ubuntu 1. If you do not already have the `apt-transport-https` package, please install it first: ```shell sudo apt-get install apt-transport-https ``` 2. Add the PagerDuty repository to your system: ```shell wget -O - https://packages.pagerduty.com/GPG-KEY-pagerduty | sudo apt-key add - sudo sh -c 'echo "deb https://packages.pagerduty.com/pdagent deb/" >/etc/apt/sources.list.d/pdagent.list' ``` > **Deprecation Guidance:** Please note that `apt-key` may be deprecated on some newer versions of Ubuntu and Debian. For more information and guidance, please reference the [manual page](https://manpages.debian.org/testing/apt/apt-key.8.en.html#DEPRECATION) from Debian. 3. Install (or update) the PagerDuty Agent packages: ```text sudo apt-get update sudo apt-get install pdagent pdagent-integrations ``` #### CentOS **Note**: The PagerDuty Agent is not compatible with CentOS 5 or lower; installation will not succeed on CentOS 5. 1. Add the PagerDuty repository to your system: ```shell sudo sh -c 'cat >/etc/yum.repos.d/pdagent.repo < **Stopped Service Limitations:** If the `pdagentd` service is stopped, events will not be sent to PagerDuty until the service is started again. Use the `-h` or `--help` option will return usage information: ```shell $ pd-send --help usage: pd-send [-h] -k SERVICE_KEY -t {trigger,acknowledge,resolve} [-d DESCRIPTION] [-i INCIDENT_KEY] [-f FIELDS]` `Queue up a trigger, acknowledge or resolve event to PagerDuty. ... ``` In the example above, `SERVICE_KEY` refers to the **Integration Key** that is shown on the service's **Integrations** tab in the PagerDuty web app. Here's an example command that would trigger an incident in PagerDuty if your service's integration key was `65d9cd0e14c04dae8ef86867277d138c`: ```shell $ export PD_SERVICE_KEY=65d9cd0e14c04dae8ef86867277d138c $ pd-send -k $PD_SERVICE_KEY -t trigger -d "Server is on fire" -i server.fire Event processed. Incident Key: server.fire ``` Here is an example command that triggers an incident in PagerDuty with multiple fields (key/value pairs). Note that `-f` is used before each pair: ```shell $ export PD_SERVICE_KEY=65d9cd0e14c04dae8ef86867277d138c $ pd-send -k $PD_SERVICE_KEY -t trigger -d "Server is on fire" -i server.fire -f MSG1=TEST1 -f MSG2=TEST2 ``` > **Integration Key:** If copying this example, remember to substitute your service’s integration key for `PD_SERVICE_KEY`. > **User Permissions:** The user that runs `pd-send` and other `pd-*` integration scripts should preferably have a umask that allows world-readability of files that the user creates (i.e., `022`). If the umask is more restrictive, the `pd-*` script changes the umask internally (only for itself) so enqueue-event files are created with the correct permissions. ## Manage the Queue Use the command `pd-queue` to check the queue of outbound alerts. This command gives an overview of the queue's status, and also allows you to retry events. This command needs to be run as root (i.e., `sudo pd-queue status`), since it requires access to directories that the `pdagent` user owns. The following commands are available to help manage the queue: - [pd-queue -h](#pd-queue--h) - [pd-queue status](#pd-queue-status) - [pd-queue retry](#pd-queue-retry) ### pd-queue -h ```shell $ pd-queue -h usage: pd-queue [-h] {status,retry} ... Access local queue of PagerDuty Agent. optional arguments: -h, --help show this help message and exit sub-commands: {status,retry} retry set up 'dead' PagerDuty events for retry. status print out status of local event queue. ``` ### pd-queue status Displays the pending, successful and in-error events in the Agent queue currently, as a table keyed by service key. ```shell $ pd-queue status -h usage: pd-queue status [-h] [-k SERVICE_KEY] Print out status of local event queue. optional arguments: -h, --help show this help message and exit -k SERVICE_KEY, --service-key SERVICE_KEY print status of events in given Service API Key $ pd-queue status Service Key Pending Success In Error ================================================================= key1 1 0 0 key2 1 0 1 $ pd-queue status -k key2 Service Key Pending Success In Error ================================================================= key2 1 0 1 ``` ### pd-queue retry This command moves all events for a specified integration key that are in an error state into a pending state. After addressing any configuration issues (e.g., shortening an event that was too long), retried events should not result in an error. If a retried event results in an error again, however, the agent will move it back to an error state. ```shell $ pd-queue retry -h usage: pd-queue retry [-h] [-k SERVICE_KEY] [-a] Set up 'dead' pagerduty events for retry. optional arguments: -h, --help show this help message and exit -k SERVICE_KEY, --service-key SERVICE_KEY retry events in given Service API Key (not to be used with -a) -a, --all-keys retry events in all Service API Keys (not to be used with -k) $ pd-queue retry -k key2 1 events set up for retry. $ pd-queue retry -a 0 event(s) set up for retry. ``` ## Integrate with Monitoring Tools PagerDuty Agent includes scripts to help you integrate with common monitoring tools. Please see the appropriate guide for more details: - [Nagios Integration Guide](https://pagerduty.com/docs/guides/nagios-integration-guide/) - [Zabbix Integration Guide](https://www.zabbix.com/integrations/pagerduty) ## Using a Proxy If your environment requires a proxy for outbound HTTP/HTTPS connections, you can specify proxy configuration by editing `/lib/systemd/system/pdagent.service` on [systemd-based systems](#systemd-based-systems) or `/etc/init.d/pdagent` on [SysV init-based systems](#sysv-init-based-systems) as the root user. > **Restart the Agent:** After adjusting the proxy configuration file, please restart the PagerDuty Agent with the command `systemctl restart pdagent`. ### systemd-based Systems ```shell sudo nano /lib/systemd/system/pdagent.service ``` Add two lines after `ExecStartPre=/bin/chown -R pdagent:pdagent /var/run/pdagent` (typically near line 8) to replace the URL with your proxy information, as shown in the example below: ```shell ExecStartPre=/bin/chown -R pdagent:pdagent /var/run/pdagent Environment="http_proxy=http://proxy:3128/" Environment="https_proxy=http://proxy:3128/" ``` ### SysV init-based Systems ```shell sudo nano /etc/init.d/pdagent ``` Add two lines after `PID_FILE=$PID_DIR/pdagentd.pid` (typically near line 48) to replace the URL with your proxy information, as shown in the example below: ```shell PID_FILE=$PID_DIR/pdagentd.pid export http_proxy=http://proxy:3128/ export https_proxy=http://proxy:3128/ ```