Fylamynt + PagerDuty: Automated Incident Response

If you use PagerDuty for incident management, this article is for you!

Prasen Shelar
March 8, 2021

Do you see a new alert and dread not having an easy way to remediate it? 

Honestly we hated it too, until we created Fylamynt!

PagerDuty does an excellent job in aggregating data from myriad monitoring tools and creating actionable insights in the form of incidents. Using PagerDuty with Fylamynt for incident response in a completely automated fashion is the match we all need today. To access this, users can tap into the real-time PagerDuty intel with the workflow automation features of Fylamynt to improve full-stack visibility and accelerated incident response.

Let’s explore this with a specific example of how Fylamynt’s PagerDuty integration works.

Interactive, Real-Time Incident Enrichment and Response

Challenge

Incident response activities often include repetitive tasks based on fragmented set of tools that lack the support of context transfer. Irrelevant alerts causing fatigue and disparate tools for different tasks all add up in mean-time-to-respond.

Solution with an example workflow

Understanding the workflow in Fylamynt with visualization and elaboration about each step.

Alert Scenario: 

PagerDuty collects data from CloudWatch and generates an incident called “RDS DB Instance is out of storage”

Alert Remediation:

Fylamynt ingests the incident from PagerDuty in real-time and triggers a workflow to allocate the right storage space on RDS.

Solution Overview:

1. Using the PagerDuty connector in Fylamynt

  • To start receiving incidents from PagerDuty in Fylamynt, set up your PagerDuty instance in Fylamynt and put Fylamynt on the listen mode with a pre-configured webhook.
  • The incident next in Fylamynt will then be ingested as a trigger to the desired workflow.


2. Creating a remediation workflow with the visual workflow editor

  • Use the visual workflow editor in Fylamynt and drag the PagerDuty Alert node on the canvas to select the right service as a trigger. (In this case, CloudWatch-RDS-monitor)

Getting incidents from selected PagerDuty Service

  • Once the trigger has been established, you can access the RDS integration action to modify the   database instance with the right storage capacity. You could also notify the other incident responders or those interested in the incident’s progress using any messaging app like Slack, Teams or Email.


Configuring AWS Action Node to modify storage of target AWS RDS instance


3. See it in action

  • Once the PagerDuty instance has been set up and the workflow created, now is the time to see it in action. As soon as an incident gets generated in PagerDuty matching custom event CloudWatch-RDS-monitor, the attached workflow PagerDuty-AWS-RDS-Storage-Remediation will be triggered automatically.

Workflow execution triggered on PagerDuty Incident

PagerDuty Incident successfully resolved after Fylamynt workflow execution

Notifying the right team on slack with Fylamynt workflow

Towards a more streamlined incident response

  • Harness real time incident ingestion from PagerDuty in Fylamynt for automated, workflow-driven response.
  • Further enrich data from PagerDuty with other AWS integrations or similar relevant integrations via Fylamynt’s orchestration.
  • Improve CloudOps efficiency by streamlining your incident response and shorten decision making with collaboration and automating key manual tasks that are repetitive.

Fylamynt workflows combined with PagerDuty incidents can standardize and speed up resolution of various cloud alerts. CloudOps gets a comprehensive view of the response workflow on a single screen. With repetitive tasks automated, the incident response team can now focus on deeper strategic actions.


To find out more about this integration, please refer to the following video.