1. Engineers that are on call should not be allocated in product sprints. The on-call engineer is protecting the rest of the engineering team from distractions and is providing service to the rest of the company. They can't do this well if they're juggling feature development work.
  2. Engineers are expected to be available to respond to and triage technical issues around the company, please use our Incident Severity Levels docs to understand the expectations for response time and escalating to out people for outages.
    1. The On-Call engineer runs the incident, investigates what's happening, determines the severity level, and is empowered to mobilize other parts of the team based on their discretion. It's totally valid to loop @Adam Haney into these decisions if you're unsure.
    2. If it’s a P3 or more severe, the On-Call communicates (or delegates communication) about the status of the incident to the rest of the company. This includes sending out a company-wide(partners) email and tagging the relevant stakeholders on Slack (leads, operators, PMs).
    3. After the incident, the On-Call engineer digs in and researches what went wrong, the steps that lead us there, and investigates our systems. They then write a Post Mortem doc that goes in the database in notion. For P0 incidents they should also email this document to the rest of the company.
  3. The On-Call engineer will be asked to troubleshoot security issues as they arise. @Drew Sutherland is the RP for security and issues will be discussed in #tool-vanta where the @egt-on-call will be tagged.
  4. On-Call should triage errors in Sentry.io to determine if those errors constitute incidents. Using their best judgment they are encouraged to ship fixes for errors in sentry during their on-call week.
  5. When the on-call is not responding to incidents they should use their time to make our systems better.
    1. Check the SYS JIRA project (https://invisible.atlassian.net/jira/software/projects/SYS/boards/50/backlog) if you need ideas for projects to work on
    2. Or fix bugs you see coming through Sentry
  6. On-call should follow #egt-alerts in Slack closely and respond to those alerts
    1. At the start of your shift make sure to take over in the @egt-oncall group in Slack. Follow this guide to set yourself as the @egt-oncall: How To: Set Yourself as egt-oncall in Slack
  7. At the end of your on-call shift, you should write up what happened while you were on call, the status of any ongoing incidents, and a description of the better engineering tasks you were able to complete. You can also refer to this template On Call Report Template to write the on-call report (please send this both as an email and add it to the database. To make it easier to write the on call report at the end of your shift it’s suggested that you create the report at the beginning of your shift to take notes in, then clean up / edit the report before the hand-off meeting.

Tools We Use

  1. Ops Genie is our on-call scheduling and alerting tool you can find it at: https://invisible.app.opsgenie.com/
    1. To view the current on-call schedule look here: https://invisible.app.opsgenie.com/settings/schedule/detail/ecab26ad-b22c-42a7-846c-712078d8281b

Setting Up On Call

General Guidance

  1. What should I do if I discover a bug (NOT an incident) should I fix it or create a bug ticket?
    1. This is very case dependent. Help when you can, prioritize incidents.