Site Reliability Engineer (SRE) EMS Production & Observability

Site Reliability Engineer (SRE)

EMS Production & Observability

Job Description

Role Overview

We are looking for an experienced Site Reliability Engineer (SRE) to support the reliability, availability, performance, and operational readiness of EMS production services.

The engineer will be responsible for production support and incident resolution, enhancing ELF, building actionable operational dashboards, improving monitoring and alerting, automating repetitive operational activities, and proactively preparing the platform for high-volume/high-visibility events such as the US Open 2026.

This role requires someone who can move beyond reactive production support and use observability, automation, capacity planning, trend analysis, and engineering improvements to identify and address reliability risks before they become customer-impacting incidents.

Key Responsibilities

Production Reliability & Incident Management

  • Own and support EMS production incidents, including investigation, triage, mitigation, recovery, and root-cause analysis.

  • Participate in production support/on-call rotations and provide timely response to critical incidents.

  • Troubleshoot application, infrastructure, API, integration, latency, capacity, and availability issues across the EMS ecosystem.

  • Drive incidents through resolution by collaborating with application engineering, infrastructure, platform, networking, database, and dependent service teams.

  • Conduct Root Cause Analysis (RCA) for significant incidents and ensure corrective/preventive actions are implemented.

  • Identify recurring production problems and convert operational fixes into permanent engineering solutions.

Observability, Monitoring & Dashboards

  • Design and maintain real-time operational dashboards providing visibility into EMS health and customer-impacting conditions.

  • Define and monitor key SLIs/SLOs around availability, latency, throughput, errors, and service dependencies.

  • Build meaningful alerting based on customer/service impact rather than relying solely on infrastructure thresholds.

  • Correlate application logs, metrics, events, and infrastructure telemetry to accelerate issue detection and troubleshooting.

  • Develop dashboards for transaction/request volume, success and failure rates, response time and latency, API/service availability, infrastructure and application health, dependency health, error trends and top failure reasons, capacity/utilization, and event-specific traffic and performance.

  • Continuously tune alerts to reduce noise and improve signal-to-noise ratio and Mean Time to Detect (MTTD).

ELF Enhancement & Automation

  • Enhance ELF capabilities to improve production monitoring, operational efficiency, troubleshooting, and reliability.

  • Identify manual production-support activities and automate them using scripting, APIs, CI/CD, or appropriate platform capabilities.

  • Build automated health checks, diagnostics, remediation, and operational reporting.

  • Develop reusable operational tools and runbooks that enable faster diagnosis and recovery.

  • Work with engineering teams to incorporate reliability and observability requirements into new releases.

Major Event Readiness

  • Lead proactive reliability planning for major business events such as US Open 2026 and other anticipated high-volume periods.

  • Analyze historical and projected traffic patterns to identify potential capacity or performance bottlenecks.

  • Establish event-specific dashboards, alerts, and operational thresholds.

  • Perform capacity planning, load/performance validation, and dependency readiness assessments.

  • Identify critical failure scenarios and develop mitigation and recovery plans before the event.

  • Conduct pre-event readiness reviews, failure simulations/game days, and operational rehearsals.

  • Establish clear event-day monitoring, escalation paths, communication channels, and incident-response procedures.

  • Produce post-event analysis covering performance, incidents, trends, and opportunities for improvement.

Reliability Engineering

The SRE will help establish and continuously improve engineering practices including SLIs, SLOs and error budgets; availability and latency measurement; capacity and performance management; automated remediation; incident management and postmortems; operational readiness reviews; dependency monitoring; resilience testing; and disaster/recovery preparedness.

A key expectation is to use production data to answer questions such as: What can fail? How will we detect it? How quickly can we isolate it? What is the customer impact? Can we automatically recover? How do we prevent it from recurring?

Required Qualifications

  • Strong experience in Site Reliability Engineering, Production Engineering, DevOps, or Production Support for enterprise-scale applications.

  • Strong hands-on production troubleshooting and incident-management experience.

  • Experience building production monitoring, alerting, and observability dashboards.

  • Strong understanding of application and infrastructure metrics, logs, distributed services, and API monitoring.

  • Experience with cloud infrastructure and containerized environments.

  • Knowledge of Kubernetes, CI/CD, Infrastructure as Code, and automation.

  • Scripting/programming experience with Python, Bash, PowerShell, or equivalent.

  • Understanding of networking concepts including DNS, load balancing, firewalls, routing, and connectivity troubleshooting.

  • Experience performing RCA and driving preventive engineering actions.

  • Ability to analyze traffic, latency, errors, and capacity trends and translate findings into engineering improvements.

  • Strong communication skills and ability to coordinate effectively during high-severity production incidents.

Preferred Qualifications

Experience with Azure, AKS/Kubernetes, Terraform, GitHub Actions, Grafana/Prometheus, Splunk or equivalent observability platforms, Application Performance Monitoring (APM), synthetic monitoring, automated remediation, chaos/game-day testing, and high-volume event readiness would be valuable.

Experience supporting highly available, customer-facing platforms where reliability during major events and peak traffic periods is critical is strongly preferred.

Back to blog

Common Interview Questions And Answers

1. HOW DO YOU PLAN YOUR DAY?

This is what this question poses: When do you focus and start working seriously? What are the hours you work optimally? Are you a night owl? A morning bird? Remote teams can be made up of people working on different shifts and around the world, so you won't necessarily be stuck in the 9-5 schedule if it's not for you...

2. HOW DO YOU USE THE DIFFERENT COMMUNICATION TOOLS IN DIFFERENT SITUATIONS?

When you're working on a remote team, there's no way to chat in the hallway between meetings or catch up on the latest project during an office carpool. Therefore, virtual communication will be absolutely essential to get your work done...

3. WHAT IS "WORKING REMOTE" REALLY FOR YOU?

Many people want to work remotely because of the flexibility it allows. You can work anywhere and at any time of the day...

4. WHAT DO YOU NEED IN YOUR PHYSICAL WORKSPACE TO SUCCEED IN YOUR WORK?

With this question, companies are looking to see what equipment they may need to provide you with and to verify how aware you are of what remote working could mean for you physically and logistically...

5. HOW DO YOU PROCESS INFORMATION?

Several years ago, I was working in a team to plan a big event. My supervisor made us all work as a team before the big day. One of our activities has been to find out how each of us processes information...

6. HOW DO YOU MANAGE THE CALENDAR AND THE PROGRAM? WHICH APPLICATIONS / SYSTEM DO YOU USE?

Or you may receive even more specific questions, such as: What's on your calendar? Do you plan blocks of time to do certain types of work? Do you have an open calendar that everyone can see?...

7. HOW DO YOU ORGANIZE FILES, LINKS, AND TABS ON YOUR COMPUTER?

Just like your schedule, how you track files and other information is very important. After all, everything is digital!...

8. HOW TO PRIORITIZE WORK?

The day I watched Marie Forleo's film separating the important from the urgent, my life changed. Not all remote jobs start fast, but most of them are...

9. HOW DO YOU PREPARE FOR A MEETING AND PREPARE A MEETING? WHAT DO YOU SEE HAPPENING DURING THE MEETING?

Just as communication is essential when working remotely, so is organization. Because you won't have those opportunities in the elevator or a casual conversation in the lunchroom, you should take advantage of the little time you have in a video or phone conference...

10. HOW DO YOU USE TECHNOLOGY ON A DAILY BASIS, IN YOUR WORK AND FOR YOUR PLEASURE?

This is a great question because it shows your comfort level with technology, which is very important for a remote worker because you will be working with technology over time...