Share
Tweet
Share
Share
The principles of Site Reliability Engineering (SRE) help technology companies address a key challenge—finding a balance between development speed and service stability. Alexandr Hacicheant, Head of Reliability Engineering at Mayflower, explains how to minimize downtime, automate incident management, and improve collaboration between teams.
Both development teams and tech company management are interested in delivering features quickly. However, the focus on speed often leads to poor-quality or insufficiently tested code being deployed to production, which performs poorly under load.
Imagine a large marketplace facing two significant events on Friday: a seasonal sale with hundreds of sellers launching their promotions and, on the same day, developers deploying a new feature they hurriedly wrote in just a few hours. The tests pass, everything appears functional, and the developers celebrate the release. However, by Saturday morning, tens of thousands of users are unable to view products. Instead of a 20 ms response time, the site experiences a 10,000 ms delay. The sudden increase in load overwhelmed the new feature, rendering it ineffective.
Under the traditional model, the first to notice the issue are customer support representatives, who escalate it to technical support. Upon realizing the severity, they call the operations team. Hours later, a crisis team assembles, and after more time passes, programmers join to diagnose and resolve the problem. Restoring normal operations could take hours, disrupting the sale and incurring significant financial losses for the business.
With development and operations teams working separately, such incidents occur regularly. Many updates are manually rolled out to all users at once, and problems are addressed reactively—only after a service has already failed.
Tensions between teams exacerbate the situation. Developers prioritize fast feature deployment, while operations engineers focus on system stability and minimizing changes. Without clear performance metrics, this leads to blame-shifting and demotivation. Responsibility becomes diluted, failures become routine, and businesses suffer revenue and reputational damage due to dissatisfied users leaving.
How SRE is Structured
Site Reliability Engineering (SRE) provides a structured approach to addressing these issues both reactively and proactively. Originally proposed by Google over 20 years ago, SRE is a set of practices, tools, and cultural principles designed to improve collaboration between teams and enhance system reliability.
At its core, SRE introduces common metrics and standards for assessing service performance and incident management. Failures are treated as inevitable, with the primary goal being to minimize their impact and extract valuable insights from them.
The approach incorporates several key principles, monitored by an SRE team, which replaces the traditional operations team:
Service Level Objective (SLO): A target level of system performance that the company commits to maintaining. It defines acceptable risk thresholds based on monitoring data, such as uptime, response time, and the percentage of successful requests.
Example: An SLO may require 99.9% availability per month, allowing for a maximum of 43 minutes of downtime. If metrics approach critical levels, engineers receive alerts via tools like Prometheus Alertmanager.
Service Level Indicator (SLI): A measurable metric that reflects how well a service performs. It can track average response time, API availability, or the percentage of successful requests. SLIs help objectively assess system health and identify issues early.
Example: Latency metrics can be monitored via Grafana, while availability metrics can be tracked using various exporters for Kubernetes cluster, databases, messages systems and others. .
Error Budget: Since failures are inevitable, an acceptable failure threshold is established. If an SLO is 99.9%, the error budget is 0.1%. This metric ensures a balance between development speed and system stability.
Example: If the error budget is exceeded, CI/CD pipelines can automatically block new releases until the issue is resolved.
Automation: A core principle of SRE is minimizing manual intervention. Tools like Ansible, Terraform, and CI/CD systems automate recovery procedures for common failures, ensuring faster and more consistent responses.
How SRE Works in Practice
Proactive Incident Management
One of the key benefits of SRE is the ability to predict and prevent problems. Setting up monitoring and alerting helps identify anomalies before they become critical.
For example, you can use metrics in Prometheus and dashboards in Grafana to set up an alert system when SLI thresholds are exceeded. Alerts – in the form of a message or a call to the responsible SRE engineer – will come if the web server response time or the percentage of unsuccessful requests exceeds the predefined SLI values.
For quick response, runbook documents are created with instructions for troubleshooting typical problems. This is a short description of how to solve problems that the team has already encountered and the contacts of people who were involved in the recovery. For example, if SLI response time exceeds 200 ms, the runbook contains a description of actions to analyze the causes of the delay, check the load on the servers, and perform optimizations:
- dashboard with recurring errors
- dashboard load and number of requests per application
- A link to a run-books document that describes how a similar problem has been solved in the past
Based on this data, you can restore service operation relatively quickly without gathering a consilium of programmers – as in the example at the beginning of the article.
Incident Retrospectives
Beyond resolving failures, SRE emphasizes learning from them to prevent recurrence. Postmortems analyze incidents in detail—documenting diagnostics, resolution timelines, and preventive measures.
Postmortems are typically stored in knowledge management systems like Confluence or Notion, following structured templates to ensure consistency.
Capacity Planning
SRE teams ensure that system performance meets current and expected user demands while avoiding excessive infrastructure costs.
Example: To prevent downtime due to resource shortages during peak hours, about 30% of a resource buffer is maintained. Auto-scalers like Kubernetes HPA (Horizontal Pod Autoscaler) or Public Cloud Autoscalers (AWS, GKE) dynamically adjust infrastructure in real time. This prevents marketplace crashes during major sales, ensuring smooth operations and customer satisfaction.
Dynamic Release Management
Instead of large, infrequent updates, SRE promotes frequent, incremental releases deployed to a limited user base first.
Example: A feature update is initially rolled out to 5% of users—preferably those who haven’t purchased a subscription—before expanding to a broader audience. If issues arise, Canary Release tools like Gitlab Canary Deployments or Spinnaker automatically roll back changes. If stable, the update is gradually expanded.
This strategy minimizes business risks while enabling faster feature rollouts and responsiveness to user needs.
Challenges in Implementing SRE
While SRE significantly reduces manual work, enhances transparency, and improves reliability, its implementation presents challenges:
- High Initial Costs: Setting up monitoring, automation, postmortems, and runbooks requires substantial time and investment.
- Defining the Right Metrics: Poorly configured SLIs may trigger excessive or insufficient alerts, leading to alert fatigue or overlooked critical failures.
- Cultural Shifts: SRE necessitates closer collaboration between developers and operations teams. Resistance may arise in organizations with established processes or punitive cultures that discourage transparency in postmortems.
- Accountability and Incentives: SRE assigns developers responsibility for business-critical functions, offering rewards such as bonuses for staying within the error budget or even stock options to align motivation with business success.
Conclusion
From a technical perspective, SRE reduces downtime through proactive monitoring, automation, and retrospectives, allowing teams to focus on strategic initiatives rather than constant firefighting.
For businesses, SRE implementation brings clear benefits: improved user satisfaction due to system stability and reduced engineer burnout due to well-structured processes and automation.
However, implementing SRE requires commitment—from technical setup to cultural transformation. Companies that embrace it gain a competitive edge, ensuring both innovation speed and long-term system reliability.
