Why it matters
A payment passes through an app, a gateway, several services, a payment switch and a core. When customers report failures, the team must find which step is failing, for whom and since when, in minutes. Observability is the ability to answer those questions from the data systems emit, without guessing or adding code during the incident.
Three kinds of signal
Metrics are numbers over time: requests per second, error rate, response time, queue depth. They are cheap to store and ideal for dashboards and alerts. Logs are records of individual events, with detail about what happened. They are essential for investigation and audit, but costly at volume. Traces follow one request across every service it touches, showing where time was spent and where it failed. Together they answer “is something wrong?”, “what happened?” and “where?”.
The landscape
| Technology | Role | Typical use in Indian banking | Notes |
|---|---|---|---|
| Prometheus | Metrics collection and alerting rules | Metrics from container platforms and services | The standard for Kubernetes metrics |
| Grafana | Dashboards across many data sources | Operations and business dashboards | Works with metrics, logs and traces |
| OpenTelemetry | Vendor-neutral standard and libraries for emitting signals | Instrumenting services once, sending anywhere | Avoids lock-in to one monitoring vendor |
| Elasticsearch or OpenSearch, Loki, Splunk | Log storage and search | Application and security logs | Plan retention and cost carefully |
| Jaeger, Tempo | Distributed tracing back ends | Following requests across microservices | Needs trace IDs passed through every hop |
| Commercial observability suites | All signals in one managed product | Teams that prefer a managed service | Check where data is stored and processed |
Typical uses are general industry patterns, not claims about named banks. Product names are for reference only.
From data to decisions: SLOs
A service level indicator (SLIService level indicator. The measurement used to judge a service against its objective, such as the share of successful payments.) is a measure of what users experience, such as the share of payments that succeed within two seconds. A service level objective (SLOService level objective. A target for how reliable or fast a service should be, measured over a period.) is the target for it, such as 99.9% over 30 days. The gap between the target and perfection is the error budget: while it lasts, the team can ship changes; when it runs out, reliability work comes first.
Scale makes these measures essential. With UPIUnified Payments Interface. NPCI's real-time system for paying from a bank account with a UPI ID, QR code or phone number, through apps on a phone. alone averaging 791 million a day1Business Standard. UPI sets new record at 24.51 bn transactions in Aug, value at ₹29.82 trn. As of 31 August 2026., a success rate that looks excellent as a percentage can still mean many failed payments. Failures also have regulatory clocks attached: RBI's framework for failed transactions, in force since 15 October 2019, sets outer time limits for reversal and requires compensation to be paid automatically, without a complaint.2Reserve Bank of India. Harmonisation of Turn Around Time (TAT) and customer compensation for failed transactions using authorised Payment Systems (RBI/2019-20/67). As of 20 September 2019.
Alerting, on-call and incident review
Good alerts fire on symptoms users would notice, such as rising payment failures, rather than on every internal blip. Each alert has an owner, a runbook and a clear severity. On-call rotations share the load fairly and include time to recover. After every significant incident, a blameless review records what happened, why it made sense to the people involved at the time and what will change. Incident timelines built from good observability data also feed regulatory reporting: A service provider must report cyber incidents to the regulated entity without undue delay, so that the entity can report them to RBI within 6 hours of the provider detecting them.3Reserve Bank of India. Master Direction on Outsourcing of Information Technology Services (RBI/2023-24/102). As of 10 April 2023.
How to choose
Instrument once with OpenTelemetry, so the choice of back end can change later. Start with the few SLIs that matter most to customers, such as payment success and response time, and build dashboards and alerts around them before adding more. Decide log retention by need and rule, not by default.
Trade-offs
| Choice | You gain | You give up |
|---|---|---|
| Open-source stack | Control, no per-host licence | A team to run and scale it |
| Managed suite | Fast start, less to operate | Cost at volume; data leaves the bank’s estate |
| Log everything | Nothing missing during an investigation | High cost and personal data in logs |
| Sample traces | Lower cost | The one failing request may not be sampled |
Pitfalls
- Dashboards nobody watches and alerts nobody owns.
- Alerting on causes, such as high CPU, instead of symptoms users feel.
- Personal data in logs, such as full account numbers or card details.
- Broken traces because one service does not pass the trace ID on.
- Reviews that assign blame, which teach people to hide problems.
Questions for a design review
- What are this service’s SLIs and SLOs, and who agreed them?
- If payments start failing, which alert fires first, and who receives it?
- Can one payment be traced from the app to the core?
- What personal data could appear in these logs, and how is it masked?
- How long are logs kept, and why that long?