Organizational Knowledge

Before the next failure, write down what your team already knows: your cluster names, how services connect, which signals are noise, what the last incident taught you. The Knowledge Library is where that context lives so NOFire can apply it the moment it’s needed. When something breaks, NOFire searches your library for entries relevant to the issue at hand and applies them before querying any data source. When you assess deploy risk, it uses your service topology and blast radius context automatically. The more you write, the faster your team reaches the right decision.

Before the next deploy

Service topology, dependencies, and known failure modes inform blast radius and deploy risk decisions

When something breaks

NOFire applies your runbooks and past incident patterns at the start of every evidence analysis

Reliability memory

Learnings from resolved incidents are captured and applied to the next one automatically

What to write

Write what a senior engineer already knows but no tool can discover: your cluster names, how services connect, which signals are noise, and what the last incident taught you. Before your next on-call rotation, these entries make every query NOFire runs more accurate. Start with these:

One service doc per critical service: owner, dependencies, log query template, known failure modes
Your cluster and namespace layout. NOFire needs this to query the right place.
Any runbook for a failure mode that has happened more than once
Observability blind spots: services with no traces, noisy log sources, unreliable metrics

Categories

Each entry belongs to one of five categories. The category tells NOFire how to interpret what you’ve written and when to apply it. Find the Knowledge Library in Settings → Knowledge Library.

Category	What it is	When NOFire uses it
Service doc	Facts about a service: owner, dependencies, logs, failure modes	Any investigation touching that service
Runbook	Step-by-step diagnosis and fix for a known failure mode	When an investigation matches the failure pattern
Incident summary	Record of a past incident: what broke, root cause, resolution	When a new incident resembles a past one
Dashboard guidance	How to read your dashboards: which panels matter, known noise	When interpreting metric signals
Investigation pattern	Non-obvious, environment-specific investigation approaches — which service to check first for a symptom, multi-signal correlations, and observability blind spots	When standard analysis isn’t enough

Write entries in Markdown. Use Affected entities to scope an entry to specific services — it’s a free-text tag field on the new item form where you type service or component names. NOFire prioritizes those entries when analyzing an issue with that service.

Service doc

Facts about a service: what it does, who owns it, how it connects to other services, where its logs are, and known failure modes. This is the most useful category to start with. NOFire applies it any time it’s investigating that service.Good to include: repo, team, SLOs, dependencies, log query templates, known failure modes, investigation guidance specific to this service.

# payments-service

**Repository:** github.com/acme/payments-service
**Team:** @payments-team
**On-call:** #payments-oncall
**SLOs:** p99 < 200ms, error rate < 0.1%, 99.9% availability

## Dependencies
- Upstream: checkout-api, order-service
- Downstream: Stripe API, payments-db (Postgres), payments-cache (Redis)

## Logs
{cluster="app3-cluster", namespace="payments", service_name="payments-service"}
Use `| detected_level="error"` to filter for errors only.

## Failure modes
- **Stripe API timeouts** — cascade upstream to checkout-api. Check `stripe_request_duration_p99` first; if elevated, root cause is Stripe-side, not payments-service. See runbook "payments-service — Stripe API timeout".
- **Redis pool exhaustion at >500 RPS** — `redis_connected_clients` near `maxclients` (default 512). See the "payments-service — Redis connection pool exhaustion" runbook for the fix.
- **DB connection leak after deploy** — check `pg_stat_activity` for idle connections; seen after connection-pool config changes. See runbook "payments-service — DB connection leak after deploy".

## Investigation guidance
Errors on payments-service frequently originate from Stripe or the DB,
not from the service code itself. Always check external dependencies before
looking at application logs. Trace propagation is enabled — use traces to
identify which downstream call is failing.

Runbook

Step-by-step diagnosis and resolution for a specific, known failure mode. NOFire uses these when the investigation points to a pattern your team has handled before.Good to include: what triggers this, how to confirm it, exact commands or queries to run, fix steps, and what to watch after the fix.

# payments-service — Redis connection pool exhaustion

## When this applies
- payments-service or checkout-api showing latency spikes
- redis_connected_clients metric near or at maxclients value
- Error logs contain "too many connections" or "connection pool timeout"

## Confirm it
Check: redis_connected_clients{service="payments-redis"} / redis_maxclients
If this ratio is above 0.9, pool exhaustion is the likely cause.

## Fix
1. Restart affected pods immediately — releases held connections
   kubectl rollout restart deployment/payments-service -n payments
2. Watch redis_connected_clients drop below 200 within 2 minutes
3. If it climbs again within 30 minutes, increase pool size:
   Set REDIS_MAX_POOL_SIZE from 50 to 100 in Helm values and redeploy

## Root cause pattern
Usually triggered by a traffic spike or a batch job during peak hours.
The nightly report job (02:00 UTC) has caused this twice — check if
the incident started around that time.

# payments-service — Stripe API timeout

## When this applies
- payments-service latency or errors, with upstream impact on checkout-api
- A Stripe (external dependency) slowdown is suspected

## Confirm it
Check `stripe_request_duration_p99` first. If elevated, the root cause is
Stripe-side, not payments-service. Check https://status.stripe.com before
investigating payments-service further.

## Action
- If `stripe_request_duration_p99` is elevated: treat as Stripe-side. Do not dig
  into payments-service application logs; communicate impact to checkout-api owners.
- If it is normal: the timeout is not Stripe — continue investigating payments-service.

# payments-service — DB connection leak after deploy

## When this applies
- payments-service degradation shortly after a deploy
- Especially after changes to connection-pool settings

## Confirm it
Check `pg_stat_activity` for idle connections accumulating on payments-db.

## Root cause pattern
Seen twice after config changes to the connection-pool settings — a deploy that
altered pool settings leaked connections.

## Fix
> TODO (service owner): add the team's actual remediation steps (e.g. revert the
> pool-config change and/or restart pods). The original docs did not include the
> fix steps, so confirm them before publishing rather than inventing them.

Incident summary

A record of a past incident: what broke, why, and how it was resolved. NOFire uses these to recognize recurring patterns and avoid re-investigating what your team has already solved.Good to include: date, what failed, root cause, how it was found, resolution, and what to watch for next time.

# 2025-09-14: Checkout latency spike (SEV2)

**Duration:** 14:20 – 15:05 UTC (45 min)
**Affected:** checkout-api, order-service (downstream impact)
**Impact:** p99 latency 280ms → 4.2s, ~1,200 failed checkouts

## Root cause
Redis connection pool exhaustion on payments-service. The nightly batch
reporting job was rescheduled to 14:15 UTC due to a cron config change in
the previous deploy, coinciding with afternoon peak traffic.

## How it was found
Checkout latency alert fired at 14:23. Initial investigation pointed to
checkout-api but traces showed the bottleneck was in payments-service.
`redis_connected_clients` was at 511/512.

## Resolution
1. Restarted payments-service pods at 14:51 — latency recovered within 3 min
2. Reverted batch job cron schedule to 02:00 UTC

## Follow-up
- Move batch job to a separate Redis instance (ticket #PAY-441)
- Add alert on `redis_connected_clients > 450`

Dashboard guidance

Notes on how to read your dashboards: which panels matter, what thresholds are significant, and what patterns are expected noise vs. real signals.Good to include: dashboard name and link, which panels to look at first, known false positives, expected periodic spikes.

# Payments Overview dashboard (Grafana)

**Link:** https://grafana.acme.com/d/payments-overview

## Key panels

**Stripe request duration (p99)**
Normal: <150ms. Anything above 500ms means Stripe is degraded, not us.
Check https://status.stripe.com before investigating further.

**Redis connections**
Normal operating range: 50–200. Above 450 is a warning sign.
Above 490 is critical — pool exhaustion is imminent.

**Payment success rate**
Drops below 99.5% only during real incidents. Occasional single-digit drops
in the 1-minute view are noise from retries — look at the 5-minute average.

## Known noise
- The "DB query duration" panel spikes to ~500ms every night at 02:00 UTC
  during the batch reporting job. This is expected.
- The "Error rate" panel shows a small bump (~0.3%) during every deploy
  due to connection draining. Ignore if it recovers within 5 minutes.

Investigation pattern

Non-obvious approaches that have worked in your environment: how signals relate to each other, which dependencies are high-risk, and where your observability has blind spots.Good to include: which service to check first for a given symptom, known blind spots, multi-service correlation patterns, services where alert signals are misleading.

# Frontend service — investigation guidance

The frontend service sits behind the frontend-proxy and is the entry point
for all user requests. Errors on the frontend frequently originate from
downstream services (auth, checkout, payments), not from frontend code itself.

Do not treat frontend errors as root cause. Use distributed traces to find
which downstream call is failing before investigating frontend logs.

Common pattern:
- Frontend p99 latency spike → look at http.url or http.path_group in
  traces to identify which downstream service is slow
- Frontend 5xx rate increase → check if error logs reference a downstream
  service name — if yes, start there

## Evidence queries
Group traces by downstream call: @http.path_group, @http.url, @http.target
Filter frontend spans to find slow dependencies: service:frontend @duration:>500ms

## Blind spot
billing-v1 does not emit structured logs or traces. If checkout is failing
and payments look healthy, check billing-v1 application logs directly.

Seed it from past postmortems

If your team has been running postmortems in Slack, NOFire can scan those channels and extract what was learned: past root causes, resolution steps, recurring failure modes. NOFire adds them to the library automatically. This gives you years of institutional memory on day one, without having to write it all manually. Go to Settings → Slack → Postmortem Scan to configure:

Postmortem channel prefixes: channels matching these prefixes will be scanned (e.g. inc)
Lookback days: how far back to scan, from 7 to 365 days
Click Scan Now to run immediately, or save to enable periodic scanning

Postmortem Scan settings showing enable toggle, channel prefix input with 'inc' tag, lookback days set to 180, and Scan Now button

NOFire reads through the channel history, extracts key findings from each incident, and adds them as knowledge entries. Channels are tracked to avoid duplicate processing.

The scan runs in the background and does not affect the channels or their members.

How NOFire adds to it automatically

The library also grows on its own. After completed investigations, NOFire captures confirmed root causes and useful patterns, and stores them in the library. You can review and remove any auto-generated entries from the library at any time.

If NOFire gets something wrong, use the feedback buttons on the investigation result to mark it incorrect and describe what was wrong. The correction is saved and applied the next time a similar incident occurs.

Get Started

Skills

Platform Features

Setup & Integrations

Security

Organizational Knowledge

Before the next deploy

When something breaks

Reliability memory

What to write

Categories

Seed it from past postmortems

How NOFire adds to it automatically

Next Steps

Autonomous Investigations

Root Cause Analysis

Before the next deploy

When something breaks

Reliability memory

​What to write

​Categories

​Seed it from past postmortems

​How NOFire adds to it automatically

​Next Steps

Autonomous Investigations

Root Cause Analysis

What to write

Categories

Seed it from past postmortems

How NOFire adds to it automatically

Next Steps