> ## Documentation Index
> Fetch the complete documentation index at: https://docs.nofire.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Network Observability
> Map the real service-to-service calls in your cluster: see blast radius before you deploy and trace the failing call when something breaks.
The DNS-derived service dependency map shows which services *can* talk to each other. Network observability adds the calls that *actually happen*: request and error rates, latency, HTTP paths, and DNS lookups. Edge reads them from a stack you already run (**Hubble**, **Istio**, or **Caretta**), with no extra agent or sidecar to deploy.
It changes two decisions. *Before a deploy*, you see the real call paths and the blast radius of what you're about to change. *When something breaks*, you trace the failing call instead of guessing from an elevated error rate. Network observability is **disabled by default**; turn it on when you want observed traffic layered on top of the service dependency map.
## Which source should I use?
**Prefer auto-detect.** Leave `source` empty and Edge activates exactly one source per collection cycle, in precedence order **Hubble → Istio → Caretta**, picking the richest one present and never counting the same call twice. Pin a source only to override the default, for example to force `hubble` when your cluster also runs Istio.
* **Hubble (Cilium)**: preferred. Kernel-level eBPF gives per-call detail: HTTP method, path, and status; latency percentiles; DNS queries and NXDOMAIN; gRPC and Kafka; forwarded vs dropped flows. Use it if you run Cilium.
* **Istio**: choose only if your workloads are already meshed. Edge reads request and error rates from Prometheus. Sidecar-injected services only (un-meshed workloads are invisible), and no latency percentiles.
* **Caretta**: a lightweight fallback when you run neither Cilium nor Istio. Connection counts across TCP and UDP, without a service mesh, but no HTTP paths, status codes, or latency.
If your cluster also exports Hubble metrics to Prometheus, Edge uses them to keep the service dependency map stable. Prometheus counters accumulate over time, so low-traffic or infrequent calls stay on the map even when a single collection window misses them. The per-call detail — latency, HTTP paths, verdicts — still comes from Hubble directly. This happens automatically with no extra configuration.
### What each source surfaces
| Signal | Hubble | Istio | Caretta |
| :-------------------------------------- | :----: | :-------------: | :-----: |
| Service-to-service call map | ✓ | ✓ (meshed only) | ✓ |
| Request & error counts | ✓ | ✓ | ✓ |
| HTTP method & path | ✓ | - | - |
| HTTP status codes | ✓ | ✓ (5xx rate) | - |
| gRPC / Kafka detection | ✓ | ✓ | - |
| DNS queries & NXDOMAIN | ✓ | - | - |
| Network verdicts (drop / forward) | ✓ | - | - |
| Top HTTP paths with error rates | ✓ | - | - |
| External service calls (out-of-cluster) | ✓ | - | ✓ |
| Latency percentiles (p50 / p95 / p99) | ✓ | - | - |
With Hubble, an investigation points at the exact failing call: the method and path behind a 5xx, an NXDOMAIN lookup, or a dropped flow, not just an elevated error rate. Istio reports request and error rates for meshed services; Caretta reports connection counts. Latency shows as `n/a` for Istio and Caretta, since neither source measures it.
## During an investigation
What you see on a dependency edge depends on which source is active:
* **Hubble**: the richest signal. Open any edge in the service graph to see the top HTTP paths with per-path error rates, the exact status codes returned, latency percentiles, DNS lookups that failed with NXDOMAIN, and any flows that were dropped by network policy (with the policy name). You can pinpoint *which call* is failing, not just *which service* is elevated.
* **Istio**: request and error rates per dependency, limited to sidecar-injected workloads. Tells you *which pair* is misbehaving and how bad the error rate is; does not surface individual calls, paths, or latency.
* **Caretta**: connection-level presence. Confirms that two services are communicating; does not show what they are saying. Useful as a last resort to rule out complete connectivity loss between services.
Hubble and Caretta both track calls leaving your cluster — external dependencies appear on the map alongside internal services. Istio covers only sidecar-injected workloads and does not observe egress to un-meshed external endpoints.
## Enable
Add the `netobs` block to your Edge values and run a helm upgrade:
```yaml theme={null}
config:
netobs:
enabled: true # disabled by default
interval: "60s" # collection cadence
source: "" # auto-detect; or pin "hubble" | "istio" | "caretta"
edgeExistenceTtl: "2h" # keep an observed call on the map this long after it was last seen, while both endpoints still exist
prometheusUrl: "" # Istio/Caretta only; empty => in-cluster discovery
prometheusTimeout: "10s" # Istio/Caretta only; per-query timeout
excludeNamespaces: # Hubble only; drop these namespaces' flows server-side, before they reach Edge
- kube-system
- monitoring
```
```bash theme={null}
helm upgrade nofire-edge nofire/nofire-edge -f values.yaml -n nofire-system
```
### Settings
| Field | Default | Description |
| :------------------ | :------------------------------ | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `enabled` | `false` | Master switch. While off, Edge starts no collector and opens no Hubble or Prometheus connection. |
| `interval` | `60s` | How often Edge pulls flows and refreshes the observed-call edges. |
| `source` | `""` (auto) | Pin a single source — `hubble`, `istio`, or `caretta`. Empty auto-detects in precedence Hubble → Istio → Caretta. |
| `edgeExistenceTtl` | `2h` | How long an observed call stays on the map after it was last seen, while **both** endpoint services still exist in the graph. Stops low-traffic calls from flickering when a single collection misses them. Floored at 2× `interval`; a smaller value is clamped up to that floor. |
| `prometheusUrl` | `""` | Istio/Caretta only. Explicit Prometheus base URL; empty uses in-cluster discovery. |
| `prometheusTimeout` | `10s` | Istio/Caretta only. Per-query (PromQL) timeout; raise it on busy clusters. |
| `excludeNamespaces` | `["kube-system", "monitoring"]` | Hubble only. Drops flows whose source or destination pod is in one of these namespaces before they leave the relay, cutting CPU and bandwidth on busy clusters. Auto-seeded from your `kube.namespaceFilter` deny-list. |
If Istio or Caretta publish metrics to Prometheus, Edge discovers Prometheus in-cluster automatically. Set `prometheusUrl` only if you need to point Edge at a specific Prometheus service.
## Verify
Check the Edge logs for the active traffic source:
```bash theme={null}
kubectl logs -l app.kubernetes.io/name=nofire-edge -n nofire-system | grep "active traffic source"
```
You should see a line naming the active source:
```text theme={null}
[netobs] active traffic source: hubble
```
In the NOFire dashboard, open the service graph and confirm dependency edges appear between your services. With Hubble, select any edge to see latency percentiles, top HTTP paths, and error rates populated.
## Requirements
* **Hubble**: Cilium with Hubble enabled; `hubble-relay` reachable from the Edge pod. GKE Dataplane V2 (`anetd`) and AKS managed Cilium are supported — Edge discovers the required TLS certificates from the cluster automatically.
* **Istio**: Istio metrics scraped by Prometheus; only sidecar-injected workloads are observed.
* **Caretta**: Caretta running with its metrics scraped by Prometheus.
If none is present, Edge keeps mapping dependencies from DNS as usual; network observability simply stays inactive until a source appears.