Operations Guide - NOFire AI

NOFire Edge

Verification

After installation, verify the agent is healthy:

Check Pod Status:
kubectl get pods -n nofire-system

Check Logs:

kubectl logs -l app.kubernetes.io/name=nofire-edge -n nofire-system

Look for “Graph published successfully”.

Check Metrics: Port-forward the agent to access metrics:
kubectl port-forward svc/nofire-edge 8080:8080 -n nofire-system
Visit http://localhost:8080/metrics.

Metrics Reference

NOFire Edge exposes 13 metrics under the resource_graph_ prefix on port 8080 at /metrics.

Metric	Type	Labels	Description
`resource_graph_nodes_total`	Gauge	`node`, `kind`	Node count in the graph by resource kind.
`resource_graph_nodes_all_total`	Gauge	`node`	Total node count in the graph.
`resource_graph_edges_total`	Gauge	`node`, `type`	Edge count by relationship type.
`resource_graph_edges_all_total`	Gauge	`node`	Total edge count in the graph.
`resource_graph_processing_seconds`	Histogram	—	Time spent processing graph updates.
`resource_graph_api_duration_seconds`	Histogram	`endpoint`	Duration of API requests (`graph`, `graph_dot`, `graph_stats`, `log_level`).
`resource_graph_workqueue_depth`	Gauge	`queue_name`	Current workqueue depth.
`resource_graph_workqueue_latency_seconds`	Histogram	`queue_name`	Time an item spends in the workqueue before processing.
`resource_graph_update_errors_total`	Counter	`type`	Failed graph update operations.
`resource_graph_publisher_errors_total`	Counter	`publisher_type`	Failed publish attempts.
`resource_graph_publisher_duration_seconds`	Histogram	`publisher_type`	Duration of publish operations.
`resource_graph_modify_duration_seconds`	Histogram	—	Time holding the graph write lock.
`resource_graph_read_duration_seconds`	Histogram	—	Time holding the graph read lock.

DNSTap connection status is not exposed as a Prometheus metric. The DNSTap collector uses internal atomic counters for diagnostics. To verify DNSTap health, check the Edge logs for connection messages and look for EDGE_TYPE_CAUSAL edges in the graph.

Alerts

Publisher Errors

rate(resource_graph_publisher_errors_total[5m]) > 0

Graph publish operations are failing. Check outbound connectivity and API key validity.

Graph Not Updating

changes(resource_graph_nodes_all_total[10m]) == 0

The graph hasn’t changed in 10 minutes. May indicate Kubernetes informer issues or the Edge process being stuck.

High Update Error Rate

rate(resource_graph_update_errors_total[5m]) > 0

Graph update operations are failing. Check logs for the specific error type.

High Graph Processing Latency

histogram_quantile(0.99, rate(resource_graph_processing_seconds_bucket[5m])) > 1

The 99th percentile of graph processing time exceeds 1 second. May indicate a very large graph or resource contention.

Troubleshooting

1. DNSTap Not Connected

Symptoms: No EDGE_TYPE_CAUSAL edges in the graph, logs show no DNSTap connection messages. Fix:

Verify CoreDNS config has the correct Edge IP.
Check if a NetworkPolicy is blocking traffic from kube-system to nofire-system on port 6000.
Ensure CoreDNS was restarted after config change.

2. Publisher Errors

Symptoms: Logs show “Failed to publish graph”. Fix:

Check outbound internet access.
Verify API Key is correct.
Check if the NOFire AI endpoint is reachable (curl -v https://my.nofire.ai/api/edge). Without a valid API key it returns 401 — that still proves the endpoint is reachable.

3. High Memory Usage

Symptoms: Pod OOMKilled. Fix:

Increase memory limit in Helm values.
Reduce graph.maxPruneAge to keep the graph smaller.

Edge Proxy

These sections apply when you have on-prem connections deployed alongside Edge.

Verification

Check Pod Status

kubectl get pods -l app.kubernetes.io/component=edge-proxy -n nofire-system

The pod should be in Running state with 1/1 ready.

Check Handshake Logs

kubectl logs -l app.kubernetes.io/component=edge-proxy -n nofire-system | grep "StreamAck"

A successful connection shows:

StreamAck received: STATUS_OK  edge_instance_id=<id>  protocol_version=1

Check Health Endpoint

kubectl exec -it deploy/nofire-edge-edge-proxy -n nofire-system -- \
  wget -qO- http://localhost:8081/healthz

Check Metrics

kubectl exec -it deploy/nofire-edge-edge-proxy -n nofire-system -- \
  wget -qO- http://localhost:8081/metrics | grep nofire_proxy

Metrics Reference

The Edge Proxy exposes 12 metrics under the nofire_proxy_ namespace on port 8081 at /metrics.

Metric	Type	Labels	Description
`nofire_proxy_requests_total`	Counter	`connection_type`, `method`, `status_code`	Total proxy requests by backend type, HTTP method, and status code.
`nofire_proxy_request_duration_seconds`	Histogram	`connection_type`, `method`	Proxy request duration by backend type and HTTP method.
`nofire_proxy_requests_in_flight`	Gauge	—	Number of proxy requests currently in flight.
`nofire_proxy_stream_connections`	Gauge	—	Whether the NOFire stream is connected (`0` or `1`).
`nofire_proxy_stream_reconnects_total`	Counter	—	Total stream reconnection attempts.
`nofire_proxy_heartbeats_sent_total`	Counter	—	Total heartbeat messages sent to NOFire.
`nofire_proxy_heartbeats_received_total`	Counter	—	Total heartbeat messages received from NOFire.
`nofire_proxy_response_chunks_total`	Counter	`connection_type`	Total response chunks sent (for large responses split across messages).
`nofire_proxy_response_bytes_total`	Counter	`connection_type`	Total bytes of responses sent to NOFire.
`nofire_proxy_redactions_applied_total`	Counter	—	Total PII redaction operations applied.
`nofire_proxy_health_checks_total`	Counter	`config_key`, `status`	Health check results by connector and status.
`nofire_proxy_errors_total`	Counter	`type`	Errors by type (`stream`, `request`, `health`).

Alerts

Stream Disconnected

nofire_proxy_stream_connections == 0

The Edge Proxy is not connected to NOFire. Check network connectivity and logs.

High Reconnect Rate

rate(nofire_proxy_stream_reconnects_total[5m]) > 0.1

More than one reconnect every 10 minutes. Indicates an unstable network or NOFire-side issues.

Backend Errors

rate(nofire_proxy_errors_total{type="request"}[5m]) > 0

Requests to data sources are failing. Check backend health and credentials.

Semaphore Saturation

nofire_proxy_requests_in_flight >= 10

All concurrent request slots are in use. Consider increasing edgeProxy.stream.maxConcurrent.

Troubleshooting

”authentication failed” in logs

Cause: The NOFIRE_API_KEY is incorrect or the nofire-credentials secret is missing. Fix:

# Verify the secret exists
kubectl get secret nofire-credentials -n nofire-system

# Re-create if needed
kubectl create secret generic nofire-credentials \
  --from-literal=api-key=YOUR_CORRECT_API_KEY \
  -n nofire-system --dry-run=client -o yaml | kubectl apply -f -

# Restart the pod
kubectl rollout restart deploy/nofire-edge-edge-proxy -n nofire-system

“cluster not found” in logs

Cause: The clusterName in your Helm values does not match any cluster registered in the NOFire dashboard. Fix: Verify config.kube.clusterName in your values file matches the cluster name shown in the NOFire dashboard.

”dial failed” or repeated reconnections

Cause: The pod cannot reach NOFire at the configured serverAddress, which must be edge.nofire.ai:443. Fix: The Edge Proxy image is distroless and has no shell, so run the probes from a throwaway pod in the same namespace:

kubectl run netcheck --rm -it --restart=Never -n nofire-system \
  --image=nicolaka/netshoot -- \
  sh -c 'nslookup edge.nofire.ai && \
    curl -s -o /dev/null -w "%{http_code} %{content_type} http/%{http_version}\n" \
      -X POST --http2 -H "content-type: application/grpc" https://edge.nofire.ai/'

Any image carrying nslookup and curl works — substitute one from your own registry if Docker Hub is not reachable or your PodSecurity policy blocks the pod. A healthy endpoint resolves and returns 200 application/grpc http/2. Anything else — a text/html content type, or http/1.1 — means the address is not terminating gRPC. Check that outbound TCP 443 is allowed through your firewall and network policies.

”PermissionDenied … 403 … content-type text/html”

Symptom: The proxy reconnects in a loop and logs:

receive StreamAck: rpc error: code = PermissionDenied desc = unexpected HTTP status code
received from server: 403 (Forbidden); transport: received unexpected content-type "text/html"

Cause: serverAddress points at an HTTPS endpoint that is not the gRPC endpoint — most often the dashboard, my.nofire.ai:443. The CDN in front of it rejects content-type: application/grpc with a 403 HTML page. PermissionDenied here is gRPC’s mapping of HTTP 403, not an authentication failure: your API key is not the problem, and no NOFire backend code ran. Fix: Set edgeProxy.stream.serverAddress to edge.nofire.ai:443 and upgrade the release.

”no connector found” for a query

Cause: NOFire sent a query for a connection type that is not configured or is disabled. Fix: Check onPremConnections in your values file. Ensure the connector has enabled: true (the default) and the type field matches.

”health check request failed” in logs

Cause: The Edge Proxy cannot reach the backend data source, or the credentials are wrong. Fix:

# Test connectivity to the backend from the pod
kubectl exec -it deploy/nofire-edge-edge-proxy -n nofire-system -- \
  wget --spider --timeout=5 http://prometheus.monitoring:9090

# Verify the secret is mounted
kubectl exec -it deploy/nofire-edge-edge-proxy -n nofire-system -- \
  ls /etc/nofire/secrets/prometheus-creds/

“proxy overloaded” responses (HTTP 503)

Cause: More concurrent requests than maxConcurrent (default: 10). Fix: Increase the concurrency limit in your values:

edgeProxy:
  stream:
    maxConcurrent: 20

Pod CrashLoopBackOff

Cause: Invalid configuration (missing required fields, bad JSON, invalid log level). Fix:

# Check the logs for the specific validation error
kubectl logs -l app.kubernetes.io/component=edge-proxy -n nofire-system --previous

Common validation errors: missing stream.serverAddress, invalid logLevel, connection with enabled: true but no url.

Shared Operations

ServiceMonitor

When monitoring.serviceMonitor.enabled is true in your Helm values, the chart creates ServiceMonitors for both NOFire Edge and the Edge Proxy. This allows Prometheus Operator to automatically scrape the /metrics endpoint of each component.

Edge: scraped on port http (8080) at /metrics
Edge Proxy: scraped on port http (8081) at /metrics (only created when onPremConnections is configured)

monitoring:
  serviceMonitor:
    enabled: true
    labels:
      release: prometheus
    interval: 30s

Upgrade and Rollback

Both components are managed by the same Helm chart. NOFire Edge is deployed as a Deployment and the Edge Proxy as a Deployment (only created when onPremConnections is configured). To upgrade:

helm upgrade nofire-edge nofire/nofire-edge -f values.yaml -n nofire-system

To rollback:

helm rollback nofire-edge -n nofire-system

The Edge Proxy uses a ConfigMap checksum annotation on its Deployment. Any change to edgeProxy.* or onPremConnections.* in your Helm values triggers a pod restart on the next helm upgrade. The Edge DaemonSet rolls out automatically when the Helm release is upgraded.

The Edge Proxy does not support hot-reload of configuration. A pod restart is required for any configuration change to take effect.

​NOFire Edge

​Verification

​Metrics Reference

​Alerts

​Publisher Errors

​Graph Not Updating

​High Update Error Rate

​High Graph Processing Latency

​Troubleshooting

​1. DNSTap Not Connected

​2. Publisher Errors

​3. High Memory Usage

​Edge Proxy

​Verification

​Check Pod Status

​Check Handshake Logs

​Check Health Endpoint

​Check Metrics

​Metrics Reference

​Alerts

​Stream Disconnected

​High Reconnect Rate

​Backend Errors

​Semaphore Saturation

​Troubleshooting

​”authentication failed” in logs

​“cluster not found” in logs

​”dial failed” or repeated reconnections

​”PermissionDenied … 403 … content-type text/html”

​”no connector found” for a query

​”health check request failed” in logs

​“proxy overloaded” responses (HTTP 503)

​Pod CrashLoopBackOff

​Shared Operations

​ServiceMonitor

​Upgrade and Rollback

NOFire Edge

Verification

Metrics Reference

Alerts

Publisher Errors

Graph Not Updating

High Update Error Rate

High Graph Processing Latency

Troubleshooting

1. DNSTap Not Connected

2. Publisher Errors

3. High Memory Usage

Edge Proxy

Verification

Check Pod Status

Check Handshake Logs

Check Health Endpoint

Check Metrics

Metrics Reference

Alerts

Stream Disconnected

High Reconnect Rate

Backend Errors

Semaphore Saturation

Troubleshooting

”authentication failed” in logs

“cluster not found” in logs

”dial failed” or repeated reconnections

”PermissionDenied … 403 … content-type text/html”

”no connector found” for a query

”health check request failed” in logs

“proxy overloaded” responses (HTTP 503)

Pod CrashLoopBackOff

Shared Operations

ServiceMonitor

Upgrade and Rollback