Introduction
This document outlines the comprehensive debugging steps for Network Performance Management (NPM) issues, including both manual debugging commands and the automated NPM debugger tool.
Manual Debugging Steps
Check NetFlow Collector Pod StatusVerify that NetFlow collector pods are operational:
kubectl get pods
This command shows the current status of all pods, including whether the NPM collector pods are running, pending, or in error states.
Verify Service ConfigurationEnsure that services exposing the NetFlow collector are properly configured:
kubectl get svc
This displays service endpoints, cluster IPs, and port configurations for the NPM collector services.
Check NPM LogsMonitor NPM logs to verify if the collector is receiving flows or encountering errors:
kubectl logs -f <pod_name> -c npm-collector-app
Replace
<pod_name>
with the actual pod name from step 1.Verify Network Flow ReceptionUse tcpdump to check if NetFlow UDP packets are arriving at the gateway:
tcpdump -i any udp port <NetFlow-port>
Replace
<NetFlow-port>
with the UDP port your collector listens on.Enable Debug LogsTo enable verbose logging in the collector configuration, run:
kubectl exec -it $(kubectl get pod -l app=npm-collector -o jsonpath='{.items[0].metadata.name}') -c npm-collector-app -- sh -c "echo 'verbose: enable' >> /etc/vflow/vflow.conf"
To disable verbose logging, run:
kubectl exec -it $(kubectl get pod -l app=npm-collector -o jsonpath='{.items[0].metadata.name}') -c npm-collector-app -- sh -c "echo 'verbose: disable' >> /etc/vflow/vflow.conf"
This command updates the
vflow.conf
file to control the level of debug output, helping with detailed troubleshooting when needed.
NPM Debugger Tool
Download the NPM Debugger Tool
Caution
Ensure you have the necessary permissions to run executables on your system. Refer to your organization’s security policies before downloading and executing external tools.To get started with automated debugging, download the NPM Debugger executable:
NPM Debugger fileUse the NPM Debugger
For comprehensive debugging, deploy the NPM debugger executable on the affected machine. The NPM debugger provides an interactive menu with the following options:
Option 1: Check NPM Collector Pod and Service Status
Verifies that the NPM collector pod(s) and associated Kubernetes services are running.
What it shows:
- Pod status (Running, CrashLoopBackOff, Pending, etc.).
- Service endpoints and cluster IPs.
Use this when:
- You suspect the collector is not operational.
- Metrics are missing.
- You want to confirm deployment health after installation.
Option 2: Update Node Port Range and Restart k3s
Modifies the node port range configuration used by Kubernetes (k3s) and restarts the service.
What it does:
- Updates the port range configuration.
- Applies the updated port range for NPM collector services.
- Restarts the k3s service to apply changes.
Use this when:
- You need to adjust which ports k3s allocates for NodePort services.
- There is a port conflict or firewall issue preventing access to collector endpoints.
Option 3: View Flow Packets Source IP Addresses
Displays live network flow packets received by the collector, focusing on source IP addresses.
What it shows:
- Real-time flow packet data.
- Source IP addresses of incoming flows.
Use this when:
- You want to validate that flow data is reaching the collector.
- You need to check which devices are sending traffic.
Option 4: Capture Flow Packets to File
Starts capturing incoming network flow packets and saves raw packet data to a file.
What it does:
- Captures network flow packets in real-time.
- Saves data for later analysis (compatible with tools like Wireshark).
Use this when:
- You need detailed packet-level troubleshooting.
- You must provide logs to support teams or vendors.
Option 5: Collect NPM Collector Logs
Gathers recent logs from the NPM collector pod(s) and compresses them for easy sharing.
What it does:
- Collects logs from NPM collector pods.
- Often compresses logs into a tarball format.
Use this when:
- Investigating errors or unexpected behavior.
- You want to archive logs for auditing or support purposes.
Option 6: Exit
Quits the NPM Debugging Tool.
Use this when:You have completed your troubleshooting tasks.
Recommended Debugging Workflow
- Start with Option 1 to check overall system health.
- Use Option 3 to verify flow reception in real-time.
- Run Option 5 to collect logs if issues are found.
- Use Option 4 for detailed packet analysis when needed.
- Apply Option 2 if port configuration issues are identified.
This systematic approach ensures comprehensive troubleshooting of NPM-related issues while providing both automated and manual debugging capabilities.