Introduction

This document outlines the comprehensive debugging steps for Network Performance Management (NPM) issues, including both manual debugging commands and the automated NPM debugger tool.

Manual Debugging Steps

1. Check NetFlow Collector Pod Status
   Verify that NetFlow collector pods are operational:

   kubectl get pods
   

   This command shows the current status of all pods, including whether the NPM collector pods are running, pending, or in error states.

2. Verify Service Configuration
   Ensure that services exposing the NetFlow collector are properly configured:

   kubectl get svc
   

   This displays service endpoints, cluster IPs, and port configurations for the NPM collector services.

3. Check NPM Logs
   Monitor NPM logs to verify if the collector is receiving flows or encountering errors:

   kubectl logs -f <pod_name> -c npm-collector-app
   

   Replace <pod_name> with the actual pod name from step 1.

4. Verify Network Flow Reception
   Use tcpdump to check if NetFlow UDP packets are arriving at the gateway:

   tcpdump -i any udp port <NetFlow-port>
   

   Replace <NetFlow-port> with the UDP port your collector listens on.

5. Enable Debug Logs
   To enable verbose logging in the collector configuration, run:

   kubectl exec -it $(kubectl get pod -l app=npm-collector -o jsonpath='{.items[0].metadata.name}') -c npm-collector-app -- sh -c "echo 'verbose: enable' >> /etc/vflow/vflow.conf"
   

   To disable verbose logging, run:

   kubectl exec -it $(kubectl get pod -l app=npm-collector -o jsonpath='{.items[0].metadata.name}') -c npm-collector-app -- sh -c "echo 'verbose: disable' >> /etc/vflow/vflow.conf"
   

   This command updates the vflow.conf file to control the level of debug output, helping with detailed troubleshooting when needed.

NPM Debugger Tool

Download the NPM Debugger Tool

To get started with automated debugging, download the NPM Debugger executable:

Use the NPM Debugger

For comprehensive debugging, deploy the NPM debugger executable on the affected machine. The NPM debugger provides an interactive menu with the following options:

Option 1: Check NPM Collector Pod and Service Status

Verifies that the NPM collector pod(s) and associated Kubernetes services are running.

What it shows:

• Pod status (Running, CrashLoopBackOff, Pending, etc.).
• Service endpoints and cluster IPs.

Use this when:

• You suspect the collector is not operational.
• Metrics are missing.
• You want to confirm deployment health after installation.

Option 2: Update Node Port Range and Restart k3s

Modifies the node port range configuration used by Kubernetes (k3s) and restarts the service.

What it does:

• Updates the port range configuration.
• Applies the updated port range for NPM collector services.
• Restarts the k3s service to apply changes.

Use this when:

• You need to adjust which ports k3s allocates for NodePort services.
• There is a port conflict or firewall issue preventing access to collector endpoints.

Option 3: View Flow Packets Source IP Addresses

Displays live network flow packets received by the collector, focusing on source IP addresses.

What it shows:

• Real-time flow packet data.
• Source IP addresses of incoming flows.

Use this when:

• You want to validate that flow data is reaching the collector.
• You need to check which devices are sending traffic.

Option 4: Capture Flow Packets to File

Starts capturing incoming network flow packets and saves raw packet data to a file.

What it does:

• Captures network flow packets in real-time.
• Saves data for later analysis (compatible with tools like Wireshark).

Use this when:

• You need detailed packet-level troubleshooting.
• You must provide logs to support teams or vendors.

Option 5: Collect NPM Collector Logs

Gathers recent logs from the NPM collector pod(s) and compresses them for easy sharing.

What it does:

• Collects logs from NPM collector pods.
• Often compresses logs into a tarball format.

Use this when:

• Investigating errors or unexpected behavior.
• You want to archive logs for auditing or support purposes.

Option 6: Exit

Quits the NPM Debugging Tool.

Use this when:
You have completed your troubleshooting tasks.

Recommended Debugging Workflow

• Start with Option 1 to check overall system health.
• Use Option 3 to verify flow reception in real-time.
• Run Option 5 to collect logs if issues are found.
• Use Option 4 for detailed packet analysis when needed.
• Apply Option 2 if port configuration issues are identified.

This systematic approach ensures comprehensive troubleshooting of NPM-related issues while providing both automated and manual debugging capabilities.