Network State Recorder － User Manual

Network State Recorder (NSR) is a tool that periodically fetches network state data from NDTwin and stores it in JSON files, which are then compressed into ZIP archives for efficient storage. The recorded data can be used for the Visualizer and Web-GUI to replay network states over time. For installation, you can follow this link

Table of Contents

• Features
• Configuration
• Usage
• Output Files
• Troubleshooting

Features

• Periodic Data Collection: Automatically fetches network flow and graph data from NDTwin at configurable intervals
• Efficient Storage: Compresses JSON data into ZIP archives to minimize disk usage
• Multi-threaded Architecture: Concurrent data fetching, writing, and compression for optimal performance
• Configurable Logging: Daily log rotation with customizable log levels
• Graceful Shutdown: Properly handles SIGINT/SIGTERM signals for clean termination

Configuration

NSR uses YAML configuration files located in the project directory.

Main Configuration (NSR.yaml)

This file configures the Nornir framework settings:

---
inventory:
  plugin: SimpleInventory
  options:
    host_file: "./setting/recorder_setting.yaml"

runner:
  plugin: threaded
  options:
    num_workers: 1

logging:
  enabled: false

Recorder Settings (setting/recorder_setting.yaml)

This file contains the NSR-specific configuration:

---
Recorder:
  data:
    ndtwin_kernel: "http://127.0.0.1:8000"
    request_interval: 5    # Data fetch interval in seconds (integer, >= 1)
    storage_interval: 2    # File rotation interval in minutes (integer, >= 1)
    display_on_console: true  # Enable real-time logging output to console (true/false)
    log_level: "DEBUG"     # Logging level: TRACE, DEBUG, INFO, WARNING, ERROR

Configuration Parameters

Parameter	Type	Default	Description
ndtwin_kernel	string	http://127.0.0.1:8000	URL of the NDTwin kernel
request_interval	integer	5	How often to fetch data from NDTwin (seconds)
storage_interval	integer	2	How often to rotate and compress JSON files (minutes)
display_on_console	boolean	true	Enable real-time logging output to console. Set to false to log only to files in logs/ directory
log_level	string	DEBUG	Minimum logging level to record

Log Levels (from most to least verbose)

• TRACE - Most detailed logging
• DEBUG - Debugging information
• INFO - General operational messages
• WARNING - Warning messages
• ERROR - Error messages only

Usage

Starting NSR

Option 1: Using the Start Script (Recommended for Background Operation)

./start_network_state_recorder.sh

This runs NSR in the background using nohup, allowing you to close the terminal while NSR continues running.

Notice: this script will change the value of display_on_console in config file to false

Option 2: Running Directly (Foreground)

python3 network_state_recorder.py

This runs NSR in the foreground. If you want to monitor the log in real-time, you need to set the display_on_console to true

Stopping NSR

Option 1: Using the Stop Script

./stop_network_state_recorder.sh

This script finds the NSR process and sends a graceful termination signal (SIGTERM).

Notice: this script will change the value of display_on_console in config file to true

Option 2: Manual Termination

If running in foreground, press Ctrl+C to stop.

If running in background:

# Send termination signal
sudo kill -15 $(pgrep -f network_state_recorder.py)

Checking NSR Status

To verify if NSR is running:

pgrep -f network_state_recorder.py

If a process ID is returned, NSR is running.

Output Files

Recorded Data Location

All recorded data is stored in the ./recorded_info/ directory (created automatically).

File Naming Convention

Files are named with the following format:

YYYY_MM_DD_HH-MM-SS_<datatype>.json

After compression:

YYYY_MM_DD_HH-MM-SS_<datatype>_json.zip

Data Types:

• flowinfo - Network flow information from NDTwin
• graphinfo - Network graph/topology information from NDTwin

File Structure

Each JSON file contains newline-delimited JSON records with the following structure:

{"timestamp": 1704067200000, ...}
{"timestamp": 1704067205000, ...}

If the JSON file is flowinfo type, then the JSON formate is as below:

{"timestamp": 1704067200000, "flowinfo":{[...]}}

If the JSON file is flowinfo type, then the JSON formate is as below:

{"timestamp": 1704067200000, "edges":{[...]}, "nodes":[{...}]}

• timestamp: Unix timestamp in milliseconds when data was fetched
• flowinfo: A specify key to the original NDTwin API response.
• edges & nodes: The original NDTwin API response.

Logs Location

Log files are stored in the ./logs/ directory with daily rotation:

logs/NSR_YYYY-MM-DD.log

Troubleshooting

Common Issues

1. "NDTwin server is not reachable"

Cause: NSR cannot connect to the NDTwin server.

Solutions:

• Verify NDTwin server is running
• Check the ndtwin_kernel URL in setting/recorder_setting.yaml
• Ensure no firewall is blocking the connection
• Test connectivity: curl http://<your ip>/ndt/get_detected_flow_data

2. "No Recorder setting found"

Cause: Configuration file is missing or malformed.

Solutions:

• Ensure setting/recorder_setting.yaml exists
• Verify YAML syntax is correct
• Check that the Recorder host is properly defined

3. Permission Denied When Running Scripts

Cause: Script files don't have execute permissions.

Solution:

chmod +x start_network_state_recorder.sh stop_network_state_recorder.sh

4. Cannot Stop NSR with stop_network_state_recorder.sh

Cause: May require sudo privileges.

Solution:

sudo ./stop_network_state_recorder.sh

Or manually:

sudo kill -15 $(pgrep -f network_state_recorder.py)

5. High Disk Usage

Cause: Data is being recorded faster than it can be compressed, or storage_interval is too long.

Solutions:

• Increase request_interval to reduce data volume
• Decrease storage_interval to compress files more frequently
• Monitor disk space regularly

Viewing Logs

To monitor NSR activity which running in background in real-time:

tail -f logs/NSR_$(date +%Y-%m-%d).log

API Endpoints Used

NSR fetches data from the following NDTwin API endpoints:

Endpoint	Description
/ndt/get_detected_flow_data	Network flow detection data
/ndt/get_graph_data	Network topology/graph data