📡 NetFlow v5 Collector – Finale Version

Für MikroTik RouterOS 7 | Container (Alpine) | HTTP Plaintext API | /30 Subnetz

⚡ Finale Lösung – Minimalistisches Python-Script, kein Differenz-Overhead, reine Rohdaten im gewünschten Format

📘 Dokumentation: NetFlow-basierter Traffic-Collector

🎯 Ziel
Ersetzt das alte /ip accounting aus RouterOS 6. Sammelt NetFlow v5 Daten auf dem Router und liefert sie im Klartext-Format:
SRC_IP DST_IP BYTES PACKETS SRC_PORT PROTOCOL

📌 Voraussetzungen

Hardware

Komponente	Mindestanforderung	Empfehlung
Router	RouterOS 7.20+ (arm64)	RB5009, CCR2004, CHR
RAM	1 GB frei	2+ GB
Speicher	200 MB frei	USB-Stick oder SMB-Share
Netzwerk	Container bekommt eigenes Subnetz (172.17.0.0/30)

Software

Komponente	Version	Hinweis
RouterOS	7.22+ (stable)	Container-Funktion muss aktivierbar sein
Container-Paket	installiert	In Extra-Paketen enthalten
Windows-PC	beliebig	Für SSH und curl-Abrufe

⚠️ Wichtiger Hinweis: Container-Modus aktivieren

Nach /system/device-mode/update container=yes muss der Router physikalisch vom Strom getrennt werden (Cold Reboot).
Ein normaler Software-Neustart reicht nicht aus!

⚙️ Schritt-für-Schritt-Anleitung

1. Container-Modus aktivieren (⚠️ Strom ziehen!)

/system/device-mode/update container=yes

Jetzt sofort den Netzstecker ziehen, 10 Sekunden warten, wieder einstecken.

2. Bridge und virtuelles Interface erstellen (mit /30 Subnetz)

/interface veth add name=veth1 address=172.17.0.2/30 gateway=172.17.0.1
/interface bridge add name=container-bridge
/ip address add address=172.17.0.1/30 interface=container-bridge
/interface bridge port add bridge=container-bridge interface=veth1

3. Masquerading für Container-Internet (NEU – wichtig!)

/ip firewall nat add chain=srcnat out-interface-list=WAN src-address=172.17.0.0/30 action=masquerade

Warum Masquerading nötig ist:
Ohne diese NAT-Regel hat der Container keine Verbindung ins Internet. Du siehst das z. B. daran, dass apk update fehlschlägt.
Wichtig: Ersetze out-interface-list=WAN durch dein tatsächliches WAN-Interface (z. B. ether1), falls du keine Interface-Liste nutzt.

4. Container erstellen und starten

/container add remote-image=arm64v8/alpine:latest root-dir=disk1/alpine interface=veth1 cmd="python3 /netflow.py" start-on-boot=yes
/container start alpine:latest
/container print

5. Python im Container installieren

/container shell alpine:latest
apk update
apk add python3
exit

6. Finales NetFlow-Script im Container erstellen

/container shell alpine:latest
cat > /netflow.py << 'EOF'
import socket
import struct
from http.server import HTTPServer, BaseHTTPRequestHandler
from threading import Thread

cache = []
MAX_CACHE = 100000

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header('Content-Type', 'text/plain')
        self.end_headers()
        self.wfile.write("\n".join(cache[-MAX_CACHE:]).encode())
    def log_message(self, *args): pass

def udp_listener():
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    sock.bind(('0.0.0.0', 2055))
    while True:
        data, addr = sock.recvfrom(65535)
        if len(data) < 24: continue
        if struct.unpack('>H', data[0:2])[0] != 5: continue
        count = struct.unpack('>H', data[2:4])[0]
        off = 24
        for i in range(count):
            if off + 48 > len(data): break
            src = socket.inet_ntoa(data[off:off+4])
            dst = socket.inet_ntoa(data[off+4:off+8])
            pkts = struct.unpack('>I', data[off+16:off+20])[0]
            bytes_flow = struct.unpack('>I', data[off+20:off+24])[0]
            port = struct.unpack('>H', data[off+32:off+34])[0]
            proto = data[off+38]
            cache.append(f"{src} {dst} {bytes_flow} {pkts} {port} {proto}")
            off += 48
        if len(cache) > 2 * MAX_CACHE:
            cache[:MAX_CACHE] = []

Thread(target=udp_listener, daemon=True).start()
HTTPServer(('0.0.0.0', 8080), Handler).serve_forever()
EOF
exit

7. Container neu starten

/container stop alpine:latest
/container start alpine:latest

8. Traffic-Flow auf Router aktivieren

/ip traffic-flow set enabled=yes interfaces=sfp-sfpplus1
/ip traffic-flow target add dst-address=172.17.0.2 port=2055 version=5
/ip traffic-flow set active-flow-timeout=5s inactive-flow-timeout=5s

Hinweis: Ersetze sfp-sfpplus1 durch das Interface, das du überwachen möchtest (z. B. ether1 oder bridge).

9. Testen

Auf dem Router Traffic erzeugen:

/tool bandwidth-test address=8.8.8.8 duration=5

Daten abrufen (vom Windows-PC):

curl.exe http://172.17.0.2:8080/

📤 Ausgabeformat

62.146.206.118 3.67.32.184 226 3 62958 6
80.190.223.26 8.210.130.2 92 2 38262 6
62.146.214.242 149.154.167.41 546 4 51960 6

Bedeutung der Felder:

Feld	Bedeutung
src_ip	Quell-IP-Adresse
dst_ip	Ziel-IP-Adresse
bytes	Bytes (kumulativ, aus NetFlow v5)
packets	Pakete (kumulativ, aus NetFlow v5)
src_port	Quell-Port (0 bei ICMP)
protocol	6=TCP, 17=UDP, 1=ICMP

🔧 Fehlersuche

Problem	Lösung
Container bleibt `S` (stopped)	Container-Modus aktivieren (Schritt 1)
`curl` liefert leere Antwort	Bandwidth-Test laufen lassen, Target prüfen
`apk add python3` schlägt fehl	Masquerading prüfen (Schritt 3), DNS setzen: `echo "nameserver 8.8.8.8" > /etc/resolv.conf`
Container startet nicht nach Reboot	`/container set alpine:latest start-on-boot=yes`
Port 8080 nicht erreichbar	Firewall-Regel auf Router prüfen, Container-IP richtig?
`veth1` ist `I - INACTIVE`	Container läuft nicht: `/container start alpine:latest`

📦 Persistente Datenspeicherung (optional)

/container/mounts add dst=/data src=disk1/netflow_data
/container set alpine:latest mounts=netflow-list
/container stop alpine:latest && /container start alpine:latest

Script anpassen: DATA_DIR = "/data"

🌐 Netzwerkübersicht (Subnetz 172.17.0.0/30)

IP	Verwendung
172.17.0.0	Netzadresse
172.17.0.1	Gateway (Router / Bridge)
172.17.0.2	Container
172.17.0.3	Broadcast

🎯 Zusammenfassung

Funktion	Status
NetFlow-Export vom Router	✅ aktiv
Container mit Python	✅ läuft
HTTP-API auf Port 8080	✅ verfügbar
Gewünschtes Ausgabeformat	✅ implementiert
Masquerading für Internet	✅ eingerichtet

📘 Documentation: NetFlow-based Traffic Collector

🎯 Goal
Replace the old /ip accounting from RouterOS 6. Collect NetFlow v5 data on the router and deliver it in plain text format:
SRC_IP DST_IP BYTES PACKETS SRC_PORT PROTOCOL

📌 Requirements

Hardware

Component	Minimum	Recommendation
Router	RouterOS 7.20+ (arm64)	RB5009, CCR2004, CHR
RAM	1 GB free	2+ GB
Storage	200 MB free	USB stick or SMB share
Network	Container gets own subnet (172.17.0.0/30)

⚠️ Important: Activate Container Mode (Cold Reboot required!)

After /system/device-mode/update container=yes, you must physically disconnect power from the router.
A normal software reboot is not sufficient!

⚙️ Step-by-Step Guide

1. Activate Container Mode (⚠️ Cold Reboot!)

/system/device-mode/update container=yes

Immediately unplug the power cord, wait 10 seconds, plug it back in.

2. Create Bridge and Virtual Interface (with /30 subnet)

/interface veth add name=veth1 address=172.17.0.2/30 gateway=172.17.0.1
/interface bridge add name=container-bridge
/ip address add address=172.17.0.1/30 interface=container-bridge
/interface bridge port add bridge=container-bridge interface=veth1

3. Masquerading for Container Internet Access (CRITICAL!)

/ip firewall nat add chain=srcnat out-interface-list=WAN src-address=172.17.0.0/30 action=masquerade

Why masquerading is needed:
Without this NAT rule, the container has no internet connection. You will see this when apk update fails.
Important: Replace out-interface-list=WAN with your actual WAN interface (e.g. ether1) if not using an interface list.

4. Create and Start Container

/container add remote-image=arm64v8/alpine:latest root-dir=disk1/alpine interface=veth1 cmd="python3 /netflow.py" start-on-boot=yes
/container start alpine:latest
/container print

5. Install Python Inside Container

/container shell alpine:latest
apk update
apk add python3
exit

6. Create Final NetFlow Script Inside Container

/container shell alpine:latest
cat > /netflow.py << 'EOF'
import socket
import struct
from http.server import HTTPServer, BaseHTTPRequestHandler
from threading import Thread

cache = []
MAX_CACHE = 5000

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header('Content-Type', 'text/plain')
        self.end_headers()
        self.wfile.write("\n".join(cache[-MAX_CACHE:]).encode())
    def log_message(self, *args): pass

def udp_listener():
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    sock.bind(('0.0.0.0', 2055))
    while True:
        data, addr = sock.recvfrom(65535)
        if len(data) < 24: continue
        if struct.unpack('>H', data[0:2])[0] != 5: continue
        count = struct.unpack('>H', data[2:4])[0]
        off = 24
        for i in range(count):
            if off + 48 > len(data): break
            src = socket.inet_ntoa(data[off:off+4])
            dst = socket.inet_ntoa(data[off+4:off+8])
            pkts = struct.unpack('>I', data[off+16:off+20])[0]
            bytes_flow = struct.unpack('>I', data[off+20:off+24])[0]
            port = struct.unpack('>H', data[off+32:off+34])[0]
            proto = data[off+38]
            cache.append(f"{src} {dst} {bytes_flow} {pkts} {port} {proto}")
            off += 48
        if len(cache) > 2 * MAX_CACHE:
            cache[:MAX_CACHE] = []

Thread(target=udp_listener, daemon=True).start()
HTTPServer(('0.0.0.0', 8080), Handler).serve_forever()
EOF
exit

7. Restart Container

/container stop alpine:latest
/container start alpine:latest

8. Enable Traffic Flow on Router

/ip traffic-flow set enabled=yes interfaces=sfp-sfpplus1
/ip traffic-flow target add dst-address=172.17.0.2 port=2055 version=5
/ip traffic-flow set active-flow-timeout=5s inactive-flow-timeout=5s

Note: Replace sfp-sfpplus1 with the interface you want to monitor (e.g. ether1 or bridge).

9. Testing

Generate traffic on router:

/tool bandwidth-test address=8.8.8.8 duration=5

Fetch data (from Windows PC):

curl.exe http://172.17.0.2:8080/

📤 Output Format

62.146.206.118 3.67.32.184 226 3 62958 6
80.190.223.26 8.210.130.2 92 2 38262 6
62.146.214.242 149.154.167.41 546 4 51960 6

Field meaning:

Field	Meaning
src_ip	Source IP address
dst_ip	Destination IP address
bytes	Bytes (cumulative from NetFlow v5)
packets	Packets (cumulative from NetFlow v5)
src_port	Source port (0 for ICMP)
protocol	6=TCP, 17=UDP, 1=ICMP

🔧 Troubleshooting

Problem	Solution
Container stays `S` (stopped)	Activate container mode (step 1)
`curl` returns empty response	Run bandwidth test, check target
`apk add python3` fails	Check masquerading (step 3), set DNS: `echo "nameserver 8.8.8.8" > /etc/resolv.conf`
Container doesn't start after reboot	`/container set alpine:latest start-on-boot=yes`
Port 8080 unreachable	Check firewall rules on router, container IP correct?
`veth1` shows `I - INACTIVE`	Container not running: `/container start alpine:latest`

📦 Persistent Storage (optional)

/container/mounts add dst=/data src=disk1/netflow_data
/container set alpine:latest mounts=netflow-list
/container stop alpine:latest && /container start alpine:latest

Adjust script: DATA_DIR = "/data"

🌐 Network Overview (Subnet 172.17.0.0/30)

IP	Usage
172.17.0.0	Network address
172.17.0.1	Gateway (Router / Bridge)
172.17.0.2	Container
172.17.0.3	Broadcast

🎯 Summary

Feature	Status
NetFlow export from router	✅ active
Container with Python	✅ running
HTTP API on port 8080	✅ available
Desired output format	✅ implemented
Masquerading for internet	✅ configured