AI Box Deployment Manual

1. Deployment Overview and Prerequisites

1.1 Scope of This Deployment

The underlying Debian 12 system has been pre-installed and booted by the vendor. This deployment covers the following items (starting from the "Software Environment" section; system flashing is not included):

1. Python 3.11 virtual environment + RKNN Lite SDK (NPU inference)
2. Transfer the project code (main.py + 15 algorithm categories in src/) to the box
3. Deploy 15 .rknn model files to the models/ directory (core step)
4. Configure the MQTT server address + camera RTSP address in config.yaml
5. Single-run verification → systemd service auto-start → acceptance

1.2 Hardware Verification

Item	Expected Value	How to Check
SoC Model	RK3588 (4×A76 + 4×A55)	cat /proc/cpuinfo
Memory	4GB LPDDR4X	free -h
Storage	32GB eMMC (expandable via MicroSD)	df -h
NPU Device Node	/dev/galcore exists	ls -la /dev/galcore
Network	Wired RJ45 / 4G module	ip addr
Camera	USB (UVC) or RTSP IP camera	lsusb / v4l2-ctl --list-devices

⚠️ Required check before deployment: If /dev/galcore does not exist, the NPU driver has not been loaded. Do not continue the installation. First run dmesg | grep rknpu to troubleshoot the driver, or contact the vendor to confirm the kernel version.

1.3 Preparing the Transfer Method

Three ways to transfer code and models from the development machine to the box:

# 方式 A：scp（推荐，同一局域网）
scp -r edge-ai-box/* root@192.168.1.200:/opt/edge-ai/

# 方式 B：U 盘拷贝（无网络时）
#   U 盘挂载：mount /dev/sda1 /mnt && cp -r /mnt/edge-ai-box/* /opt/edge-ai/

# 方式 C：git 拉取（代码已托管时）
cd /opt && git clone <仓库地址> edge-ai

2. Python Environment + RKNN SDK Installation

2.1 Installing Python 3.11 and the Virtual Environment

# 1. 安装 Python 与基础工具
apt update
apt install -y python3.11 python3.11-dev python3.11-venv python3-pip vim curl git htop

# 2. 创建项目目录与虚拟环境
mkdir -p /opt/edge-ai
python3.11 -m venv /opt/edge-ai/venv
source /opt/edge-ai/venv/bin/activate
pip install --upgrade pip setuptools wheel

Note: If apt cannot find python3.11, first run apt install -y software-properties-common && add-apt-repository ppa:deadsnakes/ppa (enable the corresponding repository on Debian) and then retry.

2.2 Installing the RKNN Lite2 SDK (NPU Inference Engine)

# 1. 下载适用于 RK3588 / aarch64 / Python3.11 的 RKNN Lite2 wheel
cd /tmp
wget https://github.com/airockchip/rknn-toolkit2/releases/download/v2.0.0/rknn_toolkit_lite2-2.0.0-cp311-cp311-linux_aarch64.whl

# 2. 安装（在 venv 内）
pip install rknn_toolkit_lite2-2.0.0-cp311-cp311-linux_aarch64.whl

# 3. 验证 NPU 可用
python3 -c "from rknnlite.api import RKNNLite; print('RKNN Lite OK')"
# 成功输出：RKNN Lite OK

⚠️ Version consistency (the most common pitfall):
① .rknn model files must be generated with an RKNN-Toolkit2 (PC-side conversion tool) of the same major version as the SDK on the box — for example, if rknn-toolkit-lite2 v2.0.0 is installed on the box, the models must be exported on a PC using rknn-toolkit2 v2.0.0;
② If the vendor firmware has a different version pre-installed, first run pip show rknn-toolkit-lite2 to confirm the version, then decide on the model conversion version accordingly.

2.3 Installing Project Dependencies

cd /opt/edge-ai
pip install -r requirements.txt
# 关键依赖：paho-mqtt / opencv-python / numpy / PyYAML / msgpack / colorlog / requests

# 可选（人脸识别 30 万库检索加速）
pip install faiss-cpu

# 摄像头依赖（Debian 需装系统级 OpenCV）
apt install -y libopencv-dev python3-opencv ffmpeg v4l-utils

# 验证 OpenCV
python3 -c "import cv2; print('OpenCV', cv2.__version__)"

Note: If pip is slow to compile or fails when installing opencv, use pip install opencv-python-headless instead (no GUI is needed for pure inference scenarios), then run apt install python3-opencv to provide GStreamer acceleration.

3. Project Code Deployment

3.1 Directory Structure

/opt/edge-ai/
├── main.py # 主程序入口
├── config.yaml # 运行配置（MQTT/摄像头/识别参数）
├── model_config.yaml # 15 类算法模型配置
├── requirements.txt
├── models/ # ← 15 个 .rknn 模型文件放这里（核心）
│ └── face_db/ # 人脸特征库
├── src/
│ ├── algorithms/ # 15 类算法实现（每类一个 .py）
│ ├── inference/ # RKNN 推理引擎
│ ├── mqtt/ # MQTT 上报客户端 + 协议
│ └── utils/
├── snapshots/ # 抓拍图（告警事件截图）
└── logs/ # 运行日志

3.2 Code Transfer and Directory Initialization

# 传输（按 1.3 任选一种），然后：
cd /opt/edge-ai
mkdir -p models/face_db snapshots logs
chmod 644 models/*.rknn 2>/dev/null || true
chmod +x main.py

3.3 Code Self-Check (Optional but Recommended)

# 语法检查全部 Python 文件
find /opt/edge-ai -name "*.py" -exec python3 -m py_compile {} \;
echo "全部通过"

4. Model File Deployment (Core Step)

✅ This step is the key to the deployment. The 15 algorithm categories correspond to 20 model files (some algorithms reuse the same model). All models use the .rknn format (converted and exported from PyTorch/ONNX using RKNN-Toolkit2).

4.1 Model File Checklist (Must Be Complete)

Algorithm	Priority	Model File (under models/)	Input Size	Description
Safety helmet safety_hat	P0	safety_hat.rknn	640×640	Detects helmet / no helmet
Reflective vest reflective_vest	P0	reflective_vest.rknn	640×640	Detects worn / not worn
Area intrusion intrusion	P0	intrusion.rknn	640×640	Person detection
Face recognition face_recognition	P1	face_detect.rknn + face_feature.rknn	320 / 112	Detection + feature extraction (dual model)
Smoking smoking	P1	smoking.rknn	416×416	Detects smoking
Phone use phone	P1	phone.rknn	416×416	Detects phone use
Crowd gathering gathering	P1	gathering.rknn	640×640	People counting + gathering determination
Vehicle attributes vehicle	P1	vehicle_detect.rknn + vehicle_attr.rknn	640	Detection + type/color
On/off duty on_off_duty	P1	person_detect.rknn	416×416	Person detection
People counting people_count	P1	people_count.rknn (can reuse person_detect)	640×640	V3.2 virtual line counting
Vehicle structuring vehicle_struct	P0	vehicle_struct_detect.rknn + vehicle_struct_type.rknn + vehicle_struct_color.rknn + vehicle_struct_dir.rknn	640	V3.2 4 models: detection + 6 vehicle types + 14 colors + 4 directions
Illegal parking vehicle_parking	P0	vehicle_parking.rknn (can reuse vehicle_detect)	640×640	V3.2 parking duration grading
Vehicle brand/model vehicle_brand	P1	brand_detect.rknn + brand_class.rknn	320	V3.2 classification across 2,000 classes
Smoke smoke	P0 Fire safety	smoke_detect.rknn + smoke_type.rknn	640	V3.2 white/black/gray smoke
Flame fire	P0 Fire safety	fire_detect.rknn + fire_type.rknn	640	V3.2 open/glowing flame

Total files required: 20 .rknn (15 algorithm categories, including multi-model combinations such as face recognition / vehicle structuring).

4.2 Model Conversion Reference (PC Side, Not the Box)

Models must be converted using rknn-toolkit2 installed on a PC (x86); the box only runs rknn-toolkit-lite2. Example conversion script:

# PC: models/convert/convert_yolo.py
from rknn.api import RKNN

rknn = RKNN()
# target_platform 必须与盒子 SoC 一致：RK3588
rknn.config(
    mean_values=[[0, 0, 0]],
    std_values=[[255, 255, 255]],
    target_platform='rk3588',
    optimization_level=3,
    quantized_dtype='asymmetric_quantized-8',  # INT8 量化
)

# 从 ONNX 转换（先导出 ONNX，再转 RKNN）
rknn.load_onnx(model='safety_hat.onnx')
rknn.build(do_quantization=True, dataset='./calibration.txt')  # 需校准图集
rknn.export_rknn('safety_hat.rknn')
rknn.release()

⚠️ Conversion key points:
① target_platform='rk3588' (note: the rk3576 mentioned in older documents is an earlier model; this box is RK3588);
② INT8 quantization requires a calibration.txt list of calibration images (20–100 representative images per category);
③ Classification models (vehicle type/color/direction/brand/smoke type/fire type) use softmax on the output layer, while detection models (YOLO) output 3 heads;
④ After conversion, verify accuracy on the PC using the simulator via rknn.build(do_quantization=True) (compare quantized output against float output).

4.3 Model File Transfer and Verification

# 传输到盒子
scp models/*.rknn root@192.168.1.200:/opt/edge-ai/models/

# 校验完整性（数量 + 权限）
ls -la /opt/edge-ai/models/*.rknn | wc -l   # 期望 20
chmod 644 /opt/edge-ai/models/*.rknn

# 单个模型推理冒烟（先验证 NPU 能跑通）
cd /opt/edge-ai && source venv/bin/activate
python3 -c "
from src.inference.engine import RKNNEngine
import cv2
engine = RKNNEngine('models/safety_hat.rknn', npu_core=2)
if engine.load():
    img = cv2.imread('snapshots/test.jpg')  # 放一张测试图
    out = engine.infer(img)
    print('推理 OK, 输出:', [o.shape for o in out])
    engine.release()
else:
    print('模型加载失败 — 检查 .rknn 文件与 SDK 版本')
"

4.4 Face Database Initialization (Optional, Only When face_recognition Is Enabled)

# 人脸库路径：config.yaml 的 recognition.face_db_path
mkdir -p /opt/edge-ai/models/face_db

# 每个人员一个 .pkl：{"name": "张三", "features": np.ndarray(512,)}
# 从后端系统同步特征文件后构建 faiss 索引（30 万级加速）
python3 -c "
import faiss, numpy as np, pickle, os
features = []
names = []
for f in sorted(os.listdir('/opt/edge-ai/models/face_db')):
    if f.endswith('.pkl'):
        with open(f'/opt/edge-ai/models/face_db/{f}', 'rb') as fp:
            d = pickle.load(fp)
            features.append(d['features']); names.append(d['name'])
if features:
    idx = faiss.IndexFlatIP(512)
    idx.add(np.array(features, dtype=np.float32))
    faiss.write_index(idx, '/opt/edge-ai/models/face_db/index.faiss')
    print(f'人脸索引构建完成: {idx.ntotal} 人')
"

5. Configuration File Changes (MQTT / Camera)

5.1 MQTT Reporting Configuration (config.yaml)

mqtt:
  broker: "192.168.1.10"        # ← 改为后端服务器 IP（必须改）
  port: 1883
  client_id: "edge-ai-box-001"  # ← 每台盒子唯一
  topic_recognition: "vehicle/ai/recognition"
  topic_telemetry: "vehicle/telemetry"
  topic_command: "vehicle/command"
  username: "edge_box"          # 后端 MQTT 账号（如有）
  password: "your_password"
  keepalive: 60
  qos: 1

Note: The MQTT broker must be changed to the actual IP of the backend server. If MQTT is not enabled on the backend, you can first verify using the HTTP fallback channel (main.py detects an MQTT failure and degrades to logging only).

5.2 Camera Configuration (config.yaml)

cameras:
  - id: 1
    name: "大门/出入口"
    rtsp_url: "rtsp://192.168.1.100:554/stream1"   # ← 改为实际 RTSP 地址
    vehicle_id: 1
    detection_area: null       # 区域入侵多边形 [[x1,y1],[x2,y2],...]
    parking_area: null         # 违停检测区域
    count_line: [[0, 540], [1920, 540]]   # 人流量虚拟线
    enable_algorithms:
      - safety_hat
      - reflective_vest
      - people_count
      - vehicle_struct
      - vehicle_brand
      - vehicle_parking
  - id: 2
    name: "驾驶位/车厢"
    rtsp_url: "rtsp://192.168.1.100:554/stream2"   # ← 改为实际地址
    vehicle_id: 1
    enable_algorithms:
      - smoking
      - phone
      - face_recognition
      - on_off_duty
      - gathering
      - intrusion
      - smoke
      - fire

⚠️ Camera configuration key points:
① For local capture from a USB camera, rtsp_url can also be set to /dev/video0 (main.py supports both);
② Assign enable_algorithms according to the camera's field of view: outdoor gates get vehicle-related algorithms + safety helmet, while the driver cabin gets behavior-related + fire-safety algorithms;
③ The detection_area/parking_area polygon coordinates must match the "AI detection area" configuration on the backend (drawn on the frontend with Leaflet and then pushed down).

5.3 Recognition Parameters (Adjust as Needed)

recognition:
  frame_interval: 5        # 每 5 帧推理 1 次（30fps → 6fps 推理）
  min_confidence: 0.5      # 低于此置信度丢弃
  auto_alert_confidence: 0.9  # ≥0.9 直接告警，不入复审
  dedup_interval: 5        # 同类型同目标去重（秒）
  gathering_threshold: 5   # 聚集人数阈值
  people_count_interval: 60    # 人流量聚合周期
  parking_duration_levels: [60, 300, 600]  # 违停分级时长
  fire_score_threshold: 0.6    # 消防更高阈值防误报

6. Startup and Functional Verification

6.1 Single Foreground Run (Verify First)

cd /opt/edge-ai && source venv/bin/activate
python3 main.py -c config.yaml -m model_config.yaml

Expected output:

[INFO] 加载模型: safety_hat OK
[INFO] 加载模型: reflective_vest OK
...
[INFO] 15 类算法全部加载
[INFO] MQTT 连接成功: 192.168.1.10:1883
[INFO] 摄像头 1 打开成功: rtsp://...
[INFO] 开始推理循环

6.2 Troubleshooting Model Loading Failures (Important)

Common errors: E RKNN: RKNN_ERR_MODEL_INVALID → the model file is corrupted or the version does not match; E RKNN: RKNN_ERR_PARAM_INVALID → input_size/anchors do not match the model.
Solution: ① Confirm the .rknn and SDK versions match (Section 4.2); ② Check that the input_size in model_config.yaml matches the model's actual input; ③ Re-export the model.

6.3 End-to-End Verification (Recognition Reporting → Backend Receipt)

# 方式 1：订阅 MQTT 观察识别事件
apt install -y mosquitto-clients
mosquitto_sub -h 192.168.1.10 -t "vehicle/ai/recognition" -v
# 期望看到：{"type":"recognition","recognitionType":"safety_hat","confidence":0.95,...}

# 方式 2：后端 API 查询（确认已入库）
curl -H "Authorization: Bearer <token>" \
  "http://192.168.1.10:8080/ai/recognition/list?pageNum=1&pageSize=5"
# 期望返回最近识别记录（含 recognitionType / confidence / snapshotUrl）

6.4 Fire-Safety Event Verification (CRITICAL, Highest Priority)

# 用打火机/手机屏幕模拟火焰靠近摄像头，期望：
# ① MQTT 收到 eventLevel=5 的 fire 事件
# ② 后端 /ai/fire-alarm/list 出现记录
# ③ 大屏触发全屏弹窗 + 声光报警（GPIO 联动）

7. Auto-Start and Supervision

7.1 Creating the systemd Service

cat > /etc/systemd/system/edge-ai-box.service << 'EOF'
[Unit]
Description=Edge AI Box — Smart Access Vehicle Recognition
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/edge-ai
Environment="PYTHONUNBUFFERED=1"
Environment="RKNN_LOG_LEVEL=2"
ExecStartPre=/bin/sleep 10        # 等待网络就绪
ExecStart=/opt/edge-ai/venv/bin/python main.py -c config.yaml -m model_config.yaml
ExecStop=/bin/kill -SIGTERM $MAINPID
Restart=on-failure
RestartSec=30
StandardOutput=append:/opt/edge-ai/logs/service.log
StandardError=append:/opt/edge-ai/logs/service_error.log
MemoryMax=2G
CPUQuota=300%

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable edge-ai-box
systemctl start edge-ai-box
systemctl status edge-ai-box

7.2 Log Rotation (Preventing Full Disk)

cat > /etc/logrotate.d/edge-ai-box << EOF
/opt/edge-ai/logs/*.log {
    daily
    rotate 7
    maxsize 100M
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}
EOF

7.3 Watchdog (Preventing a Hanging Service)

# systemd 定时器每 5 分钟健康检查，失败自动重启服务
cat > /etc/systemd/system/edge-ai-watchdog.service << 'EOF'
[Unit]
Description=Edge AI Watchdog
[Service]
Type=oneshot
ExecStart=/opt/edge-ai/venv/bin/python -c "
import requests, subprocess, sys
try:
    r = requests.get('http://192.168.1.10:8080/ai/recognition/list?pageSize=1', timeout=5)
    if r.status_code == 200:
        sys.exit(0)
except: pass
subprocess.run(['systemctl', 'restart', 'edge-ai-box'])
"
EOF
cat > /etc/systemd/system/edge-ai-watchdog.timer << 'EOF'
[Unit]
Description=Edge AI Watchdog Timer
[Timer]
OnBootSec=5min
OnUnitActiveSec=5min
[Install]
WantedBy=timers.target
EOF
systemctl enable edge-ai-watchdog.timer

Note: The watchdog's health check requests access the backend — if the backend IP changes, update it accordingly. For offline deployments (no backend), switch to a local process check instead (pgrep -f main.py).

8. Performance Tuning

8.1 Dual-Core NPU + Half Precision (model_config.yaml)

optimization:
  npu_cores: 2              # RK3588 双核 NPU 并行
  batch_size: 1
  fp16: true                # 半精度加速 1.5~2×
  perf_profile: "balance"   # performance / balance / power_saving

8.2 CPU Performance Mode (Can Be Skipped If Vehicle Stability Is the Priority)

echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu4/cpufreq/scaling_governor

8.3 Frame-Skipping Strategy (Balancing Compute)

# config.yaml
recognition:
  frame_interval: 5   # 安全帽/反光衣等低频场景
  # 吸烟/打电话: 3     # 中等频率
  # 入侵/聚集: 2       # 高频场景

8.4 Expected Metrics

Metric	Target	Description
Per-frame inference latency	< 50ms	NPU INT8
Concurrent multi-model FPS	> 10fps	Dual NPU cores + frame skipping
CPU usage	< 60%	Excluding video decoding
Memory	< 1.5GB	Including model loading
NPU temperature	< 80°C	Automatic fan speed control

9. Common Troubleshooting

Symptom	Troubleshooting Steps
NPU inference failure RKNNAPI init failed	① Check whether the driver is loaded with `dmesg
Model loading reports MODEL_INVALID	The .rknn file is corrupted or the conversion version does not match → re-export using the same-version Toolkit2 (4.2)
Cannot connect to MQTT	① ping 192.168.1.10; ② nc -zv 192.168.1.10 1883; ③ Check whether the username/password are correct; ④ Check whether MQTT is enabled on the backend (degrades to logging only if not enabled)
Camera cannot be opened	① USB: lsusb / v4l2-ctl --list-devices; ② RTSP: ffprobe rtsp://...; ③ Verify the rtsp_url spelling
Recognition events not reported	① Run in the foreground and check the logs; ② The threshold is too high (min_confidence); ③ The deduplication interval is too short; ④ The MQTT topic does not match what the backend subscribes to
Out of memory (MemoryError)	Reduce the number of concurrently loaded models / use npu_cores=1 / disable fp16 to reduce memory usage

Log Keyword Quick Reference

Log Keyword	Meaning	Resolution
RKNN init runtime failed	NPU occupied / kernel exception	rmmod rknpu && modprobe rknpu
MQTT connection refused	Broker not running / firewall	Check the Broker status and network
Failed to open RTSP stream	Camera disconnected	Check power/network, configure auto-reconnect
CUDA/OpenCL not available	Normal (NPU inference)	Ignore

10. Acceptance Checklist

• Debian 12 boots normally and /dev/galcore exists (NPU driver ready)
• Python 3.11 venv + rknn-toolkit-lite2 installed successfully (RKNN Lite OK)
• Project code deployed to /opt/edge-ai/ and dependencies installed
• 20 .rknn model files placed in models/ (ls | wc -l = 20)
• Single-model inference smoke test passed (safety_hat.rknn)
• MQTT broker in config.yaml changed to the backend IP
• Camera rtsp_url in config.yaml changed to the actual address
• python3 main.py runs in the foreground: all 15 model categories loaded with no errors
• Recognition events reported successfully over MQTT (visible via mosquitto_sub)
• The backend /ai/recognition/list returns recognition records
• Fire-safety events (smoke/flame) trigger CRITICAL + full-screen popup
• The systemd service edge-ai-box starts successfully and is enabled
• The service automatically recovers after reboot (auto-start verification)