Skip to content

Commit 0f39c0b

Browse files
committed
feat: RCA
1 parent 67b282c commit 0f39c0b

7 files changed

Lines changed: 341 additions & 228 deletions

File tree

blog/RCA/2025-12-12_1.mdx

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
---
2+
title: 2025/12/12 TREM Lite 偵測重複觸發問題
3+
authors: [yuyu1015]
4+
tags: [yuyu1015, rca]
5+
---
6+
7+
import SeverityBadge from "@site/src/components/SeverityBadge";
8+
9+
<SeverityBadge level="medium" />
10+
11+
---
12+
13+
## 事件概述
14+
15+
**發生時間:** 2025/12/12 04:24:28 UTC+8
16+
**觸發事件:** 地震後
17+
**問題描述:** TREM Lite 出現偵測重複觸發的問題
18+
**根本原因:** 因資料不一致導致系統異常
19+
20+
<iframe
21+
width="560"
22+
height="315"
23+
src="https://www.youtube.com/embed/UI9hKzP17LE"
24+
frameborder="0"
25+
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
26+
allowfullscreen
27+
></iframe>
28+
29+
## 根本原因分析
30+
31+
### 問題根源
32+
33+
Balance Server 上的 Nginx 使用了以下負載平衡設定:
34+
35+
```bash
36+
upstream core_servers {
37+
server tnn.exptech.dev:80;
38+
server tpe.exptech.dev:80;
39+
40+
hash $remote_addr consistent;
41+
}
42+
```
43+
44+
### 問題說明
45+
46+
`hash $remote_addr consistent;` 會根據用戶端的 IP 位址進行一致性雜湊(Consistent Hashing),將來自同一 IP 的請求分配到特定的 Core Server。
47+
48+
**問題點:**
49+
50+
- 當網路不穩定時,用戶端 IP 可能發生變化
51+
- 導致資料來源在兩個 upstream(`tnn.exptech.dev``tpe.exptech.dev`)之間來回切換
52+
- 造成資料不一致,進而引發 TREM Lite 偵測重複觸發的問題
53+
54+
## 解決方案
55+
56+
### 修改後的設定
57+
58+
將 Nginx upstream 設定改為以下配置:
59+
60+
```bash
61+
upstream core_servers {
62+
server tnn.exptech.dev:80 max_fails=1 fail_timeout=10s;
63+
server tpe.exptech.dev:80 backup;
64+
}
65+
```
66+
67+
### 改進說明
68+
69+
1. **主伺服器設定:** `tnn.exptech.dev` 設為主伺服器,並設定故障檢測參數
70+
71+
- `max_fails=1`:允許最多 1 次失敗
72+
- `fail_timeout=10s`:故障後 10 秒內不再嘗試該伺服器
73+
74+
2. **備援伺服器設定:** `tpe.exptech.dev` 設為備援伺服器(backup)
75+
76+
- 僅在主伺服器不可用時才會使用
77+
78+
3. **移除一致性雜湊:** 避免因 IP 變化導致的資料來源切換問題
79+
80+
## 後續觀察
81+
82+
經觀察後,故障應該已經排除。建議持續監控系統狀態,確保問題不再發生。

blog/Timyaya8732/2024-06-15-timyaya8732.mdx

Lines changed: 0 additions & 165 deletions
This file was deleted.

blog/authors.yml

Lines changed: 0 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -3,21 +3,3 @@ yuyu1015:
33
title: Full-Stack Engineer (全端工程師)
44
url: https://github.com/whes1015
55
image_url: https://github.com/whes1015.png
6-
7-
yoyo0901:
8-
name: Yoyo0901
9-
title: Flutter Engineer (Flutter工程師)
10-
url: https://github.com/Yoyochou0901
11-
image_url: https://github.com/Yoyochou0901.png
12-
13-
cstrikeasia:
14-
name: Miyashooooo
15-
title: Front-End Engineer (前端工程師)
16-
url: https://github.com/cstrikeasia
17-
image_url: https://github.com/cstrikeasia.png
18-
19-
timyaya8732:
20-
name: Timyaya8732
21-
title: Data Analyst Engineer (資料分析工程師)
22-
url: https://github.com/timyaya
23-
image_url: https://github.com/timyaya.png

blog/tags.yml

Lines changed: 4 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -3,47 +3,7 @@ yuyu1015:
33
permalink: /yuyu1015
44
description: YuYu1015
55

6-
yoyo0901:
7-
label: Yoyo0901
8-
permalink: /yoyo0901
9-
description: Yoyo0901
10-
11-
cstrikeasia:
12-
label: Miyashooooo
13-
permalink: /cstrikeasia
14-
description: Miyashooooo
15-
16-
timyaya8732:
17-
label: Timyaya8732
18-
permalink: /timyaya8732
19-
description: Timyaya8732
20-
21-
ios:
22-
label: iOS
23-
permalink: /ios
24-
description: iOS
25-
26-
android:
27-
label: Android
28-
permalink: /android
29-
description: Android
30-
31-
swift:
32-
label: Swift
33-
permalink: /swift
34-
description: Swift
35-
36-
flutter:
37-
label: Flutter
38-
permalink: /flutter
39-
description: Flutter
40-
41-
sandboxie:
42-
label: Sandboxie
43-
permalink: /sandboxie
44-
description: sandboxie
45-
46-
tremv:
47-
label: TREMV
48-
permalink: /tremv
49-
description: tremv
6+
rca:
7+
label: RCA
8+
permalink: /rca
9+
description: RCA
Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
import { JSX } from "react";
2+
import styles from "./styles.module.css";
3+
4+
type SeverityLevel = "critical" | "high" | "medium" | "low" | "info";
5+
6+
type SeverityBadgeProps = {
7+
level: SeverityLevel;
8+
title?: string;
9+
children?: string;
10+
}
11+
12+
const severityLabels: Record<SeverityLevel, string> = {
13+
critical: "嚴重",
14+
high: "高",
15+
medium: "中",
16+
low: "低",
17+
info: "資訊"
18+
};
19+
20+
export default function SeverityBadge({
21+
level,
22+
title = "嚴重程度",
23+
children
24+
}: SeverityBadgeProps): JSX.Element {
25+
const label = children || severityLabels[level];
26+
27+
return (
28+
<span className={`${styles.badge} ${styles[level]}`}>
29+
<span className={styles.indicator} />
30+
<span className={styles.title}>{title}</span>
31+
<span className={styles.divider} />
32+
<span className={styles.label}>{label}</span>
33+
</span>
34+
);
35+
}

0 commit comments

Comments
 (0)