Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
ad7ee94
Update version to 1.17.0
casionone Oct 23, 2025
cd48479
Add AI-assisted docs
kinsanities Oct 27, 2025
7f53d68
add ai rules
kinsanities Oct 27, 2025
ddc6390
add ai rules
kinsanities Oct 27, 2025
a8e121c
optimization ai rules
kinsanities Oct 28, 2025
3196c34
remove print execute_code log (#874)
aiceflower Oct 31, 2025
315147b
nodeexecution support aisql (#876)
aiceflower Oct 31, 2025
6c7af81
support starrocks task for aisql (#875)
aiceflower Oct 31, 2025
8196f6f
Feature/1.17.0 resultset sensitive field masking (#873)
v-kkhuang Nov 3, 2025
65653cd
update failed to create ec log to warn
kinsanities Nov 3, 2025
2af3ad9
添加根据数据源类型和代理用户查询发布可用数据源的接口及RPC实现 (#877)
v-kkhuang Nov 4, 2025
bd13150
add sr logic (#879)
aiceflower Nov 4, 2025
88858c0
Feature/1.17.0 resultset sensitive field masking (#878)
v-kkhuang Nov 4, 2025
c81e908
Dev 1.17.0 webank sr (#881)
casionone Nov 5, 2025
4b92111
Feature/1.17.0 resultset sensitive field masking (#882)
v-kkhuang Nov 5, 2025
9fa6464
bes sup (#884)
v-kkhuang Nov 7, 2025
46f8de3
code optimization (#883)
v-kkhuang Nov 7, 2025
3807c1d
Dev 1.17.0 code review fix (#885)
v-kkhuang Nov 11, 2025
588cf44
add runtime param code optimization
kinsanities Nov 11, 2025
2538c3b
remove bes dependency
kinsanities Nov 11, 2025
3c9e7f0
update netty version from 4.2.4 to 4.2.7 (#887)
casionone Nov 12, 2025
8cddb98
Dev 1.17.0 webank srl (#888)
casionone Nov 13, 2025
3702c15
Feature/1.17.0 resultset sensitive field masking (#889)
v-kkhuang Nov 17, 2025
e4ef0c0
template support multi creator
kinsanities Nov 18, 2025
303f6fb
Feature/1.17.0 resultset sensitive field masking (#890)
v-kkhuang Nov 19, 2025
e270c1e
Dev 1.17.0 code review fix (#891)
v-kkhuang Nov 22, 2025
442da6c
aisql whitelist logic fix
casionone Nov 25, 2025
0e9b8ce
log optimization
casionone Nov 26, 2025
e6eb0ca
fix hive ec multi task conf mix issue
kinsanities Nov 27, 2025
b8367ce
Dev 1.17.0 code review fix (#894)
v-kkhuang Nov 28, 2025
fd0d03e
task log optimization
casionone Nov 28, 2025
1aa315d
update version to 1.18.0
kinghao007 Dec 12, 2025
6063b1d
fix: 优化datasource/columns接口字段排序,确保与表定义一致
v-kkhuang Dec 18, 2025
f59eda7
Revert "fix: 优化datasource/columns接口字段排序,确保与表定义一致"
v-kkhuang Dec 18, 2025
cc2bc5e
【1.18.0】 Support finer log level filtering for openLog interface (#898)
v-kkhuang Dec 23, 2025
e4e34ae
【1.18.0】fix: forbid hadoop user login in web interface (#899)
v-kkhuang Dec 23, 2025
383ea30
【1.18.0】Optimize the result set display (#900)
v-kkhuang Dec 25, 2025
1152d81
【1.18.0】Simplify dealsparkDynamicConf method, only force set spark.py…
v-kkhuang Dec 25, 2025
2b61102
Enhance JDBCEngineConnExecutor: add parseJdbcUrl method to handle URL…
v-kkhuang Dec 25, 2025
b859b9d
【1.18.0】Development of In-Mission Diagnosis for Task Timeout (#904)
v-kkhuang Dec 25, 2025
5d9c6af
【1.18.0】StarrocksTask failure retry interception (#906)
v-kkhuang Dec 25, 2025
7289534
feat: support disallow login (#905)
daflyinbed Dec 25, 2025
32a0e0b
fix: disallow login header (#912)
daflyinbed Dec 26, 2025
995cc42
code optimization (#916)
v-kkhuang Dec 29, 2025
9fadf93
Dev 1.18.0 code review fix (#919)
v-kkhuang Jan 5, 2026
3163089
fix(httpclient): 调整连接池参数设置
Jul 23, 2025
e622651
Add pendingCount BML client connection information retrieval function…
Jan 5, 2026
10373f3
Dev 1.18.0 code review fix (#922)
v-kkhuang Jan 7, 2026
e7b478d
Dev 1.18.0 code review fix (#923)
v-kkhuang Jan 9, 2026
2053b1c
Dev 1.18.0 code review fix (#924)
v-kkhuang Jan 12, 2026
a7f2dbd
Dev 1.18.0 code review fix (#925)
v-kkhuang Jan 13, 2026
f7b2f35
Dev 1.18.0 code review fix (#926)
v-kkhuang Jan 16, 2026
7e48d07
Dev 1.18.0 code review fix (#927)
v-kkhuang Jan 20, 2026
48a1389
Dev 1.18.0 code review fix (#928)
v-kkhuang Jan 21, 2026
59bb6d5
Dev 1.18.0 code review fix (#929)
v-kkhuang Jan 23, 2026
629c221
Dev 1.18.0 code review fix (#930)
v-kkhuang Jan 26, 2026
956d09d
docs optimization
aiceflower Jan 26, 2026
41da777
Dev 1.18.0 code review fix (#931)
v-kkhuang Jan 28, 2026
17f486f
optimization log
aiceflower Jan 28, 2026
efe408b
#AI commit#yarn接口新增参数过滤 (#932)
v-kkhuang Feb 2, 2026
a5de797
#AI commit# 开发阶段:新增功能 - 添加批量获取队列资源功能支持 (#935)
v-kkhuang Feb 4, 2026
c824467
update version to 1.18.1
casionone Mar 12, 2026
65cb0c2
#AI commit# 开发阶段:管理台支持展示引擎版本 (#939)
v-kkhuang Mar 13, 2026
c54650b
[feature][linkis][engineplugin]the spark engine supports setting exec…
v-kkhuang Mar 13, 2026
f5f557f
#AI commit# 开发阶段:增加日志打印 (#943)
v-kkhuang Mar 17, 2026
424bc67
#AI commit# 开发阶段:提交全局历史页面引擎版本展示agent 相关文档 (#942)
v-kkhuang Mar 17, 2026
088edfe
[linkis][spark]optimize Spark parameter acquisition (#944)
v-kkhuang Mar 17, 2026
6bb48ee
Update version to 1.18.0
casionone Mar 17, 2026
b146736
#AI commit# 开发阶段: Hive任务禁止使用LOCATION功能
v-kkhuang Mar 26, 2026
3257fdd
#AI commit# 开发阶段: Hive任务禁止使用LOCATION功能,测试报告相关文件提交
v-kkhuang Mar 27, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
188 changes: 188 additions & 0 deletions .ai/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,188 @@
# Linkis AI 开发文档导航

> **版本信息**
> - 文档版本: 1.0.0
> - 最后更新: 2025-01-28
> - 适用版本: Apache Linkis 1.17.0+

---

## 🚀 快速开始

### 新手必读(按顺序阅读)
1. **[项目核心规约](./project-context.md)** - 包含技术栈、架构设计、开发规范和模板
2. **[强制性开发规则](./rules.md)** - 必须无条件遵守的开发规则
3. **[模块文档](#模块文档索引)** - 根据你要开发的功能选择对应模块

### 常见开发场景快速跳转
- 🆕 新增 REST 接口 → [REST接口开发模板](#rest接口开发)
- ⚙️ 添加配置项 → [配置管理规范](#配置管理)
- 🗄️ 修改数据库 → [数据库变更规范](#数据库变更)
- 🐛 异常处理 → [异常处理规范](#异常处理)
- 📝 日志记录 → [日志规范](#日志规范)

---

## 📚 核心文档索引

### 🎯 开发规范文档
| 文档 | 用途 | 何时查看 |
|------|------|----------|
| [project-context.md](./project-context.md) | 项目角色定位、技术栈、架构设计、开发模板 | 开始任何开发工作前必读 |
| [rules.md](./rules.md) | 强制性开发规则、需求实现步骤 | 每次开发新需求时参考 |

### 🏗️ 模块文档索引

#### 微服务治理服务(基础设施层)
| 服务 | 文档 | 主要功能 |
|------|------|----------|
| Gateway | [gateway.md](./modules/microservice-governance/gateway.md) | API网关、路由转发、安全认证 |
| Eureka | [eureka.md](./modules/microservice-governance/eureka.md) | 服务注册与发现 |
| 概览 | [README.md](./modules/microservice-governance/README.md) | 微服务治理服务概述 |

#### 计算治理服务(核心业务层)
| 服务 | 文档 | 主要功能 |
|------|------|----------|
| Entrance | [entrance.md](./modules/computation-governance/entrance.md) | 任务提交入口、调度管理 |
| JobHistory | [jobhistory.md](./modules/computation-governance/jobhistory.md) | 任务历史记录查询 |
| Manager | [manager.md](./modules/computation-governance/manager.md) | 资源管理、应用管理 |
| ECM | [ecm.md](./modules/computation-governance/ecm.md) | 引擎连接管理 |
| 概览 | [README.md](./modules/computation-governance/README.md) | 计算治理服务概述 |

#### 公共增强服务(支撑服务层)
| 服务 | 文档 | 主要功能 |
|------|------|----------|
| PublicService | [publicservice.md](./modules/public-enhancements/publicservice.md) | 公共服务、文件管理 |
| Configuration | [configuration.md](./modules/public-enhancements/configuration.md) | 配置管理 |
| BML | [bml.md](./modules/public-enhancements/bml.md) | 大数据物料库 |
| DataSource | [datasource.md](./modules/public-enhancements/datasource.md) | 数据源管理 |
| Context | [context.md](./modules/public-enhancements/context.md) | 上下文服务 |
| Monitor | [monitor.md](./modules/public-enhancements/monitor.md) | 监控服务 |
| 概览 | [README.md](./modules/public-enhancements/README.md) | 公共增强服务概述 |

---

## 🔍 按功能快速查找

### <a name="rest接口开发"></a>REST接口开发
- **开发模板**: [project-context.md - REST接口层](./project-context.md#1-rest接口层)
- **API规范**: [project-context.md - API设计规范](./project-context.md#api设计规范)
- **参考示例**:
- Entrance接口: [entrance.md - API Interfaces](./modules/computation-governance/entrance.md#api-interfaces)
- Configuration接口: [configuration.md - API Interfaces](./modules/public-enhancements/configuration.md#api-interfaces)

### <a name="配置管理"></a>配置管理
- **配置规范**: [project-context.md - 配置管理规范](./project-context.md#配置管理规范)
- **配置示例库**: [project-context.md - 常用配置示例库](./project-context.md#常用配置示例库)
- **配置模板**: [project-context.md - 配置类](./project-context.md#4-配置类)
- **参考实现**: linkis-jobhistory/conf/JobhistoryConfiguration

### <a name="数据库变更"></a>数据库变更
- **变更规则**: [rules.md - 数据库修改原则](./rules.md#数据库修改原则)
- **DDL脚本位置**: `linkis-dist/package/db/linkis_ddl.sql`
- **DML脚本位置**: `linkis-dist/package/db/linkis_dml.sql`
- **表结构参考**: 各模块文档的 "Database Table Structures" 章节

### <a name="异常处理"></a>异常处理
- **异常规范**: [project-context.md - 异常处理规范](./project-context.md#异常处理规范)
- **统一异常**: `org.apache.linkis.common.exception.LinkisException`
- **常见错误**: [project-context.md - 常见错误及避免方法](./project-context.md#常见错误及避免方法)

### <a name="日志规范"></a>日志规范
- **日志规范**: [project-context.md - 日志规范](./project-context.md#日志规范)
- **Logger定义**: 必须使用 `LoggerFactory.getLogger(ClassName.class)`
- **日志级别**: ERROR/WARN/INFO/DEBUG 使用场景

---

## 🎨 开发模板快速复制

### 新增功能完整流程
```
1. 查看 rules.md - 需求实现步骤
2. 创建需求文档和设计文档
3. 使用 project-context.md 中的代码模板:
- REST接口层模板
- 服务层模板
- 数据访问层模板
- 配置类模板
4. 添加功能开关(默认false)
5. 记录数据库变更
6. 编写测试和文档
```

### REST接口模板快速链接
👉 [project-context.md - 新功能开发模板](./project-context.md#新功能开发模板)

### 配置类模板快速链接
👉 [project-context.md - 配置类](./project-context.md#4-配置类)

---

## ⚠️ 重要提醒

### 🚫 禁止操作(来自 rules.md)
- **数据库结构**: 除非明确指定,严禁修改现有表结构
- **第三方依赖**: 不允许引入新的第三方依赖库
- **核心接口**: 不得修改现有公共接口的签名

### ✅ 必须遵守
- **最小改动原则**: 所有功能实现必须遵循最小改动原则
- **功能可配置**: 所有功能必须增加功能开关,默认关闭
- **向后兼容**: 新增功能必须考虑向后兼容性

---

## 💡 开发技巧

### 编程语言选择
- **Java**: REST API、Service层、Entity类、配置类
- **Scala**: 计算逻辑、RPC通信、复杂业务处理、配置对象

### 字符编码
统一使用 `StandardCharsets.UTF_8`,禁止使用字符串 `"UTF-8"`

### 统一返回体
所有REST接口返回 `org.apache.linkis.server.Message`

---

## 📖 如何使用这些文档

### 场景1: 我要在 Entrance 服务中新增一个接口
1. 阅读 [entrance.md](./modules/computation-governance/entrance.md) 了解服务结构
2. 查看 [project-context.md - REST接口层模板](./project-context.md#1-rest接口层)
3. 参考 entrance.md 中现有接口实现
4. 遵循 [rules.md](./rules.md) 中的开发规则
5. 添加功能开关配置

### 场景2: 我需要添加一个新的配置项
1. 查看 [project-context.md - 配置管理规范](./project-context.md#配置管理规范)
2. 参考 [project-context.md - 配置类模板](./project-context.md#4-配置类)
3. 查看 `JobhistoryConfiguration` 实现示例
4. 在当前模块的 conf 目录下的 Configuration 类中添加

### 场景3: 我需要修改数据库表
1. 查看 [rules.md - 数据库修改原则](./rules.md#数据库修改原则)
2. 确认是否能通过新增字段实现(优先选择)
3. 将变更记录到 `linkis-dist/package/db/linkis_ddl.sql`
4. 如有初始化数据,记录到 `linkis-dist/package/db/linkis_dml.sql`

---

## 🔄 文档更新记录

| 版本 | 日期 | 更新内容 | 更新人 |
|------|------|----------|--------|
| 1.0.0 | 2025-01-28 | 创建导航文档,优化文档结构 | AI |

---

## 📞 帮助与反馈

如果文档中有不清楚的地方,请:
1. 先查看对应模块的详细文档
2. 查看 project-context.md 中的开发模板和示例
3. 参考现有代码实现

**记住**: 遵循规范比快速开发更重要!
157 changes: 157 additions & 0 deletions .ai/modules/computation-governance/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
# Computation Governance Services

The computation governance services handle the core computation task lifecycle management in Linkis.

## Service Modules

- [Entrance Service](./entrance.md) - Task submission and entrance point
- [Manager Service](./manager.md) - Resource and application management
- [ECM Service](./ecm.md) - Engine Connection Manager
- [JobHistory Service](./jobhistory.md) - Task execution history tracking

## Overview

These services form the core of Linkis' computation governance capabilities, managing the complete lifecycle of computation tasks from submission to execution and monitoring.

## Common Features

### Task Lifecycle Management
- Task submission and validation
- Task scheduling and resource allocation
- Task execution monitoring
- Task result management
- Task error handling and recovery

### Engine Management
- Dynamic engine connection creation
- Engine lifecycle management
- Engine resource monitoring
- Engine scaling capabilities

### Resource Governance
- Multi-tenant resource isolation
- Load balancing across engines
- Resource usage tracking
- Quota management

## API Interface Summary

### Entrance Service APIs
- Task submission: `POST /api/entrance/submit`
- Task status query: `GET /api/entrance/{id}/status`
- Task progress: `GET /api/entrance/{id}/progress`
- Task log retrieval: `GET /api/entrance/{id}/log`
- Task cancellation: `GET /api/entrance/{id}/kill`

### Manager Service APIs
- Engine instance management
- Resource allocation and monitoring
- Node status querying
- Engine creation requests

### ECM Service APIs
- Engine connection management
- Engine lifecycle operations
- Resource reporting
- Engine metrics collection

### JobHistory Service APIs
- Job history querying
- Job detail retrieval
- Job statistics reporting

## Database Schema Summary

### Job History Group Table
```sql
CREATE TABLE `linkis_ps_job_history_group_history` (
`id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment',
`job_req_id` varchar(64) DEFAULT NULL COMMENT 'job execId',
`submit_user` varchar(50) DEFAULT NULL COMMENT 'who submitted this Job',
`execute_user` varchar(50) DEFAULT NULL COMMENT 'who actually executed this Job',
`source` text DEFAULT NULL COMMENT 'job source',
`labels` text DEFAULT NULL COMMENT 'job labels',
`params` text DEFAULT NULL COMMENT 'job params',
`progress` varchar(32) DEFAULT NULL COMMENT 'Job execution progress',
`status` varchar(50) DEFAULT NULL COMMENT 'Script execution status, must be one of the following: Inited, WaitForRetry, Scheduled, Running, Succeed, Failed, Cancelled, Timeout',
`log_path` varchar(200) DEFAULT NULL COMMENT 'File path of the job log',
`error_code` int DEFAULT NULL COMMENT 'Error code. Generated when the execution of the script fails',
`error_desc` varchar(1000) DEFAULT NULL COMMENT 'Execution description. Generated when the execution of script fails',
`created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time',
`updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time',
`instances` varchar(250) DEFAULT NULL COMMENT 'Entrance instances',
`metrics` text DEFAULT NULL COMMENT 'Job Metrics',
`engine_type` varchar(32) DEFAULT NULL COMMENT 'Engine type',
`execution_code` text DEFAULT NULL COMMENT 'Job origin code or code path',
`result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets',
`observe_info` varchar(500) DEFAULT NULL COMMENT 'The notification information configuration of this job',
PRIMARY KEY (`id`),
KEY `idx_created_time` (`created_time`),
KEY `idx_submit_user` (`submit_user`)
);
```

### Job History Detail Table
```sql
CREATE TABLE `linkis_ps_job_history_detail` (
`id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key, auto increment',
`job_history_id` bigint(20) NOT NULL COMMENT 'ID of JobHistory',
`result_location` varchar(500) DEFAULT NULL COMMENT 'File path of the resultsets',
`execution_content` text DEFAULT NULL COMMENT 'The script code or other execution content executed by this Job',
`result_array_size` int(4) DEFAULT 0 COMMENT 'size of result array',
`job_group_info` text DEFAULT NULL COMMENT 'Job group info/path',
`created_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Creation time',
`updated_time` datetime(3) DEFAULT CURRENT_TIMESTAMP(3) COMMENT 'Update time',
`status` varchar(32) DEFAULT NULL COMMENT 'status',
`priority` int(4) DEFAULT 0 COMMENT 'order of subjob',
PRIMARY KEY (`id`)
);
```

### Common Lock Table
```sql
CREATE TABLE `linkis_ps_common_lock` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`lock_object` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`locker` VARCHAR(255) CHARSET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL COMMENT 'locker',
`time_out` longtext COLLATE utf8_bin,
`update_time` datetime DEFAULT CURRENT_TIMESTAMP,
`create_time` datetime DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
UNIQUE KEY `uniq_lock_object` (`lock_object`)
);
```

## RPC Methods Summary

### Entrance Service RPCs
- `submitTask(TaskRequest request)`
- `getTaskStatus(String taskId)`
- `cancelTask(String taskId)`
- `getTaskResult(String taskId)`

### Manager Service RPCs
- `requestEngine(EngineRequest request)`
- `releaseEngine(String engineId)`
- `getEngineStatus(String engineId)`
- `getNodeMetrics(String nodeId)`

### ECM Service RPCs
- `createEngineConnection(EngineCreateRequest request)`
- `terminateEngineConnection(String engineId)`
- `reportEngineResourceUsage(String engineId, ResourceUsage usage)`
- `getEngineMetrics(String engineId)`

### JobHistory Service RPCs
- `saveJobHistory(JobHistory history)`
- `queryJobHistory(JobHistoryQuery query)`
- `getJobDetails(Long jobId)`
- `updateJobStatus(Long jobId, String status)`

## Dependencies

- linkis-commons - Shared utilities
- linkis-protocol - Communication protocols
- linkis-rpc - Remote procedure calls
- Various engine connection plugins
- Spring Cloud ecosystem
Loading
Loading