Compare commits
8 Commits
aaeefaed6b
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
7998350e70
|
|||
|
7dfbe907af
|
|||
|
ef9950a9d3
|
|||
|
dd3f6b4573
|
|||
|
ab1e0875a2
|
|||
|
c9a85759fb
|
|||
|
cad2c583bf
|
|||
|
5d72aa3893
|
2
.gitignore
vendored
2
.gitignore
vendored
@ -1,6 +1,6 @@
|
|||||||
# Environment
|
# Environment
|
||||||
.env
|
.env
|
||||||
.env.example
|
# .env.example is tracked for reference
|
||||||
|
|
||||||
# Python
|
# Python
|
||||||
__pycache__/
|
__pycache__/
|
||||||
|
|||||||
@ -1,66 +0,0 @@
|
|||||||
请在 `./scripts/` 文件夹下写一个 `img2typ.py` 脚本,达成下面的要求。
|
|
||||||
|
|
||||||
## 流程
|
|
||||||
|
|
||||||
遍历当前所在目录下的文件(不深入到子目录)。
|
|
||||||
|
|
||||||
找到所有的 `\S\s?[\d\.]+` 的文件名的图片。且最开头的 `\S` 不能是 `答` 或者 `A` 或者 `a`
|
|
||||||
|
|
||||||
- 这里只是大概的描述,你可能需要调整正则,或者不使用正则
|
|
||||||
- 例如,`问 13.png` `Q3.1.jpg` `R1.1.5.PNG` 等等的文件名都是合法的
|
|
||||||
|
|
||||||
检测有没有对应的 `.typ` 文件。即,看有没有后缀名改成 `typ` 的文件。
|
|
||||||
|
|
||||||
如果没有,则用 `./scripts/img2typ.prompt.txt` 为提示词,调用一个支持图片的 OpenAI 兼容 API。这个提示词文件在未来会改动,请在程序执行时动态读取它。API 的 API 端点和 API Key 应该使用一个 `.env` 文件定义,这个文件将会置放在 `./scripts/` 文件夹下。
|
|
||||||
|
|
||||||
对输出结果,如果有 Markdown 代码块包裹,则去除(可能需要你写正则或者其他任何机制)
|
|
||||||
|
|
||||||
接着,保存为对应的 `.typ` 文件。
|
|
||||||
|
|
||||||
最后,整理所有符合条件的 `.typ` 文件(包括生成失败的),整理成一个列表后,在当前根目录写入(可覆盖)`questions.json`,是一个 JSON 列表,列表的每一个项目都是一个 JSON Object。形如:
|
|
||||||
|
|
||||||
```json
|
|
||||||
[
|
|
||||||
{
|
|
||||||
"question": "Q3.1",
|
|
||||||
"format": "typst",
|
|
||||||
"target": "Q3.1.typ"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"question": "R1.1.5",
|
|
||||||
"format": "typst",
|
|
||||||
"target": "R1.1.5.typ"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
```
|
|
||||||
|
|
||||||
## 细节
|
|
||||||
|
|
||||||
调用 API 时,如果失败,则重试。最多重试 3 次(或者可定义),真的失败了不能 panic,只是在 stderr 中汇报。
|
|
||||||
|
|
||||||
## 规范
|
|
||||||
|
|
||||||
代码应该是人类可维护的。你可以(而且最好)使用 python 的比较常用的现代语法,例如直接的类型注解(使用 `list` 而不是 `typing.List`),以及尽量使用 `pathlib.Path` 而不是字符串。
|
|
||||||
|
|
||||||
这个应用是面向过程的。你应该首先对流程做拆解,然后以子函数的形式声明整个函数。每个函数都应该有 docstring。对于 AI 等一些比较重要的东西,你再使用面向对象的方式去应对。
|
|
||||||
|
|
||||||
你可以给我要求,让我依赖一些更多的外置库。当前环境有 `requests`、`dotenv`、`rich`、`tqdm` 可用。
|
|
||||||
|
|
||||||
程序往 `stderr` 的输出应该是可审计的。应该汇报:
|
|
||||||
|
|
||||||
- 将要处理的文件清单。
|
|
||||||
- 调用了什么 API,调用情况如何,输入输出多少 tokens。
|
|
||||||
- 写入了什么文件。
|
|
||||||
|
|
||||||
程序往 `stderr` 的输出应该是良好可视化的,就是说,有颜色区分,但是不要加 emoji。或者说,你应该使用自带的日志库 + 一定的格式化。
|
|
||||||
|
|
||||||
## 额外功能添补
|
|
||||||
|
|
||||||
这个脚本应该可以作为 cli 调用,支持以下参数:
|
|
||||||
|
|
||||||
- `--file` 或 `-f` 后接文件名,可重复这个参数。当存在这个参数,则解析对应的图片,而不是扫描当前目录
|
|
||||||
- `--dry-run` 不调用 AI,也不写入文件
|
|
||||||
- `--verbose` 在基础上反馈 AI 调用的输出,相当于 log level 是 `DEBUG`
|
|
||||||
- `--retry` 接数字,重试次数,默认为 3
|
|
||||||
- `-n` 接数字,并发数量,默认为 3
|
|
||||||
|
|
||||||
134
README.md
Normal file
134
README.md
Normal file
@ -0,0 +1,134 @@
|
|||||||
|
# phomework - 作业模板系统
|
||||||
|
|
||||||
|
使用 Typst 和 AI 辅助生成作业的自动化工具。
|
||||||
|
|
||||||
|
## 目录结构
|
||||||
|
|
||||||
|
```
|
||||||
|
├── data/ # 作业数据(图片、题目、答案)
|
||||||
|
├── templates/ # Typst 模板文件
|
||||||
|
├── scripts/ # Python 脚本
|
||||||
|
│ ├── img2typ.py # 图片转 typst 格式
|
||||||
|
│ ├── fix_typ.py # 修复 typst 语法
|
||||||
|
│ ├── solve.py # AI 答题
|
||||||
|
│ ├── gen_index.py # 生成 index.typ
|
||||||
|
│ └── common.py # 共用模块
|
||||||
|
├── index.typ # 生成的作业文件
|
||||||
|
└── index.pdf # 编译后的 PDF
|
||||||
|
```
|
||||||
|
|
||||||
|
## 工作流程
|
||||||
|
|
||||||
|
1. **图片转题目** (`just img2typ`)
|
||||||
|
- 扫描 `data/` 中的图片
|
||||||
|
- 使用 AI 将图片内容转换为 typst 格式
|
||||||
|
- 生成 `questions.json` 题目列表
|
||||||
|
|
||||||
|
2. **AI 答题** (`just solve`)
|
||||||
|
- 读取 `questions.json`
|
||||||
|
- 调用 AI 生成答案
|
||||||
|
- 输出 `data/A_*.md` 答案文件
|
||||||
|
|
||||||
|
3. **生成作业** (`just generate`)
|
||||||
|
- 读取题目和答案
|
||||||
|
- 生成 `index.typ`
|
||||||
|
- 可编译为 PDF
|
||||||
|
|
||||||
|
## 快速开始
|
||||||
|
|
||||||
|
### 1. 准备数据
|
||||||
|
|
||||||
|
将作业图片放入 `data/` 目录,文件名格式:
|
||||||
|
- `P15.png` - P15 题目
|
||||||
|
- `R7.png` - R7 题目
|
||||||
|
- `P40_img1.png` - P40 的附件
|
||||||
|
|
||||||
|
### 2. 配置
|
||||||
|
|
||||||
|
API 配置从以下位置按优先级读取:
|
||||||
|
1. `scripts/.env` (本地)
|
||||||
|
2. `~/.config/phomework/.env` (全局)
|
||||||
|
|
||||||
|
**方式一:全局配置(推荐)**
|
||||||
|
```bash
|
||||||
|
mkdir -p ~/.config/phomework
|
||||||
|
cp scripts/.env.example ~/.config/phomework/.env
|
||||||
|
# 编辑 ~/.config/phomework/.env
|
||||||
|
```
|
||||||
|
|
||||||
|
**方式二:本地配置**
|
||||||
|
```bash
|
||||||
|
cp scripts/.env.example scripts/.env
|
||||||
|
# 编辑 scripts/.env
|
||||||
|
```
|
||||||
|
|
||||||
|
`.env` 文件内容:
|
||||||
|
```bash
|
||||||
|
# API 端点(OpenAI 兼容)
|
||||||
|
IMG2TYP_API_ENDPOINT=https://api.openai.com/v1/chat/completions
|
||||||
|
|
||||||
|
# API 密钥
|
||||||
|
IMG2TYP_API_KEY=your-api-key-here
|
||||||
|
|
||||||
|
# 模型名称(默认 qwen-vl-plus)
|
||||||
|
IMG2TYP_MODEL=qwen-vl-plus
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. 运行
|
||||||
|
|
||||||
|
```bash
|
||||||
|
just img2typ # 图片转题目
|
||||||
|
just solve # AI 答题
|
||||||
|
just generate # 生成 index.typ
|
||||||
|
typst compile index.typ index.pdf
|
||||||
|
```
|
||||||
|
|
||||||
|
如需修复 .typ 文件语法(可选):
|
||||||
|
```bash
|
||||||
|
just fix
|
||||||
|
```
|
||||||
|
|
||||||
|
## 命令行参数
|
||||||
|
|
||||||
|
### fix_typ.py
|
||||||
|
```bash
|
||||||
|
-f, --file # 指定单个 .typ 文件
|
||||||
|
--dry-run # 干跑,不调用 AI 或写入文件
|
||||||
|
--verbose # 调试日志
|
||||||
|
-n N # 并发数(默认 3)
|
||||||
|
```
|
||||||
|
|
||||||
|
### img2typ.py
|
||||||
|
```bash
|
||||||
|
-f, --file # 指定单个图片文件
|
||||||
|
--dry-run # 干跑,不调用 AI 或写入文件
|
||||||
|
--verbose # 调试日志
|
||||||
|
--retry N # 重试次数(默认 3)
|
||||||
|
-n N # 并发数(默认 3)
|
||||||
|
```
|
||||||
|
|
||||||
|
### solve.py
|
||||||
|
```bash
|
||||||
|
-q, --question # 指定题目 ID
|
||||||
|
--dry-run # 干跑
|
||||||
|
--verbose # 调试日志
|
||||||
|
--retry N # 重试次数(默认 3)
|
||||||
|
-n N # 并发数(默认 3)
|
||||||
|
```
|
||||||
|
|
||||||
|
### gen_index.py
|
||||||
|
```bash
|
||||||
|
--dry-run # 预览生成内容
|
||||||
|
--force # 强制覆盖 index.typ
|
||||||
|
```
|
||||||
|
|
||||||
|
## 开发
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 安装依赖
|
||||||
|
pip install requests python-dotenv rich aiohttp
|
||||||
|
|
||||||
|
# 代码检查
|
||||||
|
ruff check scripts/
|
||||||
|
ruff format scripts/
|
||||||
|
```
|
||||||
3
justfile
3
justfile
@ -1,3 +1,6 @@
|
|||||||
|
fix:
|
||||||
|
python ./scripts/fix_typ.py
|
||||||
|
|
||||||
img2typ:
|
img2typ:
|
||||||
python ./scripts/img2typ.py
|
python ./scripts/img2typ.py
|
||||||
|
|
||||||
|
|||||||
8
scripts/.env.example
Normal file
8
scripts/.env.example
Normal file
@ -0,0 +1,8 @@
|
|||||||
|
# OpenAI-compatible API endpoint
|
||||||
|
IMG2TYP_API_ENDPOINT=https://api.openai.com/v1/chat/completions
|
||||||
|
|
||||||
|
# API Key for authentication
|
||||||
|
IMG2TYP_API_KEY=your-api-key-here
|
||||||
|
|
||||||
|
# Model name (default: qwen-vl-plus for DashScope)
|
||||||
|
IMG2TYP_MODEL=qwen-vl-plus
|
||||||
@ -15,7 +15,9 @@ console = Console(
|
|||||||
|
|
||||||
SCRIPT_DIR = Path(__file__).parent
|
SCRIPT_DIR = Path(__file__).parent
|
||||||
DATA_DIR = SCRIPT_DIR.parent / "data"
|
DATA_DIR = SCRIPT_DIR.parent / "data"
|
||||||
ENV_FILE = SCRIPT_DIR / ".env"
|
GLOBAL_ENV_DIR = Path.home() / ".config" / "phomework"
|
||||||
|
GLOBAL_ENV_FILE = GLOBAL_ENV_DIR / ".env"
|
||||||
|
LOCAL_ENV_FILE = SCRIPT_DIR / ".env"
|
||||||
|
|
||||||
|
|
||||||
def setup_logging(name: str, verbose: bool) -> logging.Logger:
|
def setup_logging(name: str, verbose: bool) -> logging.Logger:
|
||||||
@ -31,11 +33,23 @@ def setup_logging(name: str, verbose: bool) -> logging.Logger:
|
|||||||
|
|
||||||
|
|
||||||
def load_env() -> dict[str, str]:
|
def load_env() -> dict[str, str]:
|
||||||
"""Load environment variables from .env file."""
|
"""Load environment variables from .env file.
|
||||||
if ENV_FILE.exists():
|
|
||||||
load_dotenv(ENV_FILE)
|
Priority: scripts/.env > ~/.config/phomework/.env
|
||||||
|
"""
|
||||||
|
env_loaded = False
|
||||||
|
|
||||||
|
if LOCAL_ENV_FILE.exists():
|
||||||
|
load_dotenv(LOCAL_ENV_FILE)
|
||||||
|
env_loaded = True
|
||||||
|
elif GLOBAL_ENV_FILE.exists():
|
||||||
|
load_dotenv(GLOBAL_ENV_FILE)
|
||||||
|
env_loaded = True
|
||||||
|
console.print(f"[dim]Using config from {GLOBAL_ENV_FILE}[/dim]")
|
||||||
else:
|
else:
|
||||||
console.print(f"[yellow]Warning: .env file not found at {ENV_FILE}[/yellow]")
|
console.print(
|
||||||
|
f"[yellow]Warning: .env not found at {LOCAL_ENV_FILE} or {GLOBAL_ENV_FILE}[/yellow]"
|
||||||
|
)
|
||||||
|
|
||||||
api_endpoint = os.environ.get("IMG2TYP_API_ENDPOINT", "")
|
api_endpoint = os.environ.get("IMG2TYP_API_ENDPOINT", "")
|
||||||
api_key = os.environ.get("IMG2TYP_API_KEY", "")
|
api_key = os.environ.get("IMG2TYP_API_KEY", "")
|
||||||
|
|||||||
10
scripts/fix_typ.prompt.txt
Normal file
10
scripts/fix_typ.prompt.txt
Normal file
@ -0,0 +1,10 @@
|
|||||||
|
你是一个 Typst 格式专家。请修复提供的 Typst 代码,使其符合 Typst 语法规范。
|
||||||
|
|
||||||
|
常见的修复点:
|
||||||
|
1. **数学表达式语法**:Typst 中数学表达式使用 `$...$` 而不是 `$$...$$` 或 LaTeX 风格。行内数学用 `$...$`,显示数学用 `$ ... $`(两边有空格)。
|
||||||
|
2. **函数调用**:Typst 函数调用使用圆括号,参数用逗号分隔,如 `func(arg1, arg2)`。
|
||||||
|
3. **引号**:Typst 中字符串使用双引号 `"..."`,不是单引号。
|
||||||
|
4. **数组和字典**:Typst 中数组用 `(...)`,字典用 `(...: ...)`。
|
||||||
|
5. **逃逸字符**:Typst 中 `\` 用于转义,不是 `\\`。
|
||||||
|
|
||||||
|
请直接输出修复后的代码,不需要解释。确保输出只包含代码,用 typst 代码块包裹。
|
||||||
268
scripts/fix_typ.py
Normal file
268
scripts/fix_typ.py
Normal file
@ -0,0 +1,268 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
fix_typ.py - Fix Typst files to conform to proper Typst syntax using AI API.
|
||||||
|
|
||||||
|
This script scans the data directory for .typ files, uses AI to fix common
|
||||||
|
syntax issues (e.g., Markdown math syntax), and overwrites the original files.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import asyncio
|
||||||
|
import logging
|
||||||
|
import sys
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import aiohttp
|
||||||
|
from common import DATA_DIR, console, load_env, load_prompt, setup_logging
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class FixResult:
|
||||||
|
"""Result of fixing a typst file."""
|
||||||
|
|
||||||
|
question: str
|
||||||
|
target: str
|
||||||
|
skipped: bool
|
||||||
|
success: bool
|
||||||
|
error: str | None = None
|
||||||
|
|
||||||
|
|
||||||
|
def find_typ_files() -> list[Path]:
|
||||||
|
"""Find all .typ files in data directory."""
|
||||||
|
typ_files = []
|
||||||
|
for file_path in DATA_DIR.iterdir():
|
||||||
|
if file_path.is_file() and file_path.suffix == ".typ":
|
||||||
|
if not file_path.name.startswith("A_"):
|
||||||
|
typ_files.append(file_path)
|
||||||
|
return sorted(typ_files)
|
||||||
|
|
||||||
|
|
||||||
|
async def call_api(
|
||||||
|
session: aiohttp.ClientSession,
|
||||||
|
typ_content: str,
|
||||||
|
question_name: str,
|
||||||
|
endpoint: str,
|
||||||
|
api_key: str,
|
||||||
|
model: str,
|
||||||
|
logger: logging.Logger,
|
||||||
|
) -> str | None:
|
||||||
|
"""Call the AI API to fix typst syntax."""
|
||||||
|
headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
|
||||||
|
|
||||||
|
prompt = load_prompt("fix_typ.prompt.txt")
|
||||||
|
if not prompt:
|
||||||
|
prompt = "请修复以下 Typst 代码的语法错误,使其符合 Typst 规范。特别注意数学表达式不要使用 Markdown/LaTeX 语法。"
|
||||||
|
|
||||||
|
full_prompt = f"""{prompt}
|
||||||
|
|
||||||
|
需要修复的文件内容:
|
||||||
|
```
|
||||||
|
{typ_content}
|
||||||
|
```
|
||||||
|
"""
|
||||||
|
|
||||||
|
payload = {
|
||||||
|
"model": model,
|
||||||
|
"messages": [
|
||||||
|
{"role": "user", "content": [{"type": "text", "text": full_prompt}]}
|
||||||
|
],
|
||||||
|
"max_tokens": 4096,
|
||||||
|
}
|
||||||
|
|
||||||
|
logger.info(f"[{question_name}] Fixing... (timeout 120s)")
|
||||||
|
try:
|
||||||
|
async with session.post(
|
||||||
|
endpoint,
|
||||||
|
headers=headers,
|
||||||
|
json=payload,
|
||||||
|
timeout=aiohttp.ClientTimeout(total=120),
|
||||||
|
) as response:
|
||||||
|
response.raise_for_status()
|
||||||
|
result = await response.json()
|
||||||
|
|
||||||
|
if "choices" not in result or len(result["choices"]) == 0:
|
||||||
|
logger.error(f"Invalid API response")
|
||||||
|
return None
|
||||||
|
|
||||||
|
content = result["choices"][0]["message"]["content"]
|
||||||
|
usage = result.get("usage", {})
|
||||||
|
input_tokens = usage.get("prompt_tokens", 0)
|
||||||
|
output_tokens = usage.get("completion_tokens", 0)
|
||||||
|
|
||||||
|
logger.info(
|
||||||
|
f"[{question_name}] Done: {input_tokens} in, {output_tokens} out"
|
||||||
|
)
|
||||||
|
return content
|
||||||
|
|
||||||
|
except asyncio.TimeoutError:
|
||||||
|
logger.error(f"[{question_name}] Timeout")
|
||||||
|
return None
|
||||||
|
except asyncio.CancelledError:
|
||||||
|
logger.warning(f"[{question_name}] Cancelled")
|
||||||
|
raise
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"[{question_name}] Error: {e}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
async def fix_typ_file(
|
||||||
|
session: aiohttp.ClientSession,
|
||||||
|
typ_path: Path,
|
||||||
|
api_config: dict,
|
||||||
|
logger: logging.Logger,
|
||||||
|
dry_run: bool,
|
||||||
|
) -> FixResult:
|
||||||
|
"""Fix a single .typ file."""
|
||||||
|
question_name = typ_path.stem
|
||||||
|
|
||||||
|
if dry_run:
|
||||||
|
logger.info(f"[{question_name}] Would fix -> {typ_path.name}")
|
||||||
|
return FixResult(
|
||||||
|
question=question_name, target=typ_path.name, skipped=False, success=True
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
typ_content = typ_path.read_text(encoding="utf-8")
|
||||||
|
except IOError as e:
|
||||||
|
logger.error(f"[{question_name}] Read failed: {e}")
|
||||||
|
return FixResult(
|
||||||
|
question=question_name,
|
||||||
|
target=typ_path.name,
|
||||||
|
skipped=False,
|
||||||
|
success=False,
|
||||||
|
error=str(e),
|
||||||
|
)
|
||||||
|
|
||||||
|
fixed_content = await call_api(
|
||||||
|
session,
|
||||||
|
typ_content,
|
||||||
|
question_name,
|
||||||
|
str(api_config["endpoint"]),
|
||||||
|
api_config["key"],
|
||||||
|
api_config["model"],
|
||||||
|
logger,
|
||||||
|
)
|
||||||
|
|
||||||
|
if fixed_content is None:
|
||||||
|
return FixResult(
|
||||||
|
question=question_name,
|
||||||
|
target=typ_path.name,
|
||||||
|
skipped=False,
|
||||||
|
success=False,
|
||||||
|
error="API call failed",
|
||||||
|
)
|
||||||
|
|
||||||
|
import re
|
||||||
|
|
||||||
|
block_pattern = re.compile(r"```(?:typst)?\s*\n?(.*?)\n?```", re.DOTALL)
|
||||||
|
matches = list(block_pattern.finditer(fixed_content))
|
||||||
|
if matches:
|
||||||
|
fixed_content = matches[0].group(1).strip()
|
||||||
|
|
||||||
|
try:
|
||||||
|
typ_path.write_text(fixed_content, encoding="utf-8")
|
||||||
|
logger.info(
|
||||||
|
f"[{question_name}] Wrote {typ_path.name} ({len(fixed_content)} bytes)"
|
||||||
|
)
|
||||||
|
return FixResult(
|
||||||
|
question=question_name, target=typ_path.name, skipped=False, success=True
|
||||||
|
)
|
||||||
|
except IOError as e:
|
||||||
|
logger.error(f"[{question_name}] Write failed: {e}")
|
||||||
|
return FixResult(
|
||||||
|
question=question_name,
|
||||||
|
target=typ_path.name,
|
||||||
|
skipped=False,
|
||||||
|
success=False,
|
||||||
|
error=str(e),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args() -> argparse.Namespace:
|
||||||
|
"""Parse command line arguments."""
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="Fix .typ files to conform to Typst syntax using AI API"
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"-f",
|
||||||
|
"--file",
|
||||||
|
action="append",
|
||||||
|
dest="files",
|
||||||
|
help="Specific .typ files to fix",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--dry-run", action="store_true", help="Do not call AI or write files"
|
||||||
|
)
|
||||||
|
parser.add_argument("--verbose", action="store_true", help="Enable debug logging")
|
||||||
|
parser.add_argument("-n", type=int, default=3, help="Concurrent limit (default: 3)")
|
||||||
|
return parser.parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
async def async_main(args: argparse.Namespace, logger: logging.Logger) -> None:
|
||||||
|
"""Async main entry point."""
|
||||||
|
if args.files:
|
||||||
|
typ_paths = []
|
||||||
|
for f in args.files:
|
||||||
|
p = Path(f)
|
||||||
|
if not p.is_absolute():
|
||||||
|
p = DATA_DIR / f
|
||||||
|
typ_paths.append(p)
|
||||||
|
else:
|
||||||
|
typ_paths = find_typ_files()
|
||||||
|
|
||||||
|
logger.info(f"Found {len(typ_paths)} .typ files to fix")
|
||||||
|
|
||||||
|
if not typ_paths:
|
||||||
|
logger.warning("No .typ files found to fix")
|
||||||
|
return
|
||||||
|
|
||||||
|
api_config = load_env()
|
||||||
|
|
||||||
|
semaphore = asyncio.Semaphore(args.n)
|
||||||
|
|
||||||
|
async def limited_fix(session: aiohttp.ClientSession, typ_path: Path) -> FixResult:
|
||||||
|
async with semaphore:
|
||||||
|
return await fix_typ_file(
|
||||||
|
session, typ_path, api_config, logger, args.dry_run
|
||||||
|
)
|
||||||
|
|
||||||
|
async with aiohttp.ClientSession() as session:
|
||||||
|
tasks = [asyncio.create_task(limited_fix(session, typ)) for typ in typ_paths]
|
||||||
|
|
||||||
|
results = []
|
||||||
|
try:
|
||||||
|
for coro in asyncio.as_completed(tasks):
|
||||||
|
result = await coro
|
||||||
|
results.append(result)
|
||||||
|
except asyncio.CancelledError:
|
||||||
|
logger.warning("Cancelled! Shutting down...")
|
||||||
|
for task in tasks:
|
||||||
|
task.cancel()
|
||||||
|
await asyncio.gather(*tasks, return_exceptions=True)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
results.sort(key=lambda r: r.question)
|
||||||
|
|
||||||
|
skipped = sum(1 for r in results if r.skipped)
|
||||||
|
fixed = sum(1 for r in results if r.success and not r.skipped)
|
||||||
|
logger.info(f"Complete: {fixed}/{len(results)} fixed, {skipped} skipped")
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
"""Main entry point."""
|
||||||
|
args = parse_args()
|
||||||
|
logger = setup_logging("fix_typ", args.verbose)
|
||||||
|
|
||||||
|
logger.info(f"fix_typ starting (Dry-run: {args.dry_run}, Workers: {args.n})")
|
||||||
|
logger.info(f"Data directory: {DATA_DIR}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
asyncio.run(async_main(args, logger))
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
logger.warning("Interrupted by user")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@ -82,6 +82,18 @@ def build_prompt(question_data: dict, typ_content: str | None) -> str:
|
|||||||
return "".join(parts)
|
return "".join(parts)
|
||||||
|
|
||||||
|
|
||||||
|
def get_image_attachments(question_data: dict) -> list[tuple[str, Path]]:
|
||||||
|
"""Return list of (attachment_name, path) for image attachments."""
|
||||||
|
images = []
|
||||||
|
for att in question_data.get("attachments", []):
|
||||||
|
att_path = DATA_DIR / att
|
||||||
|
if att_path.exists() and att.lower().endswith(
|
||||||
|
(".png", ".jpg", ".jpeg", ".gif", ".bmp", ".webp")
|
||||||
|
):
|
||||||
|
images.append((att, att_path))
|
||||||
|
return images
|
||||||
|
|
||||||
|
|
||||||
async def call_api_streaming(
|
async def call_api_streaming(
|
||||||
session: aiohttp.ClientSession,
|
session: aiohttp.ClientSession,
|
||||||
question_name: str,
|
question_name: str,
|
||||||
@ -90,13 +102,34 @@ async def call_api_streaming(
|
|||||||
api_key: str,
|
api_key: str,
|
||||||
model: str,
|
model: str,
|
||||||
logger: logging.Logger,
|
logger: logging.Logger,
|
||||||
|
image_contents: list[tuple[str, Path]] | None = None,
|
||||||
) -> str | None:
|
) -> str | None:
|
||||||
"""Call the AI API with streaming to solve the question."""
|
"""Call the AI API with streaming to solve the question."""
|
||||||
|
import base64
|
||||||
|
|
||||||
headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
|
headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
|
||||||
|
|
||||||
|
messages = [{"role": "user", "content": [{"type": "text", "text": prompt}]}]
|
||||||
|
|
||||||
|
if image_contents:
|
||||||
|
for att_name, att_path in image_contents:
|
||||||
|
with open(att_path, "rb") as f:
|
||||||
|
image_data = base64.b64encode(f.read()).decode("utf-8")
|
||||||
|
suffix = att_path.suffix[1:].lower()
|
||||||
|
if suffix in ("jpg", "jpeg"):
|
||||||
|
mime_type = "image/jpeg"
|
||||||
|
else:
|
||||||
|
mime_type = f"image/{suffix}"
|
||||||
|
messages[0]["content"].append(
|
||||||
|
{
|
||||||
|
"type": "image_url",
|
||||||
|
"image_url": {"url": f"data:{mime_type};base64,{image_data}"},
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
payload = {
|
payload = {
|
||||||
"model": model,
|
"model": model,
|
||||||
"messages": [{"role": "user", "content": [{"type": "text", "text": prompt}]}],
|
"messages": messages,
|
||||||
"max_tokens": 4096,
|
"max_tokens": 4096,
|
||||||
"stream": True,
|
"stream": True,
|
||||||
}
|
}
|
||||||
@ -123,12 +156,17 @@ async def call_api_streaming(
|
|||||||
try:
|
try:
|
||||||
chunk = json.loads(data)
|
chunk = json.loads(data)
|
||||||
logger.debug(f"[{question_name}] Chunk: {chunk}")
|
logger.debug(f"[{question_name}] Chunk: {chunk}")
|
||||||
delta = chunk.get("choices", [{}])[0].get("delta", {})
|
choices = chunk.get("choices")
|
||||||
|
if not choices:
|
||||||
|
continue
|
||||||
|
delta = choices[0].get("delta", {})
|
||||||
content = delta.get("content")
|
content = delta.get("content")
|
||||||
if content:
|
if content:
|
||||||
full_content.append(content)
|
full_content.append(content)
|
||||||
except json.JSONDecodeError:
|
except json.JSONDecodeError:
|
||||||
continue
|
continue
|
||||||
|
except IndexError:
|
||||||
|
continue
|
||||||
|
|
||||||
content = "".join(full_content)
|
content = "".join(full_content)
|
||||||
if not content:
|
if not content:
|
||||||
@ -182,6 +220,7 @@ async def solve_question(
|
|||||||
)
|
)
|
||||||
|
|
||||||
prompt = build_prompt(question_data, typ_content)
|
prompt = build_prompt(question_data, typ_content)
|
||||||
|
image_contents = get_image_attachments(question_data)
|
||||||
|
|
||||||
content = await call_api_streaming(
|
content = await call_api_streaming(
|
||||||
session,
|
session,
|
||||||
@ -191,6 +230,7 @@ async def solve_question(
|
|||||||
api_config["key"],
|
api_config["key"],
|
||||||
api_config["model"],
|
api_config["model"],
|
||||||
logger,
|
logger,
|
||||||
|
image_contents,
|
||||||
)
|
)
|
||||||
|
|
||||||
if content is None:
|
if content is None:
|
||||||
|
|||||||
Reference in New Issue
Block a user