fix(markdown_normalizer): adopt safe-by-default strategy for escaping
- Set 'enable_escape_fix' to False by default to prevent accidental corruption - Improve LaTeX display math identification using regex protection - Update documentation to reflect opt-in recommendation for escape fixes - Fix Issue #57 remaining aggressive escaping bugs
This commit is contained in:
fujie
12
ISSUE_57_REPLY.md
Normal file
12
ISSUE_57_REPLY.md
Normal file
@@ -0,0 +1,12 @@
|
||||
# Reply to Issue #57
|
||||
|
||||
I have addressed these issues in **v1.2.8** with a focus on reliability and a "Safe-by-Default" approach:
|
||||
|
||||
1. **Robust Error Rollback (Items 1, 4, 5)**: I implemented a full `try...except` wrapper. If any error occurs during normalization, the plugin now returns the **100% original text**. This ensures that the output is never partially modified or corrupted.
|
||||
2. **Conservative Escaping (Item 2)**: To avoid breaking technical content like regex or paths, the escape fixer now strictly skips all code blocks, inline code, and LaTeX formulas by default. I have shifted toward an "opt-in" model where aggressive cleaning is disabled unless specifically requested.
|
||||
3. **Fixed Configuration (Item 3)**: The `enable_escape_fix_in_code_blocks` Valve was intended to handle escaping within code blocks (e.g., for fixing flat SQL output), but there was a bug preventing it from being applied. I have fixed this, and the setting is now fully functional.
|
||||
4. **Privacy & Reliability**: I have changed the default for `show_debug_log` to `False`. While it was previously enabled by default to help gather feedback and squash bugs during the initial development phase, the plugin has now undergone multiple iterations and reliability enhancements (including the new tiered protection and rollback mechanisms), making it stable enough for a "silent" and private default operation.
|
||||
|
||||
**Recommendation**: If you encounter SQL or data blocks that appear on a single line, you can now manually enable `enable_escape_fix_in_code_blocks` in the Valves to fix them safely.
|
||||
|
||||
Please update to the latest version via [OpenWebUI Community](https://openwebui.com/functions/baaa8732-9348-40b7-8359-7e009660e23c). Thank you for your valuable feedback!
|
||||
99
TEST_CASES_V1.2.8.md
Normal file
99
TEST_CASES_V1.2.8.md
Normal file
@@ -0,0 +1,99 @@
|
||||
# Markdown Normalizer v1.2.8 测试用例集
|
||||
|
||||
您可以将以下内容逐个复制到 OpenWebUI 的聊天框中，以验证插件的各项修复功能。
|
||||
|
||||
---
|
||||
|
||||
## 用例 1：验证 SQL 代码块换行修复 (需要手动开启配置)
|
||||
|
||||
**测试目的**：验证 `enable_escape_fix_in_code_blocks` 开关是否生效。
|
||||
**前提条件**：请先在插件 Valves 设置中将 `enable_escape_fix_in_code_blocks` 设置为 **开启 (True)**。
|
||||
|
||||
**复制以下内容：**
|
||||
```text
|
||||
请帮我美化这段 SQL 的排版，使其恢复正常换行：
|
||||
|
||||
```sql
|
||||
SELECT * \n FROM users \n WHERE status = 'active' \n AND created_at > '2024-01-01' \n ORDER BY id DESC;
|
||||
```
|
||||
```
|
||||
|
||||
**预期效果**：SQL 代码块内的 `\n` 消失，变为整齐的多行 SQL 语句。
|
||||
|
||||
---
|
||||
|
||||
## 用例 2：验证上下文感知保护 (防止误伤技术内容)
|
||||
|
||||
**测试目的**：验证插件是否能准确识别“纯文本”和“代码区域”，只修复该修复的地方。
|
||||
**配置要求**：默认配置即可。
|
||||
|
||||
**复制以下内容：**
|
||||
```text
|
||||
这是一个综合测试用例。
|
||||
|
||||
1. 普通文本修复测试：
|
||||
这是第一行\\n这是第二行（你应该看到这里发生了换行）。
|
||||
|
||||
2. 行内代码保护测试（不应被修改）：
|
||||
- 正则表达式：`[\n\r\t]`
|
||||
- Windows 路径：`C:\Windows\System32\drivers\etc\hosts`
|
||||
- 转义测试：`\\n` 应该保持字面量。
|
||||
|
||||
3. LaTeX 命令保护测试：
|
||||
这里的数学公式 $\times \theta \nu \sum$ 应该渲染正常，反斜杠不应被修掉。
|
||||
|
||||
4. 现代 LaTeX 定界符转换：
|
||||
\[ E = mc^2 \]
|
||||
（上面这行应该被自动转换为 $$ 包围的块级公式）
|
||||
```
|
||||
|
||||
**预期效果**：
|
||||
- 第一部分的 `\\n` 成功换行。
|
||||
- 第二部分反引号 `` ` `` 里的内容原封不动。
|
||||
- 第三部分的希腊字母公式渲染正常。
|
||||
- 第四部分的 `\[` 变成了 `$$` 且能正常显示公式。
|
||||
|
||||
---
|
||||
|
||||
## 用例 3：验证思维链与详情标签规范化
|
||||
|
||||
**测试目的**：验证对 `<thought>` 和 `<details>` 标签的排版优化。
|
||||
|
||||
**复制以下内容：**
|
||||
```text
|
||||
<thinking>
|
||||
这是一个正在思考的思维链。
|
||||
</thinking>
|
||||
<details>
|
||||
<summary>点击查看详情</summary>
|
||||
这里的排版通常容易出错。
|
||||
</details>
|
||||
紧接着详情标签的文字（应该和上面有空行隔开）。
|
||||
```
|
||||
|
||||
**预期效果**：
|
||||
- `<thinking>` 标签被统一为 `<thought>`。
|
||||
- `</details>` 标签下方自动注入了空行，防止与正文粘连导致渲染失效。
|
||||
|
||||
---
|
||||
|
||||
## 用例 4：极端压力与回滚测试 (稳定性验证)
|
||||
|
||||
**测试目的**：模拟复杂嵌套环境，验证 100% 回滚机制。
|
||||
|
||||
**复制以下内容：**
|
||||
```text
|
||||
尝试混合所有复杂元素：
|
||||
- 列表项 1
|
||||
- 列表项 2 with `inline \\n code`
|
||||
- $ \text{Math } \alpha $
|
||||
```sql
|
||||
-- SQL with nested issue
|
||||
SELECT 'literal \n string' FROM `table`;
|
||||
```
|
||||
<thought>End of test</thought>
|
||||
```
|
||||
|
||||
**预期效果**：
|
||||
- 无论内部处理逻辑多么复杂，插件都应保证输出稳定的结果。
|
||||
- 如果模拟任何内部崩溃（技术人员可用），消息会回滚至此原始文本，不会导致页面白屏。
|
||||
@@ -1,5 +1,4 @@
|
||||
# Markdown Normalizer Filter
|
||||
|
||||
**Author:** [Fu-Jie](https://github.com/Fu-Jie/openwebui-extensions) | **Version:** 1.2.8 | **Project:** [OpenWebUI Extensions](https://github.com/Fu-Jie/openwebui-extensions) | **License:** MIT
|
||||
|
||||
A powerful, context-aware content normalizer filter for Open WebUI designed to fix common Markdown formatting issues in LLM outputs. It ensures that code blocks, LaTeX formulas, Mermaid diagrams, and other structural Markdown elements are rendered flawlessly, without destroying valid technical content.
|
||||
@@ -10,6 +9,16 @@ A powerful, context-aware content normalizer filter for Open WebUI designed to f
|
||||
|
||||
---
|
||||
|
||||
## 🔥 What's New in v1.2.8
|
||||
* **Safe-by-Default Strategy**: The `enable_escape_fix` feature is now **disabled by default**. This prevents unwanted modifications to valid technical text like Windows file paths (`C:\new\test`) or complex LaTeX formulas.
|
||||
* **LaTeX Parsing Fix**: Improved the logic for identifying display math (`$$ ... $$`). Fixed a bug where LaTeX commands starting with `\n` (like `\nabla`) were incorrectly treated as newlines.
|
||||
* **Reliability Enhancement**: Complete error fallback mechanism. Guarantees 0% data loss during processing.
|
||||
* **Inline Code Protection**: Upgraded escaping logic to protect inline code blocks (`` `...` ``).
|
||||
* **Code Block Escaping Control**: The `enable_escape_fix_in_code_blocks` Valve now correctly targets broken newlines inside code blocks (perfect for fixing flat SQL queries) when enabled.
|
||||
* **Privacy Optimization**: `show_debug_log` now defaults to `False` to prevent console noise.
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Why do you need this plugin? (What does it do?)
|
||||
|
||||
Language Models (LLMs) often generate malformed Markdown due to tokenization artifacts, aggressive escaping, or hallucinated formatting. If you've ever seen:
|
||||
@@ -42,12 +51,6 @@ Before making any changes, the plugin builds a semantic map of the text to prote
|
||||
### 3. Reliability & Safety
|
||||
- **100% Rollback Guarantee**: If any normalization logic fails or crashes, the plugin catches the error and silently returns the exact original text, ensuring your chat never breaks.
|
||||
|
||||
## 🔥 What's New in v1.2.8
|
||||
* **Reliability Enhancement**: Complete error fallback mechanism. Guarantees 0% data loss during processing.
|
||||
* **Inline Code Protection**: Upgraded escaping logic to protect inline code blocks (`` `...` ``).
|
||||
* **Code Block Escaping Control**: The `enable_escape_fix_in_code_blocks` Valve now correctly targets broken newlines inside code blocks (perfect for fixing flat SQL queries) when enabled.
|
||||
* **Privacy Optimization**: `show_debug_log` now defaults to `False` to prevent console noise.
|
||||
|
||||
## 🌐 Multilingual Support
|
||||
|
||||
The plugin UI and status notifications automatically switch based on your language:
|
||||
@@ -64,7 +67,7 @@ The plugin UI and status notifications automatically switch based on your langua
|
||||
| Parameter | Default | Description |
|
||||
| :--- | :--- | :--- |
|
||||
| `priority` | `50` | Filter priority. Higher runs later (recommended to run this after all other content filters). |
|
||||
| `enable_escape_fix` | `True` | Convert excessive literal escape characters (`\n`, `\t`) to real spacing. |
|
||||
| `enable_escape_fix` | `False` | Convert excessive literal escape characters (`\n`, `\t`) to real spacing. (Default: False for safety). |
|
||||
| `enable_escape_fix_in_code_blocks` | `False` | **Pro-tip**: Turn this ON if your SQL/HTML code blocks are constantly printing on a single line. Turn OFF for Python/C++. |
|
||||
| `enable_thought_tag_fix` | `True` | Normalize `<think>` tags. |
|
||||
| `enable_details_tag_fix` | `True` | Normalize `<details>` spacing. |
|
||||
|
||||
@@ -10,6 +10,14 @@
|
||||
|
||||
---
|
||||
|
||||
## 🔥 最新更新 v1.2.8
|
||||
* **“默认安全”策略 (Safe-by-Default)**：`enable_escape_fix` 功能现在**默认禁用**。这能有效防止插件在未经授权的情况下误改 Windows 路径 (`C:\new\test`) 或复杂的 LaTeX 公式。
|
||||
* **LaTeX 解析优化**：重构了显示数学公式 (`$$ ... $$`) 的识别逻辑。修复了 LaTeX 命令如果以 `\n` 开头（如 `\nabla`）会被错误识别为换行符的 Bug。
|
||||
* **可靠性增强**：实现了完整的错误回滚机制。当修复过程发生意外错误时，保证 100% 返回原始文本，不丢失任何数据。
|
||||
* **配置项修复**：`enable_escape_fix_in_code_blocks` 配置项现在能正确作用于代码块了。**如果您遇到 SQL 挤在一行的问题，只需在设置中手动开启此项即可。**
|
||||
|
||||
---
|
||||
|
||||
## 🚀 为什么你需要这个插件？(它能解决什么问题？)
|
||||
|
||||
由于分词 (Tokenization) 伪影、过度转义或格式幻觉，LLM 经常会生成破损的 Markdown。如果你遇到过以下情况：
|
||||
@@ -42,12 +50,6 @@
|
||||
### 3. 绝对的可靠性与安全 (100% Rollback)
|
||||
- **无损回滚机制**：如果在修复过程中发生任何意外错误或崩溃，插件会立即捕获异常，并静默返回**绝对原始**的文本，确保你的对话永远不会因插件报错而丢失。
|
||||
|
||||
## 🔥 最新更新 v1.2.8
|
||||
* **可靠性增强**：修复了错误回滚机制。当规范化过程中发生意外错误时，插件现在会正确返回原始文本，而不是返回被部分修改的损坏内容。
|
||||
* **内联代码保护**：优化了转义字符清理逻辑，现在会保护内联代码块（`` `...` ``）不被错误转义，防止破坏有效的代码片段。
|
||||
* **配置项修复**：`enable_escape_fix_in_code_blocks` 配置项现在能正确作用于代码块了。**在代码块内修复换行符（比如修复 SQL）时，只需在设置中开启此选项即可。**
|
||||
* **隐私与日志优化**：将 `show_debug_log` 默认值修改为 `False`，避免将可能敏感的内容自动输出到浏览器控制台，并减少不必要的日志噪音。
|
||||
|
||||
## 🌐 多语言支持 (i18n)
|
||||
|
||||
界面的状态提示气泡会根据你的浏览器语言自动切换：
|
||||
@@ -64,7 +66,7 @@
|
||||
| 参数 | 默认值 | 描述 |
|
||||
| :--- | :--- | :--- |
|
||||
| `priority` | `50` | 过滤器优先级。数值越大越靠后（建议放在其他内容过滤器之后运行）。 |
|
||||
| `enable_escape_fix` | `True` | 修复过度的转义字符（将字面量 `\n` 转换为实际换行）。 |
|
||||
| `enable_escape_fix` | `False` | 修复过度的转义字符（将字面量 `\n` 转换为实际换行）。**默认禁用以保证安全。** |
|
||||
| `enable_escape_fix_in_code_blocks` | `False` | **高阶技巧**：如果你的 SQL 或 HTML 代码块总是挤在一行，**请开启此项**。如果你经常写 Python/C++，建议保持关闭。 |
|
||||
| `enable_thought_tag_fix` | `True` | 规范化思维标签为 `<thought>`。 |
|
||||
| `enable_details_tag_fix` | `True` | 修复 `<details>` 标签的排版间距。 |
|
||||
@@ -84,4 +86,4 @@
|
||||
如果这个插件拯救了你的排版，欢迎到 [OpenWebUI Extensions](https://github.com/Fu-Jie/openwebui-extensions) 点个 Star，这是我持续改进的最大动力。感谢支持！
|
||||
|
||||
## 🧩 其他
|
||||
* **故障排除**：遇到“负向修复”（即原本正常的排版被修坏了）？请开启 `show_debug_log`，在 F12 控制台复制出原始文本，并在 GitHub 提交 Issue：[提交 Issue](https://github.com/Fu-Jie/openwebui-extensions/issues)
|
||||
* **故障排除**：遇到“负向修复”（即原本正常的排版被修坏了）？请开启 `show_debug_log`，在 F12 控制台复制出原始文本，并在 GitHub 提交 Issue：[提交 Issue](https://github.com/Fu-Jie/openwebui-extensions/issues)
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
# Markdown Normalizer Filter
|
||||
|
||||
**Author:** [Fu-Jie](https://github.com/Fu-Jie/openwebui-extensions) | **Version:** 1.2.8 | **Project:** [OpenWebUI Extensions](https://github.com/Fu-Jie/openwebui-extensions) | **License:** MIT
|
||||
|
||||
A powerful, context-aware content normalizer filter for Open WebUI designed to fix common Markdown formatting issues in LLM outputs. It ensures that code blocks, LaTeX formulas, Mermaid diagrams, and other structural Markdown elements are rendered flawlessly, without destroying valid technical content.
|
||||
@@ -10,6 +9,16 @@ A powerful, context-aware content normalizer filter for Open WebUI designed to f
|
||||
|
||||
---
|
||||
|
||||
## 🔥 What's New in v1.2.8
|
||||
* **Safe-by-Default Strategy**: The `enable_escape_fix` feature is now **disabled by default**. This prevents unwanted modifications to valid technical text like Windows file paths (`C:\new\test`) or complex LaTeX formulas.
|
||||
* **LaTeX Parsing Fix**: Improved the logic for identifying display math (`$$ ... $$`). Fixed a bug where LaTeX commands starting with `\n` (like `\nabla`) were incorrectly treated as newlines.
|
||||
* **Reliability Enhancement**: Complete error fallback mechanism. Guarantees 0% data loss during processing.
|
||||
* **Inline Code Protection**: Upgraded escaping logic to protect inline code blocks (`` `...` ``).
|
||||
* **Code Block Escaping Control**: The `enable_escape_fix_in_code_blocks` Valve now correctly targets broken newlines inside code blocks (perfect for fixing flat SQL queries) when enabled.
|
||||
* **Privacy Optimization**: `show_debug_log` now defaults to `False` to prevent console noise.
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Why do you need this plugin? (What does it do?)
|
||||
|
||||
Language Models (LLMs) often generate malformed Markdown due to tokenization artifacts, aggressive escaping, or hallucinated formatting. If you've ever seen:
|
||||
@@ -42,12 +51,6 @@ Before making any changes, the plugin builds a semantic map of the text to prote
|
||||
### 3. Reliability & Safety
|
||||
- **100% Rollback Guarantee**: If any normalization logic fails or crashes, the plugin catches the error and silently returns the exact original text, ensuring your chat never breaks.
|
||||
|
||||
## 🔥 What's New in v1.2.8
|
||||
* **Reliability Enhancement**: Complete error fallback mechanism. Guarantees 0% data loss during processing.
|
||||
* **Inline Code Protection**: Upgraded escaping logic to protect inline code blocks (`` `...` ``).
|
||||
* **Code Block Escaping Control**: The `enable_escape_fix_in_code_blocks` Valve now correctly targets broken newlines inside code blocks (perfect for fixing flat SQL queries) when enabled.
|
||||
* **Privacy Optimization**: `show_debug_log` now defaults to `False` to prevent console noise.
|
||||
|
||||
## 🌐 Multilingual Support
|
||||
|
||||
The plugin UI and status notifications automatically switch based on your language:
|
||||
@@ -64,7 +67,7 @@ The plugin UI and status notifications automatically switch based on your langua
|
||||
| Parameter | Default | Description |
|
||||
| :--- | :--- | :--- |
|
||||
| `priority` | `50` | Filter priority. Higher runs later (recommended to run this after all other content filters). |
|
||||
| `enable_escape_fix` | `True` | Convert excessive literal escape characters (`\n`, `\t`) to real spacing. |
|
||||
| `enable_escape_fix` | `False` | Convert excessive literal escape characters (`\n`, `\t`) to real spacing. (Default: False for safety). |
|
||||
| `enable_escape_fix_in_code_blocks` | `False` | **Pro-tip**: Turn this ON if your SQL/HTML code blocks are constantly printing on a single line. Turn OFF for Python/C++. |
|
||||
| `enable_thought_tag_fix` | `True` | Normalize `<think>` tags. |
|
||||
| `enable_details_tag_fix` | `True` | Normalize `<details>` spacing. |
|
||||
|
||||
@@ -10,6 +10,14 @@
|
||||
|
||||
---
|
||||
|
||||
## 🔥 最新更新 v1.2.8
|
||||
* **“默认安全”策略 (Safe-by-Default)**：`enable_escape_fix` 功能现在**默认禁用**。这能有效防止插件在未经授权的情况下误改 Windows 路径 (`C:\new\test`) 或复杂的 LaTeX 公式。
|
||||
* **LaTeX 解析优化**：重构了显示数学公式 (`$$ ... $$`) 的识别逻辑。修复了 LaTeX 命令如果以 `\n` 开头（如 `\nabla`）会被错误识别为换行符的 Bug。
|
||||
* **可靠性增强**：实现了完整的错误回滚机制。当修复过程发生意外错误时，保证 100% 返回原始文本，不丢失任何数据。
|
||||
* **配置项修复**：`enable_escape_fix_in_code_blocks` 配置项现在能正确作用于代码块了。**如果您遇到 SQL 挤在一行的问题，只需在设置中手动开启此项即可。**
|
||||
|
||||
---
|
||||
|
||||
## 🚀 为什么你需要这个插件？(它能解决什么问题？)
|
||||
|
||||
由于分词 (Tokenization) 伪影、过度转义或格式幻觉，LLM 经常会生成破损的 Markdown。如果你遇到过以下情况：
|
||||
@@ -42,12 +50,6 @@
|
||||
### 3. 绝对的可靠性与安全 (100% Rollback)
|
||||
- **无损回滚机制**：如果在修复过程中发生任何意外错误或崩溃，插件会立即捕获异常，并静默返回**绝对原始**的文本，确保你的对话永远不会因插件报错而丢失。
|
||||
|
||||
## 🔥 最新更新 v1.2.8
|
||||
* **可靠性增强**：修复了错误回滚机制。当规范化过程中发生意外错误时，插件现在会正确返回原始文本，而不是返回被部分修改的损坏内容。
|
||||
* **内联代码保护**：优化了转义字符清理逻辑，现在会保护内联代码块（`` `...` ``）不被错误转义，防止破坏有效的代码片段。
|
||||
* **配置项修复**：`enable_escape_fix_in_code_blocks` 配置项现在能正确作用于代码块了。**在代码块内修复换行符（比如修复 SQL）时，只需在设置中开启此选项即可。**
|
||||
* **隐私与日志优化**：将 `show_debug_log` 默认值修改为 `False`，避免将可能敏感的内容自动输出到浏览器控制台，并减少不必要的日志噪音。
|
||||
|
||||
## 🌐 多语言支持 (i18n)
|
||||
|
||||
界面的状态提示气泡会根据你的浏览器语言自动切换：
|
||||
@@ -64,7 +66,7 @@
|
||||
| 参数 | 默认值 | 描述 |
|
||||
| :--- | :--- | :--- |
|
||||
| `priority` | `50` | 过滤器优先级。数值越大越靠后（建议放在其他内容过滤器之后运行）。 |
|
||||
| `enable_escape_fix` | `True` | 修复过度的转义字符（将字面量 `\n` 转换为实际换行）。 |
|
||||
| `enable_escape_fix` | `False` | 修复过度的转义字符（将字面量 `\n` 转换为实际换行）。**默认禁用以保证安全。** |
|
||||
| `enable_escape_fix_in_code_blocks` | `False` | **高阶技巧**：如果你的 SQL 或 HTML 代码块总是挤在一行，**请开启此项**。如果你经常写 Python/C++，建议保持关闭。 |
|
||||
| `enable_thought_tag_fix` | `True` | 规范化思维标签为 `<thought>`。 |
|
||||
| `enable_details_tag_fix` | `True` | 修复 `<details>` 标签的排版间距。 |
|
||||
@@ -84,4 +86,4 @@
|
||||
如果这个插件拯救了你的排版，欢迎到 [OpenWebUI Extensions](https://github.com/Fu-Jie/openwebui-extensions) 点个 Star，这是我持续改进的最大动力。感谢支持！
|
||||
|
||||
## 🧩 其他
|
||||
* **故障排除**：遇到“负向修复”（即原本正常的排版被修坏了）？请开启 `show_debug_log`，在 F12 控制台复制出原始文本，并在 GitHub 提交 Issue：[提交 Issue](https://github.com/Fu-Jie/openwebui-extensions/issues)
|
||||
* **故障排除**：遇到“负向修复”（即原本正常的排版被修坏了）？请开启 `show_debug_log`，在 F12 控制台复制出原始文本，并在 GitHub 提交 Issue：[提交 Issue](https://github.com/Fu-Jie/openwebui-extensions/issues)
|
||||
|
||||
@@ -236,7 +236,7 @@ TRANSLATIONS = {
|
||||
class NormalizerConfig:
|
||||
"""Configuration class for enabling/disabling specific normalization rules"""
|
||||
|
||||
enable_escape_fix: bool = True # Fix excessive escape characters
|
||||
enable_escape_fix: bool = False # Fix excessive escape characters (Default False for safety)
|
||||
enable_escape_fix_in_code_blocks: bool = (
|
||||
False # Apply escape fix inside code blocks (default: False for safety)
|
||||
)
|
||||
@@ -485,12 +485,14 @@ class ContentNormalizer:
|
||||
# 2. Protect inline code
|
||||
inline_parts = parts[i].split("`")
|
||||
for k in range(0, len(inline_parts), 2): # Even indices are non-inline-code text
|
||||
# 3. Protect LaTeX formulas within text
|
||||
# Split by $ to find inline/block math
|
||||
sub_parts = inline_parts[k].split("$")
|
||||
# 3. Protect LaTeX formulas within text (safe for $$ and $)
|
||||
# Use regex to split and keep delimiters
|
||||
sub_parts = re.split(
|
||||
r"(\$\$.*?\$\$|\$.*?\$)", inline_parts[k], flags=re.DOTALL
|
||||
)
|
||||
for j in range(0, len(sub_parts), 2): # Even indices are non-math text
|
||||
sub_parts[j] = clean_text(sub_parts[j])
|
||||
inline_parts[k] = "$".join(sub_parts)
|
||||
inline_parts[k] = "".join(sub_parts)
|
||||
parts[i] = "`".join(inline_parts)
|
||||
else:
|
||||
# Inside code block and enable_escape_fix_in_code_blocks is True
|
||||
@@ -724,8 +726,8 @@ class Filter:
|
||||
description="Priority level (lower = earlier).",
|
||||
)
|
||||
enable_escape_fix: bool = Field(
|
||||
default=True,
|
||||
description="Fix excessive escape characters (\\n, \\t, etc.).",
|
||||
default=False,
|
||||
description="Fix excessive escape characters (\\n, \\t, etc.). Default: False for safety.",
|
||||
)
|
||||
enable_escape_fix_in_code_blocks: bool = Field(
|
||||
default=False,
|
||||
|
||||
@@ -54,6 +54,15 @@ from open_webui.storage.provider import Storage
|
||||
import mimetypes
|
||||
import uuid
|
||||
|
||||
if os.path.exists("/app/backend/data"):
|
||||
CHAT_MAPPING_FILE = Path(
|
||||
"/app/backend/data/copilot_workspace/api_key_chat_id_mapping.json"
|
||||
)
|
||||
else:
|
||||
CHAT_MAPPING_FILE = (
|
||||
Path(os.getcwd()) / "copilot_workspace" / "api_key_chat_id_mapping.json"
|
||||
)
|
||||
|
||||
# Get OpenWebUI version for capability detection
|
||||
try:
|
||||
from open_webui.env import VERSION as open_webui_version
|
||||
@@ -4879,6 +4888,41 @@ class Pipe:
|
||||
|
||||
return cwd
|
||||
|
||||
def _record_user_chat_mapping(
|
||||
self, user_id: Optional[str], chat_id: Optional[str]
|
||||
) -> None:
|
||||
"""Persist the latest chat_id for the current user."""
|
||||
if not user_id or not chat_id:
|
||||
return
|
||||
|
||||
mapping_file = CHAT_MAPPING_FILE
|
||||
|
||||
try:
|
||||
mapping_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
mapping: Dict[str, str] = {}
|
||||
if mapping_file.exists():
|
||||
try:
|
||||
loaded = json.loads(mapping_file.read_text(encoding="utf-8"))
|
||||
if isinstance(loaded, dict):
|
||||
mapping = {str(k): str(v) for k, v in loaded.items()}
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
f"[Session Tracking] Failed to read mapping file {mapping_file}: {e}"
|
||||
)
|
||||
|
||||
mapping[str(user_id)] = str(chat_id)
|
||||
|
||||
temp_file = mapping_file.with_suffix(mapping_file.suffix + ".tmp")
|
||||
temp_file.write_text(
|
||||
json.dumps(mapping, ensure_ascii=False, indent=2, sort_keys=True)
|
||||
+ "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
temp_file.replace(mapping_file)
|
||||
except Exception as e:
|
||||
logger.warning(f"[Session Tracking] Failed to persist mapping: {e}")
|
||||
|
||||
def _build_client_config(self, user_id: str = None, chat_id: str = None) -> dict:
|
||||
"""Build CopilotClient config from valves and request body."""
|
||||
cwd = self._get_workspace_dir(user_id=user_id, chat_id=chat_id)
|
||||
@@ -5837,6 +5881,8 @@ class Pipe:
|
||||
)
|
||||
is_admin = user_data.get("role") == "admin"
|
||||
|
||||
self._record_user_chat_mapping(user_data.get("id"), __chat_id__)
|
||||
|
||||
# Robustly parse User Valves
|
||||
user_valves = self._get_user_valves(__user__)
|
||||
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
from plugins.filters.markdown_normalizer.markdown_normalizer import ContentNormalizer, NormalizerConfig
|
||||
|
||||
def test_latex_display_math_protection():
|
||||
"""Verify that $$\nabla$$ is NOT broken by escape fix."""
|
||||
config = NormalizerConfig(enable_escape_fix=True)
|
||||
norm = ContentNormalizer(config)
|
||||
|
||||
# Input has literal backslash + n (represented as \\n in python code)
|
||||
# Total input: $$ \ n a b l a $$
|
||||
text = r"$$\nabla$$"
|
||||
res = norm.normalize(text)
|
||||
|
||||
# It should NOT change literal \n to a newline inside $$
|
||||
assert "\n" not in res, f"LaTeX display math was corrupted with a real newline: {repr(res)}"
|
||||
assert res == text, f"Expected {repr(text)}, got {repr(res)}"
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
test_latex_display_math_protection()
|
||||
print("✅ LaTeX protection test passed.")
|
||||
except AssertionError as e:
|
||||
print(f"❌ LaTeX protection test FAILED: {e}")
|
||||
Reference in New Issue
Block a user