fz-mirrors/Build-A-Large-Language-Model-CN
1
0
Fork 0
mirror of https://github.com/skindhu/Build-A-Large-Language-Model-CN.git synced 2026-08-29 21:30:25 +08:00
Code Issues Packages Projects Releases Wiki Activity

Revise token ID list and corresponding explanations

Browse Source 
Updated token ID examples and explanations in the document.
This commit is contained in:
FMLS
2026-07-04 22:53:33 +08:00
committed by GitHub
parent 592cc35a7a
commit b58c369e29
1 changed files with 3 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
cn-Book/2.处理文本数据.md
View File

@@ -448,13 +448,13 @@ tokenizer = SimpleTokenizerV2(vocab)
print(tokenizer.encode(text)) print(tokenizer.encode(text))
``` ```
这将输出以下token ID列表： 这将打印以下词元ID：
``` ```
[1160, 5, 362, 1155, 642, 1000, 10, 1159, 57, 1013, 981, 1009, 738, 1013, 1160, 7] [1131, 5, 355, 1126, 628, 975, 10, 1130, 55, 988, 956, 984, 722, 988, 1131, 7]
``` ```
从上面的内容可以看出，token ID 列表中包含了 1159 ,它对应于 <|endoftext|> 分隔token，以及两个 1160 ，用于表示未知单词。 从上面的内容可以看出，token ID 列表中包含了 1130 ,它对应于 <|endoftext|> 分隔token，以及两个 1131 ，用于表示未知单词。
```python ```python
print(tokenizer.decode(tokenizer.encode(text))) print(tokenizer.decode(tokenizer.encode(text)))