fz-mirrors/Build-A-Large-Language-Model-CN
1
0
Fork 0
mirror of https://github.com/skindhu/Build-A-Large-Language-Model-CN.git synced 2026-08-29 21:30:25 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #30 from FMLS/patch-1

Browse Source 
Revise token ID list and corresponding explanations
This commit is contained in:
long_long_ago
2026-08-10 10:54:48 +08:00
committed by GitHub
parent 592cc35a7a b58c369e29
commit 7074262f7d
1 changed files with 3 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
cn-Book/2.处理文本数据.md
View File

@@ -448,13 +448,13 @@ tokenizer = SimpleTokenizerV2(vocab)
print(tokenizer.encode(text))
```
这将输出以下token ID列表：
这将打印以下词元ID：
```
[1160, 5, 362, 1155, 642, 1000, 10, 1159, 57, 1013, 981, 1009, 738, 1013, 1160, 7]
[1131, 5, 355, 1126, 628, 975, 10, 1130, 55, 988, 956, 984, 722, 988, 1131, 7]
```
从上面的内容可以看出，token ID 列表中包含了 1159 ,它对应于 <|endoftext|> 分隔token，以及两个 1160 ，用于表示未知单词。
从上面的内容可以看出，token ID 列表中包含了 1130 ,它对应于 <|endoftext|> 分隔token，以及两个 1131 ，用于表示未知单词。
```python
print(tokenizer.decode(tokenizer.encode(text)))