CalvinBackup/P4RS3LT0NGV3
1
0
Fork 0
mirror of https://github.com/elder-plinius/P4RS3LT0NGV3.git synced 2026-09-22 17:20:45 +02:00
Code Issues Packages Projects Releases Wiki Activity

Tokenizer tab: add OpenAI encodings via browser-safe gpt-tokenizer dynamic import; UI: fit all five Tokenade preset buttons on one line (tighter buttons)

Browse Source
This commit is contained in:
EP
2025-08-20 18:00:29 -07:00
parent 030d90bcca
commit a3d4fc33df
3 changed files with 17 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
index.html
+4 -1
View File
@@ -204,7 +204,10 @@
<select v-model="tokenizerEngine" @change="runTokenizer">
<option value="byte">UTF-8 bytes</option>
<option value="word">Naive words</option>
<option value="gpt3">Experimental: gpt-3-encoder (if available)</option>
<option value="cl100k">OpenAI: cl100k_base (GPT‑3.5/4)</option>
<option value="o200k">OpenAI: o200k_base (GPT‑4o)</option>
<option value="p50k">OpenAI: p50k_base</option>
<option value="r50k">OpenAI: r50k_base</option>
</select>
</label>
</div>