README.md — AI Engine – The Chatbot, AI Framework & MCP for WordPress 3.1.3

ai-engine / vendor / yethee / tiktoken / README.md

ai-engine / vendor / yethee / tiktoken Last commit date

src 10 months ago LICENSE 10 months ago README.md 10 months ago

README.md

75 lines

1	# tiktoken-php
2
3	![Packagist Version](https://img.shields.io/packagist/v/yethee/tiktoken)
4	![Build status](https://img.shields.io/github/actions/workflow/status/yethee/tiktoken-php/ci.yml?branch=master)
5	![License](https://img.shields.io/github/license/yethee/tiktoken-php)
6
7	This is a port of the [](https://github.com/openai/tiktokentiktoken](https://github.com/openai/tiktoken](https://github.com/openai/tiktoken).
8
9	## Installation
10
11	```bash
12	$ composer require yethee/tiktoken
13	```
14
15	## Usage
16
17	```php
18
19	use Yethee\Tiktoken\EncoderProvider;
20
21	$provider = new EncoderProvider();
22
23	$encoder = $provider->getForModel('gpt-3.5-turbo-0301');
24	$tokens = $encoder->encode('Hello world!');
25	print_r($tokens);
26	// OUT: [9906, 1917, 0]
27
28	$encoder = $provider->get('p50k_base');
29	$tokens = $encoder->encode('Hello world!');
30	print_r($tokens);
31	// OUT: [15496, 995, 0]
32	```
33
34	## Cache
35
36	The encoder uses an external vocabularies, so caching is used by default
37	to avoid performance issues.
38
39	By default, the [directory for temporary files](https://www.php.net/manual/en/function.sys-get-temp-dir.php) is used.
40	You can override the directory for cache via environment variable `TIKTOKEN_CACHE_DIR`
41	or use `EncoderProvider::setVocabCache()`:
42
43	```php
44	use Yethee\Tiktoken\EncoderProvider;
45
46	$encProvider = new EncoderProvider();
47	$encProvider->setVocabCache('/path/to/cache');
48
49	// Using the provider
50	```
51
52	### Disable cache
53
54	You can disable the cache, if there are reasons for this,
55	in one of the following ways:
56
57	* Set an empty string for the environment variable `TIKTOKEN_CACHE_DIR`.
58	* Programmatically:
59
60	```php
61	use Yethee\Tiktoken\EncoderProvider;
62
63	$encProvider = new EncoderProvider();
64	$encProvider->setVocabCache(null); // disable the cache
65	```
66
67	## Limitations
68
69	* Encoding for GPT-2 is not supported.
70	* Special tokens (like `<\|endofprompt\|>`) are not supported.
71
72	## License
73
74	[MIT](./LICENSE)
75