rtiktoken/DESCRIPTION at master · DavZim/rtiktoken · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Package: rtiktoken
Title: A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models
Version: 0.11.0-1
Authors@R: c(
    person("David", "Zimmermann-Kollenda", , "david_j_zimmermann@hotmail.com", role = c("aut", "cre")),
    person("Roger Zurawicki", role = "aut", comment = "tiktoken-rs Rust library"),
    person("Authors of the dependent Rust crates", role = "aut", comment = "see AUTHORS file")
  )
Description: A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.
License: MIT + file LICENSE
URL: https://davzim.github.io/rtiktoken/, https://github.com/DavZim/rtiktoken/
BugReports: https://github.com/DavZim/rtiktoken/issues
Suggests:
    testthat (>= 3.0.0)
SystemRequirements: Cargo (Rust's package manager), rustc >= 1.65.0
Encoding: UTF-8
Roxygen: list(markdown = TRUE)
RoxygenNote: 7.3.3
Config/rextendr/version: 0.5.0
Config/testthat/edition: 3
Config/rtiktoken/MSRV: 1.65.0
Depends:
    R (>= 4.2)