Search results
461 packages found
A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 / Claude Instant / Claude 2
- BPE
- encoder
- decoder
- tokenizer
- GPT
- GPT-2
- GPT-3
- GPT-3.5
- GPT-4
- NLP
- Natural Language Processing
- Text Generation
- OpenAI
- Machine Learning
Tokenizer inspired by Laravel's Sanctum
A class Tokenizer to convert text documents into sequences of tokens
Wix Restaurants credit-cards tokenizer
TS tokenizer for Mistral-based LLMs
Isomorphic utilities for GPT-3 tokenization and prompt building.
A React Native supported JavaScript implementation of Japanese morphological analyzer
JavaScript implementation of Japanese morphological analyzer
Fastly VCL tokenizer
Tokenizes a string that represents a regular expression.
Simple synchronous string tokenizer using Regex
Simple, but powerful lexical scanner that is a more minimal implementation of X-Scanner
- y-scanner
- yscanner
- x-scanner
- xscanner
- stringscanner
- scanner
- string
- text
- textscanner
- lex
- lexer
- lexical
- parse
- parser
- View more
Simple algorithm to tokenize Chinese texts into words using CC-CEDICT.
A CLI tool to concatenate all text files in your CWD with headers for GPT prompt engineering.
- text
- concatenate
- CLI
- Current Working Directory
- GPT
- tokens
- tokenizer
- ChatGPT
- prompt engineering
- token-count
- text manipulation
- file concatenation
- command line interface
- GPT-3
- View more
TS tokenizer for Mistral-based LLMs
A streaming JSON tokenizer
Time a JavaScript tokenizer
A tokenizer for Google-like search queries
A small ECMAScript parser, tokenizer and minifier written in JavaScript.