Announcing pg_tiktoken: A Postgres Extension for Fast BPE Tokenization

T1Covered by 1 source · first reported by Databricks / ClickHouse / DuckDB / Supabase / Neon / PlanetScale at 14 Mar 2023, 17:04 UTC

We’re excited to announce the release of the pg_tiktoken extension on Neon. This new Postgres extension provides fast and efficient tokenization using the BPE (Byte Pair Encoding) algorithm. pg_tiktoken is a wrapper around OpenAI’s tokenizer, known for its speed and performance i...