This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| -- Generated Lang model program. | |
| -- Source architecture: qwen2 | |
| -- Layers: 24 | |
| -- Plan operations: 508 | |
| -- The model constants below are generated from GGUF metadata. The function | |
| -- composition is ordinary Lang; kernel launches are supplied by typeclass | |
| -- evidence and can be replaced one family at a time. | |
| import Lang.Prelude | |
| import Lang.Prelude.Open |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| const { GPT4Tokenizer } = require('gpt4-tokenizer') | |
| const json = require('./conversations.json') | |
| const tokenizer = new GPT4Tokenizer({type:'gpt-3'}) | |
| const nodes = [] | |
| for (const row of json) { | |
| let id = row.current_node | |
| const { mapping } = row | |
| while (id) { | |
| const node = mapping[id] | |
| if (node.message) { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Converts OpenAI compatible function calling JSON schema to a prompt that instructs the LLM to return | |
| # a JSON object that is a choice of a function call conforming to one of the functions or a message reply | |
| def convert_schema_to_typescript(schema): | |
| if not schema: | |
| return 'any' | |
| if '$ref' in schema: | |
| return schema['$ref'].replace('#/definitions/', '') |