Skip to content

Instantly share code, notes, and snippets.

@the-crypt-keeper

the-crypt-keeper/can-ai-code-v2-results.csv

Created June 6, 2023 23:27
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save the-crypt-keeper/6412e678dccda1a93785052aa8893576 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save the-crypt-keeper/6412e678dccda1a93785052aa8893576 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
can-ai-code-v2-results.csv
idx Languages Model Params Template Passed Score
37 python openai-chatgpt precise chat-generic 65 1.0
38 javascript openai-chatgpt precise chat-generic 65 1.0
52 javascript ggml-vicuna-13B-1.1-q5 precise Vicuna-1p1-v2 58 0.8923076923076924
31 javascript ai21-j2-jumbo-instruct precise chat-generic 57 0.8769230769230769
4 python ggml-vicuna-13B-1.1-q5 precise Vicuna-1p1 57 0.8769230769230769
5 javascript ggml-vicuna-13B-1.1-q5 precise Vicuna-1p1 57 0.8769230769230769
30 python ai21-j2-jumbo-instruct precise chat-generic 57 0.8769230769230769
35 python tsumeone-llama-30b-supercot-4bit-cuda precise Alpaca 56 0.8615384615384616
26 python airoboros-13b-gpt4.ggmlv3.q5 precise Vicuna-FastChat 55 0.8461538461538461
0 python cohere-command-nightly precise chat-generic 55 0.8461538461538461
7 javascript nous-hermes-13b.ggmlv3.q5 precise Alpaca-Input 54 0.8307692307692308
20 python HuggingFaceH4-starchat-playground-hf-space precise chat-generic 53 0.8153846153846154
11 javascript HuggingFaceH4-starchat-playground-hf-space precise starchat 53 0.8153846153846154
51 python ggml-vicuna-13B-1.1-q5 precise Vicuna-1p1-v2 52 0.8
33 python WizardLM-13B-1.0.ggmlv3.q5 precise Wizard-Vicuna 51 0.7846153846153846
50 javascript nous-hermes-13b.ggmlv3.q5 precise Alpaca 51 0.7846153846153846
55 python WizardLM-7B-uncensored.ggmlv3.q5 precise Wizard-Vicuna 51 0.7846153846153846
22 python TheBloke-VicUnlocked-30B-LoRA-GPTQ precise Vicuna-1p1-v2 51 0.7846153846153846
21 javascript HuggingFaceH4-starchat-playground-hf-space precise chat-generic 51 0.7846153846153846
54 javascript Manticore-13B-Chat-Pyg-Guanaco-GGML-q4 precise Manticore-YearZero 50 0.7692307692307693
36 javascript tsumeone-llama-30b-supercot-4bit-cuda precise Alpaca 50 0.7692307692307693
40 javascript Manticore-13B-Chat-Pyg-Guanaco-GGML-q4 mirostat Manticore-YearZero 50 0.7692307692307693
34 javascript WizardLM-13B-1.0.ggmlv3.q5 precise Wizard-Vicuna 50 0.7692307692307693
1 javascript cohere-command-nightly precise chat-generic 49 0.7538461538461538
49 python nous-hermes-13b.ggmlv3.q5 precise Alpaca 49 0.7538461538461538
45 python TheBloke-VicUnlocked-30B-LoRA-GPTQ precise Vicuna-1p1 49 0.7538461538461538
6 python nous-hermes-13b.ggmlv3.q5 precise Alpaca-Input 48 0.7384615384615385
10 python HuggingFaceH4-starchat-playground-hf-space precise starchat 48 0.7384615384615385
25 javascript Wizard-Vicuna-13B-Uncensored.ggmlv3.q5 precise Wizard-Vicuna 48 0.7384615384615385
46 javascript TheBloke-VicUnlocked-30B-LoRA-GPTQ precise Vicuna-1p1 48 0.7384615384615385
12 python TheBloke-WizardLM-30B-Uncensored-GPTQ precise Wizard-Vicuna 48 0.7384615384615385
23 javascript TheBloke-VicUnlocked-30B-LoRA-GPTQ precise Vicuna-1p1-v2 48 0.7384615384615385
27 javascript airoboros-13b-gpt4.ggmlv3.q5 precise Vicuna-FastChat 47 0.7230769230769231
29 javascript ggml-vicuna-7b-1.1-q5 precise Vicuna-1p1-v2 47 0.7230769230769231
13 javascript TheBloke-WizardLM-30B-Uncensored-GPTQ precise Wizard-Vicuna 47 0.7230769230769231
8 python bigcode-starcoder precise starcoder-fim-input 46 0.7076923076923077
28 python ggml-vicuna-7b-1.1-q5 precise Vicuna-1p1-v2 45 0.6923076923076923
9 javascript bigcode-starcoder precise starcoder-fim-input 45 0.6923076923076923
2 python ggml-vicuna-7b-1.1-q5 precise Vicuna-1p1 44 0.676923076923077
53 python Manticore-13B-Chat-Pyg-Guanaco-GGML-q4 precise Manticore-YearZero 43 0.6615384615384615
39 python Manticore-13B-Chat-Pyg-Guanaco-GGML-q4 mirostat Manticore-YearZero 43 0.6615384615384615
41 python Manticore-13B.ggmlv3.q5 precise Wizard-Vicuna 42 0.6461538461538462
16 python llama-deus-7b-v3.ggmlv3.q5 precise Alpaca-Input 42 0.6461538461538462
15 javascript llama-deus-7b-v3.ggmlv3.q5 precise Alpaca 42 0.6461538461538462
43 python guanaco-7B.ggmlv3.q5 precise Guanaco 41 0.6307692307692307
44 javascript guanaco-7B.ggmlv3.q5 precise Guanaco 41 0.6307692307692307
3 javascript ggml-vicuna-7b-1.1-q5 precise Vicuna-1p1 41 0.6307692307692307
48 javascript Manticore-13B.ggmlv3.q5 precise Manticore 41 0.6307692307692307
42 javascript Manticore-13B.ggmlv3.q5 precise Wizard-Vicuna 40 0.6153846153846154
19 javascript guanaco-13B.ggmlv3.q5 precise Guanaco 39 0.6
32 python bigcode-tiny precise starcoder-fim-input 38 0.5846153846153846
17 javascript llama-deus-7b-v3.ggmlv3.q5 precise Alpaca-Input 38 0.5846153846153846
14 python llama-deus-7b-v3.ggmlv3.q5 precise Alpaca 38 0.5846153846153846
56 javascript WizardLM-7B-uncensored.ggmlv3.q5 precise Wizard-Vicuna 37 0.5692307692307692
47 python Manticore-13B.ggmlv3.q5 precise Manticore 36 0.5538461538461539
24 python Wizard-Vicuna-13B-Uncensored.ggmlv3.q5 precise Wizard-Vicuna 31 0.47692307692307695
18 python guanaco-13B.ggmlv3.q5 precise Guanaco 29 0.4461538461538462
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment