Thomas Schranz tosh

Official site (kparc.com)

I have a parquet file named ah.parquet.

It contains Apple Health data and has the following columns:

Model	AGIEval	GPT4All	TruthfulQA	Bigbench	Average
pandafish-3-7B-32k	40.85	73.57	56.3	42.17	53.22

Task	Version	Metric	Value		Stderr
agieval_aqua_rat	0	acc	20.47	±	2.54
		acc_norm	20.87	±	2.55
agieval_logiqa_en	0	acc	34.10	±	1.86

Model	AGIEval	GPT4All	TruthfulQA	Bigbench	Average
pandafish-2-7b-32k	40.8	73.35	57.46	42.69	53.57

Task	Version	Metric	Value		Stderr
agieval_aqua_rat	0	acc	22.05	±	2.61
		acc_norm	19.69	±	2.50
agieval_logiqa_en	0	acc	35.94	±	1.88

Model	AGIEval	GPT4All	TruthfulQA	Bigbench	Average
dolphin-2.8-mistral-7b-v02	38.99	72.22	51.96	40.41	50.9

Task	Version	Metric	Value		Stderr
agieval_aqua_rat	0	acc	21.65	±	2.59
		acc_norm	20.47	±	2.54
agieval_logiqa_en	0	acc	35.79	±	1.88

Model	AGIEval	GPT4All	TruthfulQA	Bigbench	Average
Mistral-7B-Instruct-v0.2	38.5	71.64	66.82	42.29	54.81

Task	Version	Metric	Value		Stderr
agieval_aqua_rat	0	acc	23.62	±	2.67
		acc_norm	22.05	±	2.61
agieval_logiqa_en	0	acc	36.10	±	1.88

Model	AGIEval	GPT4All	TruthfulQA	Bigbench	Average
HeatherSpellGen3	44.88	76.87	78.3	49.89	62.48

Task	Version	Metric	Value		Stderr
agieval_aqua_rat	0	acc	27.56	±	2.81
		acc_norm	25.20	±	2.73
agieval_logiqa_en	0	acc	39.02	±	1.91

Model	AGIEval	GPT4All	TruthfulQA	Bigbench	Average
pandafish-dt-7b	45.24	77.19	78.41	49.76	62.65

Task	Version	Metric	Value		Stderr
agieval_aqua_rat	0	acc	27.95	±	2.82
		acc_norm	26.38	±	2.77
agieval_logiqa_en	0	acc	39.32	±	1.92

	import json
	import sys
	from subprocess import getoutput as run_shell
	from urllib.request import Request, urlopen

	MODEL = "gpt-5.6"
	CONTEXT_WINDOW_TOKENS = 1_050_000

	endpoint_url = sys.argv[1]
	history = []

	import json,sys;from subprocess import getoutput as sh;from urllib.request import Request as R,urlopen
	url=sys.argv[1];h=[];b=dict(model="gpt-5.6",input=h,tools=[dict(type="custom",name="sh")])
	while p:=input("> "):
	h+=[dict(role="user",content=p)];H={"Content-Type":"application/json"}
	while True:
	o=(r:=json.load(urlopen(R(url,json.dumps(b).encode(),H))))["output"]
	h+=o;c=[i for i in o if i["type"]=="custom_tool_call"];z=r["usage"]["total_tokens"]/10500
	if not c:print(o[-1]["content"][0]["text"],f'\n[{z:06.3f}%]');break
	h+=[dict(type="custom_tool_call_output",call_id=i["call_id"],output=sh(i["input"])) for i in c]