14 Oct 2026 · inteligencia artificial, infrastructure, testes · 6 min
How to tell if the model got worse after an upgrade
Upgrading a model version is easy, comparing the two is the annoying part. The short answer: build a small set of questions with checkable answers, run the same set on the old and the new version several times, and compare the range rather than a single number.
The same model, same version, produces different scores each round. That is not the model changing behaviour: it is the set answering differently depending on time of day, machine load, and the sampling mood of the run.
The set: 20 questions you can check
Every question has an answer that can be verified without opinion: arithmetic, sequences, units, dates, hexadecimal, XOR, SQL, regex, shell, HTTP status. Nothing like “write a poem about the sea”, which has no right answer and makes whoever evaluates judge style instead of correctness.
The script ships with all 20 built in, each line carrying its check rule. Two of them:
("p01", "aritmetica",
"Calcule 47 * 83 - 19. Responda so com o numero.",
"numero=3882"),
("p11", "regex",
"Escreva apenas a expressao regular que casa um IP v4 como 192.168.0.1, "
"sem outros caracteres.",
"regex-ip"),
The regex-ip rule deserves explanation, because it is the sneakiest trap in a set like this. Judging a regex by the shape of the text fails a correct answer: both \d and [0-9] match a valid IP. The right way is to compile whatever the model returned and test the behaviour:
def regex_para_ip(expressao: str) -> bool:
padrao = expressao.strip().strip("`")
try:
rx = re.compile(padrao)
except re.error:
return False
return bool(rx.fullmatch("192.168.0.1")) and not rx.match("192.168.0.1x") \
and not rx.match("999.1.1.1")
That kept me from failing both models on a question they got right.
The script
Save it as avaliacao.py. It installs nothing and uses only the standard library. BASE_URL is an argument, because the question is not about any particular model.
#!/usr/bin/env python3
"""avaliacao.py: 20 perguntas de resposta conferivel, rodadas repetidas e comparacao.
A ideia e simples: um placar so nao diz nada. O mesmo modelo, na mesma versao,
erra em perguntas diferentes a cada rodada. Entao o script roda o conjunto N vezes
por modelo, guarda cada rodada em um arquivo, e a comparacao mostra a faixa,
nao so o ultimo numero.
export CHAVE_API="sua-chave"
python3 avaliacao.py rodar gpt-4o-mini --rodadas 3
python3 avaliacao.py rodar llama-3.1-8b --rodadas 3
python3 avaliacao.py comparar
O BASE_URL e o MODELO sao argumentos, porque a pergunta nao e sobre modelo nenhum:
e sobre ter um metodo antes de trocar de versao.
"""
from __future__ import annotations
import argparse
import glob
import json
import os
import pathlib
import re
import statistics
import sys
import time
import unicodedata
import urllib.error
import urllib.request
BASE_URL = os.environ.get("BASE_URL", "https://api.openai.com/v1")
# Orcamento generoso de proposito: com 900 tokens o raciocinio do modelo consome
# tudo e a resposta volta vazia com finish_reason "length". Isso acontecia em
# 3 das 20 perguntas na medicao deste texto.
MAX_TOKENS = 4000
# --------------------------------------------------------------- o conjunto
# Cada linha traz a resposta conferivel. "contem" ignora acento e caixa.
PERGUNTAS = [
("p01", "aritmetica",
"Calcule 47 * 83 - 19. Responda so com o numero.",
"numero=3882"),
("p02", "ordenacao",
"Ordene estes numeros do menor para o maior e responda so com eles, separados por "
"virgula: 91, 17, 4, 63, 28, 55",
"numero=4, 17, 28, 55, 63, 91"),
("p03", "sequencia",
"Complete a sequencia e responda so com o proximo numero: 2, 6, 18, 54, 162, ...",
"numero=486"),
("p04", "unidades",
"Um processo consome 2,5 GB de RAM, e 1 GB vale 1000 MB. Quantos MB sao isso? "
"Responda so com o numero.",
"numero=2500"),
("p05", "data",
"Quantos dias tem o mes de fevereiro de 2028, ano bissexto? Responda so com o numero.",
"numero=29"),
("p06", "premissa",
"Se todo retangulo tem quatro lados, e esta figura e um retangulo, quantos lados "
"ela tem? Responda so com o numero.",
"numero=4"),
("p07", "prosao",
"Quantos segundos tem 15 minutos? Responda so com o numero.",
"numero=900"),
("p08", "percentual",
"De 1200 requisicoes, 47 por cento falharam. Quantas requisicoes falharam? "
"Responda so com o numero.",
"numero=564"),
("p09", "texto",
"Qual destes e um comando que existe no coreutils do Linux: 'troca-tudo' ou 'cp'? "
"Responda so com o comando.",
"contem=cp"),
("p10", "sql",
"Escreva apenas o comando SQL que lista as 10 contas com maior gasto da tabela "
"conta(id, nome, gasto). Sem explicacao.",
"contem=order by"),
("p11", "regex",
"Escreva apenas a expressao regular que casa um IP v4 como 192.168.0.1, "
"sem outros caracteres.",
"regex-ip"),
("p12", "shell",
"Escreva apenas o comando bash que mostra as 5 maiores linhas de tamanho em bytes "
"de todos os .log do diretorio atual.",
"contem=sort"),
("p13", "bits",
"Quantos bytes tem 1 GiB, considerando 1 GiB = 2 elevado a 30? Responda so com "
"o numero.",
"numero=1073741824"),
("p14", "geometria",
"Um retangulo tem area 240 cm2 e largura 15 cm. Quanto mede o comprimento, em cm? "
"Responda so com o numero.",
"numero=16"),
("p15", "bits",
"Em hexadecimal, quanto e 255 mais 1? Responda so com o valor em hexadecimal, "
"precedido de 0x.",
"numero=0x100"),
("p16", "bits",
"Quanto e 1234 XOR 5678, em decimal? Responda so com o numero.",
"numero=4860"),
("p17", "shell",
"Escreva apenas o comando shell que conta quantas linhas do arquivo access.log "
"contem a palavra ' 500 '. Sem explicacao.",
"contem=grep"),
("p18", "http",
"Qual codigo de status HTTP significa 'Sem Autorizacao'? Responda so com o numero.",
"numero=401"),
("p19", "logica",
"Se chove, o cha fica molhado. Hoje o cha esta seco. Choveu hoje? "
"Responda apenas 'sim' ou 'nao'.",
"contem=nao"),
("p20", "inversao",
"Inverta a ordem destes quatro termos e responda so com eles, separados por virgula: "
"chave publica, hash, chave privada, assinatura",
"contem=assinatura, chave privada, hash, chave publica"),
]
def normaliza(texto: str) -> str:
"""Minuscula, sem acento, espacos colapsados e sem espaco em volta de virgula.
O espaco depois da virgula nao pode decidir ponto: o mesmo modelo responde
'4,17,28' numa rodada e '4, 17, 28' na outra, e a resposta e a mesma.
"""
n = unicodedata.normalize("NFD", texto.lower())
n = "".join(c for c in n if unicodedata.category(c) != "Mn")
n = re.sub(r"\s+", " ", n)
n = re.sub(r"\s*,\s*", ",", n)
return n.strip()
def regex_para_ip(expressao: str) -> bool:
"""A resposta para a p11 e certa se casar 192.168.0.1, recusar 999.1.1.1 e recusar
192.168.0.1x. Julgar pela forma do texto reprova resposta certa: tanto \\d quanto
[0-9] casam IP, e na medicao deste texto os dois modelos responderam certo."""
padrao = expressao.strip().strip("`")
try:
rx = re.compile(padrao)
except re.error:
return False
return bool(rx.fullmatch("192.168.0.1")) and not rx.match("192.168.0.1x") \
and not rx.match("999.1.1.1")
def conferivel(regra: str, texto: str) -> bool:
if regra == "regex-ip":
return regex_para_ip(texto)
return normaliza(regra.split("=", 1)[1]) in normaliza(texto)
def uma_rodada(modelo: str, base: str, chave: str, cabecalhos: dict[str, str],
max_tokens: int) -> dict:
acertos, disciplina, falhas, sem_resposta = 0, {}, [], []
for pid, _categoria, pergunta, regra in PERGUNTAS:
t0 = time.monotonic()
try:
cabec = {"Authorization": f"Bearer {chave}",
"Content-Type": "application/json", **cabecalhos}
corpo = json.dumps({
"model": modelo,
"messages": [{"role": "user", "content": pergunta}],
"temperature": 0,
"max_tokens": max_tokens,
}).encode()
pedido = urllib.request.Request(
f"{base.rstrip('/')}/chat/completions", data=corpo, headers=cabec)
with urllib.request.urlopen(pedido, timeout=180) as r:
d = json.load(r)
except urllib.error.HTTPError as e:
falhas.append(f"{pid}: HTTP {e.code} "
f"{e.read()[:80].decode(errors='replace').strip()}")
disciplina[pid] = "erro"
continue
except Exception as e: # noqa: BLE001
falhas.append(f"{pid}: {type(e).__name__} {e}")
disciplina[pid] = "erro"
continue
escolha = (d.get("choices") or [{}])[0]
texto = ((escolha.get("message") or {}).get("content") or "").strip()
if conferivel(regra, texto):
acertos += 1
disciplina[pid] = "ok"
elif escolha.get("finish_reason") == "length" and not texto:
# o modelo gastou o orcamento inteiro pensando e nao sobrou resposta
disciplina[pid] = "sem-resposta"
sem_resposta.append(pid)
else:
disciplina[pid] = "errou"
del t0
return {"acertos": acertos, "de": len(PERGUNTAS), "disciplina": disciplina,
"falhas_api": falhas, "sem_resposta": sem_resposta}
def cmd_rodar(args) -> int:
chave = args.chave or os.environ.get("CHAVE_API", "")
if not chave:
print("avaliacao.py: falta a chave. Faca assim:")
print(" export CHAVE_API=\"sua-chave\"")
print(" python3 avaliacao.py rodar MODELO_ANTIGO --rodadas 3")
print(" python3 avaliacao.py rodar MODELO_NOVO --rodadas 3")
print(" python3 avaliacao.py comparar")
return 1
cabecalhos = {}
for bruto in args.cabecalho:
if ":" not in bruto:
print(f"cabecalho invalido, use 'Nome: valor': {bruto}")
return 1
nome, valor = bruto.split(":", 1)
cabecalhos[nome.strip()] = valor.strip()
rodadas = []
for n in range(1, args.rodadas + 1):
print(f" rodada {n}/{args.rodadas} de {args.modelo} ...", flush=True)
r = uma_rodada(args.modelo, args.base, chave, cabecalhos, args.max_tokens)
rodadas.append(r)
extra = ""
if r["falhas_api"] or r["sem_resposta"]:
extra = (f", {len(r['falhas_api'])} falha(s) de API"
+ (f", {len(r['sem_resposta'])} sem resposta" if r["sem_resposta"] else ""))
print(f" {r['acertos']}/{len(PERGUNTAS)}{extra}", flush=True)
time.sleep(args.pausa)
arquivo = pathlib.Path(f"avaliacao-{args.modelo.replace('/', '_')}.json")
arquivo.write_text(json.dumps(
{"modelo": args.modelo, "base": args.base, "rodadas": rodadas},
ensure_ascii=False, indent=1), encoding="utf-8")
notas = [r["acertos"] for r in rodadas]
erros = sum(len(r["falhas_api"]) for r in rodadas)
print(f"\n{args.modelo}: {notas}")
print(f" media {statistics.mean(notas):.1f} "
f"min {min(notas)} max {max(notas)} desvio {statistics.pstdev(notas):.1f}")
print(f" falhas de API somando as rodadas: {erros}")
if erros:
print(" PLACAR INVALIDO: com chamada falhando, o numero mede a sua rede, nao o modelo")
for falha in rodadas[0]["falhas_api"][:1]:
print(f" primeiro erro: {falha}")
print(" se o provedor pedir cabecalho extra, use --cabecalho \"Nome: valor\"")
print(f" gravado em {arquivo}")
return 0
def cmd_comparar(args) -> int:
arquivos = sorted(glob.glob("avaliacao-*.json"))
if not arquivos:
print("nenhum arquivo avaliacao-*.json ainda. Rode uma vez em cada modelo:")
print(" CHAVE_API=\"sua-chave\" python3 avaliacao.py rodar MODELO_ANTIGO --rodadas 3")
print(" CHAVE_API=\"sua-chave\" python3 avaliacao.py rodar MODELO_NOVO --rodadas 3")
print(" python3 avaliacao.py comparar")
return 0
print(f"{'modelo':28} {'rodadas':>18} {'media':>6} {'min':>4} {'max':>4} "
f"{'desvio':>7} {'erros':>6}")
resumo = []
for arq in arquivos:
d = json.loads(pathlib.Path(arq).read_text(encoding="utf-8"))
notas = [r["acertos"] for r in d["rodadas"]]
erros = sum(len(r.get("falhas_api", [])) for r in d["rodadas"])
media = statistics.mean(notas)
desvio = statistics.pstdev(notas) if len(notas) > 1 else 0.0
resumo.append((d["modelo"], media, min(notas), max(notas), desvio, notas, erros))
print(f"{d['modelo'][:28]:28} {str(notas):>18} {media:6.1f} {min(notas):4} "
f"{max(notas):4} {desvio:7.1f} {erros:6}")
print()
for modelo, media, _mn, _mx, _desvio, notas, erros in resumo:
print(f" {modelo}: media {media:.1f}, cada rodada foi {notas}"
+ (" INVALIDO" if erros else ""))
if len(resumo) == 2:
a, b = resumo
print()
if a[6] or b[6]:
print(" uma das rodadas teve chamada falhando: a diferenca mede a rede, "
"nao os modelos")
elif abs(a[1] - b[1]) <= max(a[4], b[4]):
print(f" diferenca de {abs(a[1] - b[1]):.1f} pontos, folga de ruido "
f"{max(a[4], b[4]):.1f}: INDISTINGUIVEL com estas rodadas")
else:
print(f" diferenca de {abs(a[1] - b[1]):.1f} pontos acima da folga de "
f"{max(a[4], b[4]):.1f}: diferenca real")
print(" para fechar: mais rodadas nos dois, e so se a diferenca continuar acima "
"da folga vale concluir")
return 0
def main() -> int:
ap = argparse.ArgumentParser(description="20 perguntas conferiveis para comparar modelos")
ap.add_argument("--chave", default="", help="ou use CHAVE_API no ambiente")
ap.add_argument("--base", default=BASE_URL, help=f"padrao {BASE_URL}")
ap.add_argument("--cabecalho", action="append", default=[],
help="'Nome: valor', repetivel")
ap.add_argument("--max-tokens", type=int, default=MAX_TOKENS)
sub = ap.add_subparsers(dest="cmd")
r = sub.add_parser("rodar", help="roda o conjunto contra um modelo")
r.add_argument("modelo")
r.add_argument("--rodadas", type=int, default=3)
r.add_argument("--pausa", type=float, default=1.0)
r.set_defaults(func=cmd_rodar)
c = sub.add_parser("comparar", help="compara os arquivos ja gravados")
c.set_defaults(func=cmd_comparar)
args = ap.parse_args()
if not getattr(args, "func", None):
# sem subcomando o script ainda tem de responder algo util, em vez de sair calado
ap.print_help()
print()
print("comeco:")
print(" export CHAVE_API=\"sua-chave\"")
print(" python3 avaliacao.py rodar MODELO_ANTIGO --rodadas 3")
print(" python3 avaliacao.py rodar MODELO_NOVO --rodadas 3")
print(" python3 avaliacao.py comparar")
return 0
return args.func(args)
if __name__ == "__main__":
sys.exit(main())```
Running it on both models, then comparing:
```bash
export CHAVE_API="sua-chave"
python3 avaliacao.py rodar MODELO_ANTIGO --rodadas 3
python3 avaliacao.py rodar MODELO_NOVO --rodadas 3
python3 avaliacao.py comparar
What the measurement said
I ran the 20 questions against two models, 3 rounds each, 120 calls, none failed. With the default 4000 token ceiling, both scored 20 out of 20 in all three rounds. The set hit the ceiling and separated nobody, and that is the honest result: 20 questions like these do not help you choose between these two models.
The parameter that moved the score most was a different one. A 900 token ceiling dropped both averages to 18.3 and produced 8 empty answers, all with finish_reason equal to length: the model spent the budget thinking and no text was left, as the endpoint documentation describes. Compare versions with a short ceiling and part of the difference you blame on the model is your parameter.
How to verify
The command and the expected output with both models run:
python3 avaliacao.py comparar
modelo rodadas media min max desvio erros
cat-jam-free [20, 20, 20] 20.0 20 20 0.0 0
space-bunny [20, 20, 20] 20.0 20 20 0.0 0
cat-jam-free: media 20.0, cada rodada foi [20, 20, 20]
space-bunny: media 20.0, cada rodada foi [20, 20, 20]
diferenca de 0.0 pontos, folga de ruido 0.0: INDISTINGUIVEL com estas rodadas
The erros column is the first one to look at: it sums the calls that failed, and a score with failed calls measures your network, not the model. In the first version of my script this was missing, and the result was 0 out of 20 with no explanation.
To see the noise range of a single model, change the round count:
python3 avaliacao.py rodar MODELO --rodadas 5
When it goes wrong
Every call fails with HTTP 400 and the message MissingSessionID. That gateway wants a session header on top of the key: pass it with --cabecalho "x-opencode-session: my-session". Without it the script marks everything as an error and warns the score is invalid.
Two rounds give very different numbers. That is the normal case, and the reason there are several rounds. Go to 5 and compare the average against the spread.
The model is right and the script marks it wrong. The rule is too narrow, and that happened to me on two questions before I fixed the set: a rule requiring \d failed a correct regex using [0-9], and a false premise about spiders measured disagreement rather than quality. Before blaming the model, test the checker against a response you know is right.
Comparison of ways to compare
| approach | what it measures | cost | when to use |
|---|---|---|---|
| one score only | the last draw | 1 round | only as a guess |
| average of 3 rounds | mean and spread | 3 rounds | routine comparison |
| 20 checkable questions | objective correctness | 20 calls | minimum bar |
| a set from your domain | what matters to you | you write it | specific task |
| LLM-as-judge | style and quality | costly, subjective | criteria written first |
A set from your own domain beats any generic one. The 20 questions here show the method; for your service, write 20 questions from it and run the same script.
FAQ
Do I need 20 questions? Fewer than 10 will not separate models: the spread of a single round already eats the difference you are trying to measure. Above 30 the call cost matters, so pick the questions closest to your use.
How do I know the difference I found is real? Only after repeating and it staying above the spread. With 3 rounds, 1 point against a 0.8 spread closes nothing. The script says so in plain text so nobody skips it.
I change the set and the model together, do I redo it? Yes, and the old set loses its value. The comparison only holds with the questions fixed across both versions.
Does this work with any provider? With any endpoint speaking the chat completion format: BASE_URL is an argument for that. Some gateways want extra headers, and the script has --cabecalho.
What about open-ended questions? They do not belong here. An open answer needs a criterion written down first, and a written criterion generally turns into a human judge, slower and more expensive than reading three rounds of scores. The path, if you want to measure that anyway, is a set from your own domain with an expected answer.
Sources
- Chat endpoint reference, OpenAI, checked on October 14, 2026. The definition of
finish_reasonequal tolength. - Working with evals, OpenAI, checked on October 14, 2026. Describe the task before measuring, plus the deprecation notice.
- Evaluation best practices, OpenAI, checked on October 14, 2026. The five design steps and the nondeterminism note.
If you already measure call cost per consumer and want that in the method, see measuring things on your own machine. And if you still compare versions by vibes, replacing the server password with an SSH key shows the other side: switching without measuring is the fastest way to find out you regressed.