Formats
The files of the suite, the uploads and the checker's output, and the APIs. A runner for a new system follows this.
# math.osp.no: the formats
math.osp.no compares computer algebra systems on a suite of problems with known answers. This is
the contract between its parts:
- the **generator**, which writes the suite (samara, `bench/` in its repository);
- the **runners**, one for each system, which give the suite to the system and write an upload;
- the **site** (`Osp.Math` in the osp.no app), which takes the uploads and shows the results;
- the **graders**, our machines that run samara's **checker**, which fetch grading units from the
site and post the grades back.
The formats are neutral: nothing here is of one system. The answers are text in each system's own
syntax, with the system's name, so that the checker can parse them. Each format has a tag, as
`math-upload/1`; a change that an older reader cannot take comes with a new tag, and the site takes
both while the old one is in use.
This file is served as the page `/formats` of the site. A change to a format changes this file in
the same commit.
## 1. Words
- **Suite**: the problems, in versions, as `2026.1`. Each version has its hash, published.
- **Problem**: a task with a known answer, of one of 14 categories. Its **id** names it forever.
- **Subset**: `full`, every problem of a version; `quick`, about 2,000 of them; and the
**calibration** subset, about 20 problems that every runner runs first, to measure the machine.
- **Level**: how the problem was built. `textbook` is one inverse step from its answer, `medium`
two or three stacked rewrites, `gnarly` four or more.
- **Famous problem**: one of the literature, solved, impossible or open (section 2.3).
- **System**, **version**: a computer algebra system, and its version exactly as the runner reads
it from the system.
- **Run**: one system version's answers to a subset of one suite version, on one machine.
- **Grade**: `solved`, `unsolved`, `wrong`, `timeout` or `error`, given by the checker or, where no
checker is needed, by the site. The site keeps one more, `review` (section 7.4).
- **Unit**: a chunk of a run's answers, graded at once by one grader.
## 2. The suite file
A version of the suite is a file of JSON lines, UTF-8, one problem to a line, each line ended by
`\n`, no blank lines. It is published gzipped, as `suite.jsonl.gz`, beside `quick.jsonl.gz`, the
lines marked `quick` in the same order, and `manifest.json`. Its **hash** is the SHA-256, in lower
case hex, of the file unzipped:
```
gunzip -c suite.jsonl.gz | sha256sum
```
The generator writes the file; the site takes it as it is, checks every line, and refuses the whole
version where a line is wrong. Its lines may come in any order; the site keeps it.
### 2.1 The fields
| field | | |
|---|---|---|
| `id` | required | 1 to 80 of `a-z`, `0-9`, `_`, `.` and `-`, starting with a letter or digit, as `indefinite_integral-000123` |
| `category` | required | one of the 14 below |
| `origin` | required | `generated` or `famous` |
| `method_class` | generated | the kind of method the construction calls for, as `substitution`, snake_case; null for a famous problem |
| `level` | generated | `textbook`, `medium` or `gnarly`; null for a famous problem |
| `statement` | optional | a short sentence, as `Integrate.` |
| `problem` | required | the task: `{"op": ..., "args": {...}}`, section 4 |
| `problem_latex` | optional | the task in LaTeX, as the site shows it |
| `answer` | see below | the known answer, in the syntax of section 3 |
| `answer_latex` | optional | the answer in LaTeX |
| `construction` | generated | how it was built, section 2.2 |
| `famous` | famous | section 2.3 |
| `quick` | optional | true where the problem is in the quick subset |
| `calibration` | optional | true where it is in the calibration subset; these are in the quick subset too |
The categories:
| `category` | |
|---|---|
| `indefinite_integral` | indefinite integrals |
| `definite_integral` | definite integrals |
| `ode` | ordinary differential equations |
| `pde` | partial differential equations |
| `sum` | sums |
| `recurrence` | recurrences |
| `limit` | limits |
| `series` | series |
| `transform` | integral transforms and the z-transform |
| `equation` | equations and inequalities |
| `simplification` | simplification |
| `polynomial` | polynomials |
| `linear_algebra` | linear algebra |
| `special_function` | special functions |
A generated problem has an `answer`, and so has a famous problem that is solved; for one impossible
or open the answer is the known value where there is one (an impossible antiderivative may have one
in special functions), else null.
**An id names one problem forever.** Each version that has an id has the same problem under it:
every field the same but `quick` and `calibration`. A problem that changes gets a new id. The site
refuses a version that changes a problem of a known id, so that the answers to it from older runs
still answer the same problem. A version may leave out problems of earlier ones, and add new ones.
The **levels** are fixed by the construction: a `textbook` problem has 1 step, a `medium` one 2 or
3, a `gnarly` one 4 or more. The site refuses a line where they do not fit.
### 2.2 The construction
A generated problem is built backwards from its answer, the seed, by rewriting steps:
```json
{
"seed": "exp(x^2)",
"steps": [
{"move": "differentiate", "args": {"var": "x"}, "result": "2*x*exp(x^2)"},
{"move": "substitute", "args": {"x": "sqrt(u)"}, "result": "..."}
]
}
```
`seed` is the answer the construction starts from (an expression, section 3), each step's `move` is
the name of its rewrite, snake_case, `args` its parameters, and `result` the expression after it,
the last one's being the problem's. The site shows them on the problem's page. `move` is required;
`args` and `result` are optional.
### 2.3 The famous problems
A famous problem is of the literature: the famous set is collected in samara's
`bench/famous/*.jsonl`, with the fields `id`, `title`, `category`, `status`, `problem`, `answer`,
`claim`, `who`, `year`, `references`, `history`, `partial` and `verified`. The generator writes each
as a suite line of `origin` `famous`, its id `famous-<id>`, its `problem` and `answer` in the forms
of sections 3 and 4, and the rest in `famous`:
| field of `famous` | | |
|---|---|---|
| `title` | required | as `The Basel problem` |
| `status` | required | `solved`, `impossible` or `open` |
| `claim` | optional | what is claimed: for an impossible problem, which kind of closed form does not exist, as `no elementary antiderivative`; for an open one, what is not known |
| `who` | optional | who settled it, or posed it where it is open |
| `year` | optional | the year, a number |
| `references` | optional | a list of texts, each a citation or a URL |
| `history` | optional | a few sentences |
| `partial` | optional | partial results, of an open problem |
| `verified` | optional | true where the answer, or the impossibility, is verified, or a text that tells how |
The grades of a famous problem:
- **solved**: graded as any problem, against its answer.
- **impossible**: a closed form of the impossible kind is wrong, as an elementary antiderivative of
`exp(-x^2)`: the checker grades it `wrong`, as it is (it cannot be right). An answer in terms of
functions not of that kind, as `erf`, is graded against the answer, where there is one.
- **open**: no answer is known. Where the checker finds an answer right, the site does not take it
as solved: it keeps the answer as `review`, shown as "for review", until the admin has looked at
it, and grades it then. An open problem is never solved by a machine alone.
### 2.4 Examples
```json
{"id":"indefinite_integral-000123","category":"indefinite_integral","origin":"generated","method_class":"substitution","level":"textbook","statement":"Integrate.","problem":{"op":"integrate","args":{"integrand":"2*x*exp(x^2)","var":"x"}},"problem_latex":"\\int 2x e^{x^2}\\,dx","answer":"exp(x^2)","answer_latex":"e^{x^2}","construction":{"seed":"exp(x^2)","steps":[{"move":"differentiate","args":{"var":"x"},"result":"2*x*exp(x^2)"}]},"quick":true}
{"id":"famous-basel","category":"sum","origin":"famous","method_class":null,"level":null,"statement":"Find the sum.","problem":{"op":"sum","args":{"term":"1/n^2","var":"n","from":"1","to":"infinity"}},"problem_latex":"\\sum_{n=1}^{\\infty}\\frac{1}{n^2}","answer":"pi^2/6","answer_latex":"\\frac{\\pi^2}{6}","construction":null,"famous":{"title":"The Basel problem","status":"solved","claim":null,"who":"Leonhard Euler","year":1734,"references":["L. Euler, De summis serierum reciprocarum, 1740"],"history":"Posed by Pietro Mengoli in 1650.","partial":null,"verified":true},"quick":true}
```
### 2.5 The manifest
`manifest.json`, written by the site when the version is published:
```json
{
"format": "math-suite/1",
"version": "2026.1",
"published": "2026-11-04T10:00:00Z",
"sha256": "…",
"problems": 60000,
"quick": {"sha256": "…", "problems": 2000},
"calibration": ["indefinite_integral-000007", "…"],
"categories": {"indefinite_integral": 6000, "…": 0},
"notes": "…"
}
```
## 3. Expressions
The expressions of the suite (the problems' arguments, the answers, the seeds) are text in one
neutral syntax, close to what most systems read. A runner translates them into its system's syntax;
the checker reads them too.
- **Numbers**: integers, as `12`, and rationals as quotients, `3/4`. Decimals, as `0.25`, only
where a problem is about floating point, which it says.
- **Symbols**: a letter, then letters, digits and `_`, as `x`, `a1`, `alpha`. The names below
(constants and functions) are not used as symbols.
- **Operators**, from the tightest: `^` (to the right: `a^b^c` is `a^(b^c)`); unary `-` (`-x^2` is
`-(x^2)`); `*` and `/` (to the left); `+` and `-`; the relations `=`, `!=`, `<`, `<=`, `>`, `>=`;
`not`; `and`; `or`. Products are written out: `2*x`, never `2x`.
- **Calls**: `name(a, b)`; **lists**: `[a, b]`; a matrix is a list of its rows, `[[1, 2], [3, 4]]`.
- **Constants**: `pi`, `e` (Euler's number), `i` (the imaginary unit), `infinity`, `euler_gamma`,
`catalan`; `-infinity` for minus infinity, and `undefined` where a limit has none.
- **Unknown functions**: `y(x)`, `u(x, t)`, `a(n)`. A derivative is `diff(y(x), x)`, the second
`diff(y(x), x, 2)`, a mixed one `diff(u(x, t), x, t)`. A derivative's value at a point, as in a
condition, is `subs(diff(y(x), x), x = 0)`, so `y'(0) = 2` is `subs(diff(y(x), x), x = 0) = 2`
and `u_t(x, 0) = 0` is `subs(diff(u(x, t), t), t = 0) = 0`.
- **Unevaluated forms**, in a famous problem's known answer only, where the literature gives the
value as one: `integrate(f, x)`, `integrate(f, x, a, b)`, `sum(term, n, a, b)`, as the sophomore's
dream `integrate(x^(-x), x, 0, 1) = sum(n^(-n), n, 1, infinity)`.
- **Arbitrary constants** in answers: `C1`, `C2`, … of ODEs, PDEs and recurrences; `F1`, `F2`, …
arbitrary functions, as `F1(x - t)`. An antiderivative is up to a constant, which is not written.
- **Assumptions**, in a problem's `args.assumptions`: relations, as `a > 0`, and
`element(n, integers)`, the sets being `naturals` (from 0), `integers`, `rationals`, `reals` and
`complexes`. Where there are none, the symbols are complex, and the variable of integration,
summation or a limit real.
The functions, each by its definition in the [DLMF](https://dlmf.nist.gov):
| function | | DLMF |
|---|---|---|
| `exp(x)`, `log(x)` (natural), `log(b, x)` (base b), `sqrt(x)`, `root(x, n)` (principal) | | 4.2, 4.4 |
| `sin cos tan cot sec csc`, `asin acos atan acot asec acsc`, `atan2(y, x)` | principal branches | 4.14, 4.23 |
| `sinh cosh tanh coth sech csch`, `asinh acosh atanh acoth asech acsch` | | 4.28, 4.37 |
| `abs sign floor ceiling`, `re im arg conjugate` | | |
| `factorial(n)`, `binomial(n, k)`, `pochhammer(a, n)` | | 5.2, 26.3 |
| `gamma(x)`, `gamma(a, x)` (upper incomplete), `log_gamma(x)`, `digamma(x)`, `polygamma(n, x)`, `beta(a, b)` | | 5, 8.2 |
| `erf erfc erfi`, `fresnel_s fresnel_c` | | 7.2 |
| `ei(x)`, `li(x)`, `si(x)`, `ci(x)`, `shi(x)`, `chi(x)`, `expint(n, x)` (E_n) | | 6.2, 8.19 |
| `polylog(s, z)`, `zeta(s)`, `zeta(s, a)` (Hurwitz), `lerch_phi(z, s, a)` | | 25 |
| `lambert_w(x)` (principal), `lambert_w(k, x)` | | 4.13 |
| `bessel_j bessel_y bessel_i bessel_k (nu, x)`, `hankel_1 hankel_2 (nu, x)` | | 10 |
| `airy_ai airy_bi (x)` and their derivatives `airy_ai_prime airy_bi_prime` | | 9 |
| `elliptic_k(m)`, `elliptic_e(m)`, `elliptic_f(phi, m)`, `elliptic_e(phi, m)`, `elliptic_pi(n, m)`, `elliptic_pi(n, phi, m)` | the parameter m = k² | 19.2 |
| `hypergeometric([a1, …], [b1, …], z)` (pFq) | | 16.2 |
| `legendre_p legendre_q (n, x)`, `hermite_h laguerre_l chebyshev_t chebyshev_u (n, x)`, `gegenbauer_c(n, a, x)`, `jacobi_p(n, a, b, x)` | | 18, 14 |
| `heaviside(x)`, `dirac_delta(x)`, `kronecker_delta(m, n)` | | |
| `struve_h struve_l (nu, x)`, `kelvin_ber kelvin_bei (nu, x)` | | 11, 10.61 |
The generator may use a function not here: it adds it to this table, with its DLMF section, in the
same commit, so that the runners and the checker know it.
## 4. The problems' tasks
`problem` is `{"op": ..., "args": {...}}`. The ops, their arguments, and what the answer is:
| category | `op` | `args` | the answer |
|---|---|---|---|
| indefinite integrals | `integrate` | `integrand`, `var` | an antiderivative |
| definite integrals | `integrate` | `integrand`, `var`, `from`, `to` | the value |
| ODEs | `solve_ode` | `equations` (a list), `function` (as `y(x)`) or `functions` (a list, for a system), `conditions` (optional, as `y(0) = 1` or `subs(diff(y(x), x), x = 0) = 2`) | `y(x) = …`, or a list of them; general solutions with `C1`, …; an implicit solution as an equation in `y(x)` and `x` |
| PDEs | `solve_pde` | `equation`, `function` (as `u(x, t)`), `conditions` (optional) | `u(x, t) = …` |
| sums | `sum` | `term`, `var`, `from`, `to` | the value, or the closed form in the bounds |
| | `product` | as `sum` | |
| recurrences | `solve_recurrence` | `equations`, `function` (as `a(n)`), `conditions` (optional) | `a(n) = …` |
| limits | `limit` | `expr`, `var`, `point`, `direction` (`+`, `-`, or none for both) | the value, `infinity`, `-infinity` or `undefined` |
| series | `series` | `expr`, `var`, `point`, `order` (a number) | the terms of order below `order`, without the O-term: the powers of `var - point`, or of `1/var` where `point` is `infinity`, below `order`, fractional powers (Puiseux) and logarithmic factors included |
| transforms | `laplace`, `inverse_laplace`, `fourier`, `inverse_fourier`, `mellin`, `inverse_mellin`, `z_transform`, `inverse_z_transform` | `expr`, `var`, `transform_var`, `convention` (optional) | the transform |
| equations | `solve` | `equations` (a list), `vars` (a list), `domain` (`complex` or `real`) | the solutions: a list of solutions, each a list of `var = value`, as `[[x = -2], [x = 2]]`; a family with its condition, as `[[x = pi*k, element(k, integers)]]`; `[]` where there is none |
| inequalities | `solve` | as `solve`, the relations inequalities, `domain` `real` | a condition on the vars, as `x < -1 or x > 1` |
| simplification | `simplify` | `expr`, `assumptions` (optional) | an equal expression, as small as the known answer |
| polynomials | `factor`, `gcd`, `lcm`, `resultant`, `discriminant`, `partial_fractions`, `groebner`, `minimal_polynomial` | `expr` or `polynomials`, `var` or `vars`, `domain` (as `rationals`), `order` (of `groebner`) | the result: a factorization into the irreducible factors over `domain`, a gcd or lcm up to a unit, a reduced Gröbner basis, the minimal polynomial of the algebraic number `expr` over the rationals in `var` up to a constant |
| linear algebra | `determinant`, `inverse`, `rank`, `nullspace`, `eigenvalues`, `eigenvectors`, `jordan_form`, `solve_linear` | `matrix`, `rhs` (of `solve_linear`) | the result: eigenvalues as a list with multiplicity; eigenvectors as `[[value, multiplicity, [v1, …]], …]`, the vectors a basis of the eigenspace; `jordan_form` the matrix J alone, up to the order of its blocks; `nullspace` a list of basis vectors; `solve_linear` the solution vector, its free parameters `t1`, `t2`, … |
| special functions | `evaluate`, `simplify`, `hyperexpand`, `integrate`, `sum` | `expr`, or as the ops above | a closed form |
The transforms' conventions, unless `convention` says other: Laplace `F(s) = ∫₀^∞ f(t) e^(-s t) dt`;
Fourier `F(w) = ∫ f(t) e^(-i w t) dt` over the reals, its inverse `(1/(2 pi)) ∫ F(w) e^(i w t) dw`;
Mellin `F(s) = ∫₀^∞ x^(s-1) f(x) dx`; the z-transform `F(z) = Σ_{n≥0} f(n) z^(-n)`.
The generator may add ops and arguments; each is added to this table in the same commit.
## 5. The upload file
A run is uploaded as one file of JSON lines, plain or gzipped, of at most 100 MB as uploaded: a
header, then one line for each problem the runner gave the system. The upload claims no grades,
and no visibility.
### 5.1 The header
```json
{
"format": "math-upload/1",
"suite": "2026.1",
"suite_sha256": "…",
"subset": "full",
"system": {"name": "Maxima", "version": "5.47.0"},
"runner": {"name": "maxima-runner", "version": "1.0.0"},
"platform": "Linux 6.8.0 x86_64",
"machine": {"cpu": "AMD Ryzen 9 7950X 16-Core Processor", "cores": 16, "memory_gb": 64},
"timeout": 30,
"started": "2026-11-04T10:00:00Z",
"finished": "2026-11-04T18:20:00Z"
}
```
| field | | |
|---|---|---|
| `format` | required | `math-upload/1` |
| `suite` | required | the suite's version, published |
| `suite_sha256` | optional | its hash; where it is given, it must be the version's |
| `subset` | required | `full` or `quick` |
| `system.name` | required | the system's name, at most 60 bytes; the same in every run of the system. Its slug, the name in lower case with each run of other characters than letters and digits made `-`, names the system on the site, and tells the checker how to read the answers |
| `system.version` | required | the version exactly as the runner reads it from the system, at most 100 bytes, as `5.47.0` or `14.1.0 for Linux x86 (64-bit) (July 16, 2024)`. The site orders a system's versions by it, the numbers in it compared as numbers, so that a development build is best read as `0.9.0-12-gabc1234`, as git describe gives |
| `runner.name`, `runner.version` | required | the runner, at most 60 bytes each |
| `platform` | required | the operating system, at most 200 bytes |
| `machine.cpu` | required | the CPU's model, at most 200 bytes |
| `machine.cores` | required | the cores, a number from 1 |
| `machine.memory_gb` | optional | the memory |
| `timeout` | required | the seconds each problem had |
| `started` | required | when the run started, ISO 8601 |
| `finished` | optional | when it ended |
Other fields are kept, and not used, except `visibility`, `private`, `public` and `hidden`, here or
in `system`: an upload that has one is refused. The uploads are public; only the admin makes a
system or a run private.
### 5.2 The answers
```json
{"id": "indefinite_integral-000123", "status": "ok", "answer": "%e^x^2", "wall": 0.042, "cpu": 0.040}
{"id": "ode-004711", "status": "timeout", "wall": 30.0, "cpu": 29.97}
{"id": "pde-000815", "status": "error", "wall": 0.31, "cpu": 0.30, "message": "Division by zero"}
{"id": "transform-000042", "status": "unsupported"}
```
| field | | |
|---|---|---|
| `id` | required | a problem of the subset, once in the file |
| `status` | required | `ok`: the system returned; `timeout`: it ran out of time; `error`: it failed, or the runner could not read its result; `unsupported`: the runner has no way to give the system this op |
| `answer` | with `ok` | what the system returned, as text in its own input syntax, at most 1 MB: the problem unevaluated or partly evaluated as well as a result. The checker reads it with the system's syntax, so that it is best printed as the system reads it back (Maxima's `string`, Mathematica's `InputForm`, SymPy's `srepr` or `str`, Maple's `lprint`) |
| `wall`, `cpu` | optional | seconds, of the system's work on this problem only; numbers, or null where unknown |
| `message` | optional | the error's text |
- The problems of the subset that the file does not have count as **missing**: not solved.
- The site grades `timeout` as timeout, `error` as error and `unsupported` as unsolved, itself; it
sends the `ok` answers to the checker.
- An answer is kept by the site, and shown, up to 20 KB, cut there at a character and marked cut;
the checker gets it whole. The upload is kept as it was uploaded (gzipped where it was not).
### 5.3 Calibration
The runner runs the calibration subset first, and writes its answers as any others. The site sums
each run's times on it, the cpu time or else the wall time, and gives the run its **calibration
factor**: that sum over the median of the sums of the system's runs (the public runs' median for a
public run). A run on a machine twice as slow as the system's usual has the factor 2. The pages
show the raw times, and the times divided by the factor where it is not 1, so that the runs of a
system on different machines compare. As the calibration problems are solved by the system itself,
the factor compares machines for one system, not systems; a run that leaves out a calibration
problem has no factor.
### 5.4 What a runner does
1. Fetch the suite, check its hash (`GET /api/suites` lists the versions, hashes and files).
2. Start the system; read its version from it.
3. Give it each problem of the calibration subset, then of the subset, each in a fresh state (a
new session, or the state reset), with the timeout, and time it.
4. Write the header and a line for each problem.
5. Upload the file (section 6).
### 5.5 The systems' answer syntax
An answer is written the way the system writes its input on one line, so that the checker reads it
back with that system's syntax. The runners of the systems the site knows use these; a runner of
another system writes its system's own one-line input form, and the checker learns it.
| system | `system.name` | the answer as | example |
|---|---|---|---|
| SymPy | `SymPy` | `str(result)`, which `sympify` reads back | `exp(x**2)` |
| Maxima | `Maxima` | `string(result)` with `display2d: false` | `%e^x^2` |
| FriCAS | `FriCAS` | `unparse(result::InputForm)` | `exp(x^2)` |
| Giac | `Giac` | `string(result)` | `exp(x^2)` |
| Maple | `Maple` | `lprint(result)` (`convert(result, string)`) | `exp(x^2)` |
| Mathematica | `Mathematica` | `ToString[result, InputForm]` | `E^x^2` |
A result that is not one expression (a list of solutions, a matrix, a set of equations) is written
the same way, in the system's own form of it, as `[[x = -2], [x = 2]]` in Maxima or
`{{x -> -2}, {x -> 2}}` in Mathematica. Where the runner gives the system the task by a command that
returns a structure (a solver's record, a series object), it writes the part that is the answer, and
says which in the runner's documentation.
## 6. The upload API
Registering at `/register`, with an email address, gives an upload token, sent with each request as
`Authorization: Bearer <token>`. A new token, from `/account`, replaces it. The web form at
`/upload` takes the same file.
### `POST /api/runs`
The body is the upload file itself, with the type `application/x-ndjson`, or `application/gzip`
where it is gzipped.
```
curl -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/gzip" \
--data-binary @run.jsonl.gz https://math.osp.no/api/runs
```
- `202`: `{"run": "r-k3j9x2m4q8", "status": "processing", "url": "https://math.osp.no/runs/r-k3j9x2m4q8"}`.
The file is checked in full before this answer; it is then stored, and its answers cut into
grading units, in the background (`processing`, then `grading`, then `graded`).
- `401`: no token, or not a valid one.
- `413`: the file is over 100 MB.
- `415`: the type is not one of the above.
- `422`: `{"errors": ["line 1: the timeout is not a number of seconds", "line 9: ode-1 is not in the subset"]}`,
at most 20 of them. Nothing is kept.
### `GET /api/runs/<run>`
The status of one of the uploader's runs: `{"run", "status", "error", "answers", "problems", "url"}`,
and `totals` once it is graded, where the run is public: `{"problems", "solved", "unsolved",
"wrong", "timeout", "error", "review", "missing"}`. A run of a private system shows its totals on
the admin's pages only. `404` for a run that is not the uploader's.
### `GET /api/suites`
The suite's versions, newest first: `{"suites": [{"version", "sha256", "problems", "published",
"quick": {"sha256", "problems"}, "files": {"suite", "quick", "manifest"}}]}`. No token is needed.
## 7. The grader API
A grader is one of our machines, running samara's checker in a loop. Each has a token, the Fly
secret `MATH_GRADER_TOKEN_<NAME>`, sent as `Authorization: Bearer <token>`; its name is NAME in lower
case. The answers are JSON, with an `error` where the request is wrong.
### 7.1 `POST /api/grader/units/next`
Deals the next unit, the oldest run's first: `200` with the unit, or `204` where there is no work.
```json
{
"unit": "g-k3j9x2m4q8-0",
"run": "r-k3j9x2m4q8",
"suite": "2026.1",
"system": {"name": "Maxima", "slug": "maxima", "version": "5.47.0"},
"lease_until": 1793400900,
"items": [
{
"problem": {
"id": "indefinite_integral-000123",
"category": "indefinite_integral",
"origin": "generated",
"method_class": "substitution",
"level": "textbook",
"problem": {"op": "integrate", "args": {"integrand": "2*x*exp(x^2)", "var": "x"}},
"answer": "exp(x^2)",
"famous": null
},
"status": "ok",
"answer": "%e^x^2",
"wall": 0.042,
"cpu": 0.040
}
]
}
```
A unit has at most 250 answers, and at most about 2 MB of them, all of status `ok`, each whole. For
a famous problem `famous` is `{"status", "claim"}`.
### 7.2 `POST /api/grader/units/<unit>/extend`
A dealt unit is the grader's for 15 minutes; the grader extends it every 5 minutes while it works:
`200` `{"lease_until": …}`, or `410` where the lease is lost, as it ran out and the unit was dealt
again. A unit not extended in time is dealt again.
### 7.3 `POST /api/grader/units/<unit>/result`
The grades of every answer of the unit, once:
```json
{
"checker": {"name": "samara-check", "version": "9e498f1"},
"grades": [
{
"problem": "indefinite_integral-000123",
"grade": "solved",
"latex": "e^{x^{2}}",
"summary": "8 points, residual at most 3e-31 at 40 digits",
"output": {"…": "section 8"}
}
]
}
```
- `grade`: one of `solved`, `unsolved`, `wrong`, `timeout`, `error` (section 7.4).
- `latex`: the answer in LaTeX, as the site shows it with KaTeX, or null; the raw answer is shown
where it is null, over 20 KB, or KaTeX cannot read it.
- `summary`: a line on the problem's page, at most 300 characters, optional.
- `output`: the checker's output for the answer, at most 16 KB as JSON (section 8): a few test
points, not all of them.
The body is JSON, of at most 8 MB.
The answers: `200` `{"graded": true}`; `404` for a unit not known (or graded and gone); `410`
where it was graded already, by another grader; `422` `{"error": …}` where a grade is missing,
twice, of a problem not in the unit, or not one of the five; `503` where the site could not store
the output, to be posted again later. The output is stored before the grades are taken, so that a
grade is never without its output.
### 7.4 The grades
- **solved**: the answer is right, and is a result: what the problem asks for, with no unevaluated
integral, sum, limit or solve left; of an antiderivative up to a constant; of a simplification no
larger than the known answer, by the checker's measure (the count of the nodes of the answer as
written); of a series the terms asked for.
- **unsolved**: the system returned the problem, or a part of it, unevaluated, or declined; or gave a
decimal approximation where the problem asks for an exact result.
- **wrong**: the answer is a result, and not right; or, for an impossible famous problem, a closed
form of the impossible kind.
- **timeout**, **error**: from the runner's status, given by the site; the checker gives `error`
where it cannot read an answer, and where it cannot decide whether a result is right, with a
message that starts `the checker cannot decide`. Such an answer is never graded solved.
The site keeps what the checker says, except that a `solved` answer to an open famous problem is
kept as `review` until the admin grades it by hand.
### 7.5 `GET /api/grader/status`
`{"units": {"pending": n, "leased": n, "done": n}, "grading": runs, "processing": runs}`.
### 7.6 A grader's loop
```
loop:
POST /api/grader/units/next
204: sleep a minute, again
200: grade the items, extending the lease every 5 minutes;
POST the result; on 503, post it again later; on 410, drop it
```
## 8. The checker's output
The site keeps the output of each unit as it was posted, as `runs/<run>/checker/<n>.jsonl.gz` in its
bucket, one line for each answer:
```json
{
"run": "r-k3j9x2m4q8",
"problem": "indefinite_integral-000123",
"grade": "solved",
"latex": "e^{x^{2}}",
"summary": "8 points, residual at most 3e-31 at 40 digits",
"checker": {"name": "samara-check", "version": "9e498f1"},
"grader": "bowser",
"output": {
"method": "numeric",
"precision": 40,
"tolerance": "1e-30",
"points": [
{"at": {"x": "0.7071067811865475244"}, "expected": "1.0", "actual": "1.0", "residual": "3.1e-31"}
],
"checks": [{"name": "derivative", "passed": true, "detail": "d/dx of the answer minus the integrand"}],
"message": null,
"seconds": 0.12
}
}
```
`output` is the checker's, and its fields are its own; these are the ones it uses, so that a
reader knows them:
| field of `output` | |
|---|---|
| `method` | how it checked: `numeric`, `symbolic`, `structural`, … |
| `precision` | the digits it computed with |
| `tolerance` | the residual it took as zero |
| `points` | the test points: `at`, the values of the variables (and parameters); `expected` and `actual`, the values compared; `residual`, their difference. Numbers are text, so that their digits are kept |
| `checks` | the checks made: `name`, `passed`, `detail` |
| `message` | a note, as why an answer is unsolved |
| `seconds` | the checker's time |
Anyone can download it once the run is graded, to verify a grade (the checker's source is not
public):
- `GET /runs/<run>/checker`: every answer's, the units' files one after another, which is a valid
gzip file of JSON lines;
- `GET /runs/<run>/checker/<problem>`: one answer's line, as JSON.
## 9. What the site shows, and keeps
- **Visibility**: a system is public or private, and a run may override its system's; only the
admin sets either. A new system is public, except those the site's config names, samara, which
start private. The uploads are public: an upload, or a user, has
no way to make a run private. The public pages show and count the public runs only: a private
system's answers and grades, and its part in the counts (how many systems solve a problem, which
one alone, the conflicts, the problems no system solves), appear nowhere but in the admin's full
view. A hidden user's runs are private.
- **Counts**: each system counts once for a problem, by its newest answer: of its newest version,
and that version's newest run. Two systems' answers **conflict** where one is solved (or for
review) and the other wrong.
- **Retention**: the site is not an archive. Each system keeps in full the newest run of each
subset of its newest three versions, of all its runs and of its public runs. An older run keeps
its totals by category and level, and the problems that changed against the version before,
counted before its answers go; its upload and checker output are deleted.
## 10. The bucket
s3://osp-math at Scaleway, private; the site reads and writes it, and rebuilds its database from it
(`Osp.Math.Rebuild`).
```
suites/<version>/manifest.json the suite's manifest
suites/<version>/suite.jsonl.gz the suite, every problem
suites/<version>/quick.jsonl.gz the quick subset
runners/<name>/<version>/<file> a runner, and runner.json beside it
users/<user>.json a user, with the email
systems/<slug>.json a system, with its visibility
runs/<run>/run.json a run's header and state
runs/<run>/upload.jsonl.gz the upload, as uploaded (gzipped)
runs/<run>/units/<n>.jsonl.gz a grading unit's items, until graded
runs/<run>/checker/<n>.jsonl.gz the checker's output of the unit
runs/<run>/reviews.json the admin's grades of answers for review
runs/<run>/summary.json a cut-down run's totals and changes
```