Appendix II: jobAsync Generator/Comparator Bundle Contracts

This appendix is the authoring reference for a jobAsync challenge variant’s generator and comparator bundles: the exact files each bundle must produce, how bundle content is kept private from the student, and the orchestration format used for more comprehensive, explicit-invocation authoring. For the overall jobAsync mechanism and current availability, see AI, Jobs, and Integrations.

Enclave: Private, Persistent Job-Group Storage

The enclave is a private, encrypted storage area scoped to one student’s job group — every job created for the same student-and-assignment pair shares one job group, not one per individual attempt. Content written there persists for every later job in that same group and is invisible to every other job group (a different student, or a different assignment/challenge).

Why it exists. The enclave is the actual privacy boundary for a jobAsync challenge’s generator and comparator bundles. The instructor-authored bundle content is staged into the enclave once — on the first job created for a job group — and every later job in the same group reuses it directly, with no re-staging. This is a different, complementary property from the job-execution-content upload store’s own privacy guarantee (see “Getting a Bundle Into a Variant” below): the upload store means bundle bytes are never served back over HTTP to any client at all; the enclave separately means that even within a running job, the student’s own submitted-code phase can never read the instructor’s bundle content.

Persistence characteristics. Nothing about the enclave is configured via a variant-create request field — there is no stickiness or build-cache toggle in evaluationConfigJSON. Fresh generator input/expected output is still the default on every run (see “Generator Bundle Contract” below); the enclave is what a generator uses instead of relying on that default, if it wants persistent or repeatable behavior.

Scope. Persisted content is scoped to the job group only — never shared across different students, assignments, or challenges.

Maturity. The enclave primitive itself is already deployed, but the phase-definition wiring that would exercise it for jobAsync — for both the generator and comparator bundles — is not confirmed landed yet. Treat this alongside the same “not yet fully available” status as attempt execution itself.

flowchart TD
    A[Job group's first job] --> B{Bundles already staged?}
    B -->|No| C[Stage the variant's generator/\ncomparator bundles into the enclave]
    C --> D[Phase execution reads\nfrom the now-populated enclave]
    B -->|Yes — every later job\nin the same group| D

How this jobAsync case uses the enclave:

  1. Bundle staging (privacy). Described above — the mechanism that actually hides instructor content from the student’s own code. Applies to both the generator and comparator bundles equally.
  2. Build-cache. A compiled generator’s or comparator’s build script (see each contract’s Build convention below — documented symmetrically for both) writes generator.bin/comparator.bin into this same persistent area, so it is compiled once per job group and reused, not rebuilt on every attempt.
  3. Stickiness (generator only). A generator that wants genuinely repeatable — not fresh-random — output across every attempt in the same job group can use this same mechanism for its own state. This is documented for the generator specifically; the comparator contract has no equivalent stickiness use.

Bundle File Contracts

A jobAsync variant references two bundles — a generator and a comparator — each addressed by a digest (see “Getting a Bundle Into a Variant”). The two bundles communicate through a small set of fixed file names:

flowchart LR
    A[Generator bundle] -->|./input| B[Student's run phase]
    A -->|./expected| C[Comparator bundle]
    B -->|student's produced output| C
    C -->|./result.json| D[ChallengeResult]

Generator Bundle Contract

Comparator Bundle Contract

Orchestration Format (generatorRunCommand/comparatorRunCommand)

ChallengeVariantEvaluationConfigJSON carries two configType shapes for a jobAsync-strategy variant: the minimal jobAsync configType (testCaseCount only — bundles are referenced solely by digest, with no author-specified invocation), and the fuller jobSpec configType, which additionally requires generatorRunCommand/comparatorRunCommand. This section documents that second, fuller shape — needed for more comprehensive, explicit-invocation authoring.

Each of generatorRunCommand/comparatorRunCommand is an object with:

Only the invocation shape belongs on this object — a retry count, timeout, poll cadence, queue name, or any other runtime/orchestration-platform concern is never a valid field here.

Known gap. What the platform invokes when the simpler jobAsync configType is used (no explicit run command at all) is not yet part of the published contract. This gap is not closed by the runtime-selected pipeline below (#4648): runtime is valid only for the jobSpec configType, not jobAsync — see the next section for exactly what it closes instead.

Runtime-Selected Pipeline (runtime/comparator/submissionFileName)

ChallengeVariantEvaluationConfigJSON also carries an optional runtime field (issue #4648), valid only for the jobSpec configType. It closes a different, narrower gap than the one named above: for a jobSpec variant, an author no longer needs to write generatorRunCommand/comparatorRunCommand invocation steps or author a custom comparator bundle — runtime selects a fully system-provided pipeline for both halves at once. What still remains open is the jobAsync configType’s own question (what the platform invokes with no explicit run command and no runtime field at all); that is unaffected by this section.

runtime is an object with:

Together, language/toolchain must match one of a fixed, server-side allow-list of runtime profiles. Two are recognized today: cpp/gcc (submission extension cpp) and python/python3 (submission extension py, gated on run/python/stdin@1 actually shipping). An unrecognized pair is rejected at variant-authoring time.

When runtime is present:

Assembled pipeline. The platform assembles three phases in order: generate/enclave-runner@1 (unzips and stages the instructor’s enclave-provided content) -> the runtime’s own run phase (e.g. run/cpp/lint-compile-run-stdin@1, which lints, compiles, and runs the student’s submission against stdin) -> compare/standard@1 (the built-in comparator, configured by comparator.json). This differs from the generatorRunCommand/comparatorRunCommand shape above in one structural way: it also transfers a THIRD security policy (.merlin.securityPolicy.standard-compile-and-run, selected by the compile/run phase) alongside the schema and standard-untrusted policy every jobAsync document already transfers.

Staging. The student’s submission is staged as submission.<ext> (extension from the resolved runtime, e.g. submission.cpp), and a sibling comparator.json is staged alongside it – a canonical JSON document with all four comparator keys always present (mode, ignoreLineWhitespace, significantFigures, caseInsensitive), significantFigures an explicit JSON null when the mode is not numeric.

An absent runtime leaves the generatorRunCommand/comparatorRunCommand shape above entirely unchanged; a variant may use either shape (or neither, plus phaseDefinitionSet), never both.

System Phase-Definition Sets (phaseDefinitionSet)

ChallengeVariantEvaluationConfigJSON also carries an optional phaseDefinitionSet field (issue #4507), valid for either the jobAsync or jobSpec configType. This is a separate mechanism from generatorRunCommand/ comparatorRunCommand above: phaseDefinitionSet selects among a small, fixed set of system-authored generate/run/compare phase-definitions, not per-instructor bundle execution — the per-instructor bundle/enclave-runner pipeline this appendix otherwise documents is a distinct, larger capability that this field does not touch. It is also mutually exclusive with runtime (issue #4648, previous section) — a jobSpec variant selects at most one of the two.

Three named values are recognized:

Each fault-injected set swaps exactly one phase, not both: a single “both generate and compare fault-inject” set would only ever surface generatorFailure, since the pipeline aborts at generate before compare ever runs — making comparatorFailure permanently unreachable through it.

An absent or unrecognized phaseDefinitionSet value falls back to python-echo-pair at execution time (the backend also rejects an unrecognized value at variant-authoring time — see validateJobAsyncPhaseDefinitionSet in internal/domains/lessons/delivery.go).

Getting a Bundle Into a Variant

  1. Upload the bundle bytes to the job-execution-content store — uploadJobExecutionContentBundleV2 for bundles up to 4 MiB, or initializeExecutionContentUploadV2 for larger bundles — to receive a digest.
  2. Pass the returned digest into createChallengeVariantForChallengeV2 as generatorBundleDigest/comparatorBundleDigest.

generatorBundleDigest is always required this way. comparatorBundleDigest may be skipped entirely when the variant instead selects the standard (built-in) comparator via evaluationConfigJSON.comparator (#4649) — there is no bundle to upload or reference in that case, since the comparison logic is one of the platform’s built-in modes rather than authored source.

Bundle content has no read-back HTTP route anywhere — only the addressing digest is ever visible on a variant.

See AI, Jobs, and Integrations for the overall jobAsync mechanism and current availability.