Scoring Gates

A scoring gate is a quality checkpoint that validates upstream output against a list of deterministic rules before spending a single token on an LLM. You declare what "good" looks like — a substring present, a regex matched, a JSON schema satisfied, a length in range, or code that compiles — and taskflow evaluates every check at zero cost. When all checks pass, the gate auto-passes. When they fail, an optional LLM judge can decide whether to let the flow continue or halt it.

You might be wondering: why not just use a regular gate with an LLM? Because deterministic checks are faster, cheaper, and more predictable. A regex either matches or it doesn't — there's no hallucination risk. Scorers run before the judge, so if your checks are sufficient, you never pay for a token at all.

When to reach for scoring gates

A running example

The rest of this page builds a single, realistic flow: validate a generated API response before shipping it. You will see the gate grow from a single check into a multi-scorer gate with a judge fallback as each new need introduces a new scorer type.

Save this as validate.json — we will keep extending it:

{
  "name": "validate",
  "phases": [
    {
      "id": "generate",
      "type": "agent",
      "agent": "executor",
      "task": "Generate a JSON API response for GET /users. Include status, data, and message fields.",
      "output": "json"
    },
    {
      "id": "check-response",
      "type": "gate",
      "dependsOn": ["generate"],
      "score": {
        "scorers": [
          { "type": "contains", "value": "\"status\"" }
        ]
      },
      "task": "Is this a valid API response? VERDICT: PASS or VERDICT: BLOCK."
    }
  ]
}

The six scorer types

Each scorer is a deterministic check that returns passed: true or passed: false with an optional detail message. Scorers are pure (except code-compiles) and never throw — a bad configuration fails the scorer with a diagnostic rather than crashing the run.

exact-match

Tests whether the target string is byte-for-byte equal to the value.

{
  "type": "exact-match",
  "value": "PASS"
}

Fields:

Pass semantics: target === value. Case-sensitive, no trimming.

When to use: Exact sentinel values, fixed status codes, known-good outputs.

contains

Tests whether the target string includes the value as a substring.

{
  "type": "contains",
  "value": "0 errors"
}

Fields:

Pass semantics: target.includes(value). Case-sensitive.

When to use: Checking for required phrases, error counts, status markers.

regex

Tests whether the target matches a JavaScript regular expression.

{
  "type": "regex",
  "pattern": "\"status\"\\s*:\\s*\"success\"",
  "negate": false
}

Fields:

Pass semantics: new RegExp(pattern).test(target), inverted when negate: true.

Error handling: An invalid pattern fails the scorer with a detail message (e.g., "invalid pattern: Invalid regular expression"). The gate then falls through to the judge or task — it does not crash.

When to use: Pattern matching, format validation, negative checks (e.g., "does not contain ERROR").

json-schema

Validates the parsed target against a TypeBox-style contract (the same schema language used by expect).

{
  "type": "json-schema",
  "schema": {
    "type": "object",
    "required": ["status", "data"],
    "properties": {
      "status": { "enum": ["success", "error"] },
      "data": { "type": "object" }
    }
  }
}

Fields:

Pass semantics: The target is parsed with the same lenient JSON parser used by output: "json" (fenced code blocks are extracted). If parsing fails, the scorer fails. Otherwise, contractViolations checks the parsed value against the schema.

When to use: Validating structured output, enforcing response shapes, checking enum values.

length-range

Tests whether the target's character count falls within an inclusive range.

{
  "type": "length-range",
  "min": 100,
  "max": 10000
}

Fields:

Pass semantics: len >= min && len <= max. If only min is provided, there's no upper bound. If only max is provided, the lower bound is 0.

Validation: min must be <= max when both are present. Both must be >= 0.

When to use: Sanity checks on output size, preventing empty responses, capping verbosity.

code-compiles

Spawns a real compiler in an isolated temp directory and checks whether the target parses or compiles.

{
  "type": "code-compiles",
  "language": "typescript"
}

Fields:

Pass semantics:

• javascript: Runs node --check <file>. Syntax check only — no execution, no dependencies.
• typescript: Runs npx --no-install tsc --noEmit --skipLibCheck <file>. Requires a resolvable tsc in the environment. Its absence fails the scorer with a detail.

Code extraction: If the target contains exactly one fenced code block, its body is compiled. Multiple fences or no fences means the raw target is compiled.

Security model: The compiler runs in an isolated temp directory (mkdtempSync with 0700 permissions), not the phase's working directory. This prevents it from resolving a repo-planted node_modules/.bin/tsc or accessing project files. The --no-install flag prevents npx from downloading packages.

Timeout: 30 seconds per compiler invocation. A timeout fails the scorer with a detail.

Error handling: An unavailable compiler, a spawn error, or a timeout is a failed scorer (not a crash). The gate's fail-open path then decides what a failed check means.

When to use: Validating generated code, catching syntax errors before shipping, type-checking TypeScript output.

Combining scorers

A score block declares a list of scorers and a combine mode that decides how their results are aggregated. The default is all, which requires every scorer to pass.

all (default)

Every scorer must pass. The combined score is 1 if all pass, 0 if any fail.

{
  "scorers": [
    { "type": "contains", "value": "\"status\"" },
    { "type": "regex", "pattern": "\"status\"\\s*:\\s*\"success\"" }
  ],
  "combine": "all"
}

Semantics: results.every(r => r.passed). Binary outcome.

any

At least one scorer must pass. The combined score is 1 if any pass, 0 if all fail.

{
  "scorers": [
    { "type": "contains", "value": "PASS" },
    { "type": "contains", "value": "OK" }
  ],
  "combine": "any"
}

Semantics: results.some(r => r.passed). Binary outcome.

weighted

Each scorer contributes a weighted score to a total, which is compared against a threshold.

{
  "scorers": [
    { "type": "contains", "value": "\"status\"" },
    { "type": "json-schema", "schema": { "type": "object" } }
  ],
  "combine": "weighted",
  "weights": [0.3, 0.7],
  "threshold": 0.5
}

Fields:

Semantics: Each scorer's score is 0 (fail) or 1 (pass). The combined score is sum(score[i] * weight[i]) / sum(weight[i]). When a judge is present, its weight is appended to the weights array and its score folds into the combination.

Early cutoff: When a judge is configured but hasn't run yet, the deterministic-only combination is a lower bound. If it already clears the threshold, the judge's score could not change the outcome and the gate auto-passes without spending judge tokens.

The judge fallback

When deterministic scorers fail, an optional LLM judge can decide whether to let the flow continue. The judge runs only when the scorers don't pass — if they do, the gate auto-passes with zero tokens spent on the judge.

{
  "id": "quality",
  "type": "gate",
  "dependsOn": ["generate"],
  "score": {
    "scorers": [
      { "type": "json-schema", "schema": { "type": "object", "required": ["status"] } }
    ],
    "judge": {
      "agent": "reviewer",
      "task": "Is this a reasonable API response even if it doesn't match the schema exactly?"
    }
  }
}

Judge fields:

Judge output parsing: The judge can respond in three formats:

1. JSON: {"score": 0.0-1.0, "verdict": "pass"|"block", "reason": "..."}
2. Bare verdict: {"verdict": "pass"} or {"verdict": "block"}
3. Text markers: SCORE: 0.8 and/or VERDICT: PASS (case-insensitive, last match wins)

Fail-open semantics: An unparseable judge output passes with score: 1 and verdict: "pass". Ambiguity must not block the flow.

Combining with scorers:

• weighted mode: The judge's score folds into the weighted combination using the last weight in weights. The threshold decides the verdict.
• all or any mode: The judge's verdict is authoritative. If the judge says pass, the gate passes regardless of scorer results.

The target field

By default, scorers evaluate the upstream phase's output — the text produced by the phase listed in dependsOn. You can override this with the target field, which accepts an interpolation expression.

{
  "id": "check-json",
  "type": "gate",
  "dependsOn": ["generate"],
  "score": {
    "target": "{steps.generate.json.response}",
    "scorers": [
      { "type": "contains", "value": "\"status\"" }
    ]
  }
}

Default: "{previous.output}" — the output of the immediately upstream phase.

Interpolation: The target is interpolated before scorers run, so you can reach into JSON fields with {steps.<id>.json.<field>} or use any other placeholder.

Complete gate with scoring

Here's the running example after adding multiple scorers, a weighted combination, and a judge fallback:

{
  "name": "validate",
  "phases": [
    {
      "id": "generate",
      "type": "agent",
      "agent": "executor",
      "task": "Generate a JSON API response for GET /users. Include status, data, and message fields.",
      "output": "json"
    },
    {
      "id": "check-response",
      "type": "gate",
      "dependsOn": ["generate"],
      "score": {
        "target": "{steps.generate.output}",
        "scorers": [
          { "type": "contains", "value": "\"status\"", "name": "has-status-field" },
          { "type": "regex", "pattern": "\"status\"\\s*:\\s*\"success\"", "name": "status-is-success" },
          { "type": "json-schema", "schema": { "type": "object", "required": ["status", "data"] }, "name": "valid-schema" },
          { "type": "length-range", "min": 50, "max": 5000, "name": "reasonable-length" }
        ],
        "combine": "weighted",
        "weights": [0.2, 0.3, 0.4, 0.1],
        "threshold": 0.7,
        "judge": {
          "agent": "reviewer",
          "task": "Is this a reasonable API response even if some checks failed?"
        }
      }
    }
  ]
}

The gate evaluates four scorers at zero tokens. If the weighted combination clears 0.7, it auto-passes. Otherwise, the judge runs and decides.

Scorer naming

Each scorer can carry an optional name field. If omitted, the runtime generates a default label: <type>-<index> (e.g., contains-0, regex-1).

{
  "scorers": [
    { "type": "contains", "value": "PASS", "name": "contains-pass-marker" },
    { "type": "regex", "pattern": "\\d+", "name": "has-numeric-id" }
  ]
}

Names appear in the scorer report that's appended to the judge's prompt, making it easier for the LLM to understand which checks failed.

Fail-open semantics

Scoring gates follow the same fail-open principle as the rest of taskflow: ambiguity must not block the flow.

Accessing scorer results downstream

The gate's structured result is exposed as PhaseState.json, reachable via interpolation:

{
  "id": "report",
  "type": "agent",
  "dependsOn": ["check-response"],
  "task": "The gate verdict was {steps.check-response.json.verdict} with a combined score of {steps.check-response.json.combined}. Summarize."
}

Interaction with eval and task

A gate can have eval, score, and task fields. They run in this order:

1. eval (if present): Zero-token machine checks. If all pass, the gate auto-passes. If any fail, continue to step 2.
2. score (if present): Deterministic scorers. If they pass (per combine and threshold), the gate auto-passes. If they fail and a judge is configured, the judge runs. If they fail and no judge is configured, continue to step 3.
3. task (if present): The LLM gate runs as normal, parsing a VERDICT: marker.

Security restrictions

The code-compiles scorer is the only scorer that spawns a child process. It runs under strict isolation:

• Isolated temp directory: Created with mkdtempSync (atomic, 0700 permissions). Closed the predictable-name symlink/TOCTOU race on shared /tmp.
• No project access: The compiler runs with the temp dir as its working directory, not the phase's cwd. It cannot resolve a repo-planted node_modules/.bin/tsc.
• No network: npx --no-install prevents downloading packages. The compiler can only check syntax and types from the standard library.
• Timeout: 30-second wall-clock cap per invocation. A timeout kills the process with SIGKILL.
• Cleanup: The temp directory is removed with rmSync({recursive: true, force: true}) in a finally block. Best-effort cleanup — a failure to delete is swallowed.

All other scorers are pure and run in-process with no filesystem or network access.

Field reference

score object

Scorer object

Every scorer has these common fields:

Per-type fields:

Validation: A field set on a scorer whose type ignores it is a shape error. For example, {type: "contains", negate: true} is rejected — silently dropping negate would let the author believe negation happened.

Gate verdict grammar

A gate phase (with or without score) parses its output into a verdict using this grammar:

JSON format:

{"continue": true}  → pass
{"continue": false} → block
{"pass": true}      → pass
{"pass": false}     → block
{"verdict": "PASS"} → pass
{"verdict": "BLOCK"} → block
{"verdict": "pass"} → pass (case-insensitive)
{"verdict": "block"} → block
{"verdict": "anything-else"} → pass (fail-open)

Text format:

VERDICT: PASS    → pass
VERDICT: BLOCK   → block
VERDICT: FAIL    → block
VERDICT: STOP    → block
VERDICT: OK      → pass
VERDICT: REJECT  → block
VERDICT: HALT    → block

• Case-insensitive.
• Last occurrence wins (if the output contains multiple VERDICT: lines).
• Supports both : and = separators: VERDICT=PASS works.

Fail-open: Ambiguous output (no JSON, no VERDICT: marker) passes. A gate never accidentally loses your work.

Tournament winner grammar

A tournament judge picks the winning variant using this grammar:

JSON format:

{"winner": 2}     → variant 2
{"best": 3}       → variant 3
{"choice": 1}     → variant 1

Text format:

WINNER: 2         → variant 2
WINNER: #3        → variant 3 (hash prefix optional)
WINNER=1          → variant 1 (supports = separator)

• 1-based index.
• Last occurrence wins.
• Clamped to [1, variant-count]: a verdict of 0 becomes 1, a verdict of 99 becomes the last variant.

Fail-open: An unparseable verdict defaults to variant 1 with reason "no parseable winner; defaulted to variant 1". The tournament never loses your work.

Common errors

Next

Last updated on