Configuration

The ExuluEval constructor takes a single options object. This page documents every option, verified against the current @exulu/backend source.

import { ExuluEval } from "@exulu/backend";

const eval = new ExuluEval({
  id,           // string — required
  name,         // string — required
  description,  // string — required
  llm,          // boolean — required
  execute,      // function — required
  config,       // config param array
  queue,        // Promise<ExuluQueueConfig>
});

Identity

string

required

Unique identifier for this eval function, used for registration and deduplication.

name

string

required

Human-readable name shown in the evals workbench.

description

string

required

What this eval measures, shown to administrators when selecting evals for an agent.

LLM flag

llm

boolean

required

Whether this eval calls a language model internally (LLM-as-judge). The platform UI uses this flag to warn administrators about the cost and latency of eval runs that use an LLM. Set true when execute calls provider.generateSync() or any other model-calling API; set false for deterministic, code-only evals.

Execute function

execute

function

required

The scoring function. Receives a single params object and must return a Promise<number> where the number is in [0, 100]. Scores outside that range cause run() to throw.

execute: async (params: {
  agent: ExuluAgent;
  provider: ExuluProvider;
  messages: UIMessage[];
  testCase: TestCase;
  config?: Record<string, any>;
}) => Promise<number>

execute.params.agent

ExuluAgent

required

The database agent record being evaluated — the row resolved by app.agent(id). Contains the agent’s id, name, description, instructions, model, and tool configuration.

execute.params.provider

ExuluProvider

required

The ExuluProvider instance that ran the generation. Useful when the eval itself needs to call provider.generateSync() for LLM-as-judge scoring.

execute.params.messages

UIMessage[]

required

The full conversation as Vercel AI-SDK UIMessage objects, including user messages, assistant turns, and tool invocations. The last message is typically the assistant’s final response.

execute.params.testCase

TestCase

required

The structured test case with inputs and expected outputs.

execute.params.config

Record<string, any>

Runtime configuration values passed to run() by the caller (or sourced from admin-set config params in the platform UI).

The TestCase type carries these fields:

type TestCase = {
  id: string;
  name: string;
  description?: string;
  inputs: UIMessage[];                       // Conversation inputs for the agent
  expected_output: string;                   // Expected final response
  expected_tools?: string[];                 // Tool names the agent should call
  expected_knowledge_sources?: string[];     // Context IDs the agent should search
  expected_agent_tools?: string[];           // Agent-tool names the agent should call
  createdAt: string;
  updatedAt: string;
};

Common scoring patterns

Exact match:

execute: async ({ messages, testCase }) => {
  const response = (messages[messages.length - 1]?.content ?? "").trim();
  return response === testCase.expected_output.trim() ? 100 : 0;
}

Keyword presence:

execute: async ({ messages, testCase, config }) => {
  const response = (messages[messages.length - 1]?.content ?? "").toLowerCase();
  const keywords: string[] = config?.keywords ?? [];
  if (keywords.length === 0) return 100;
  const found = keywords.filter((kw) => response.includes(kw.toLowerCase()));
  return (found.length / keywords.length) * 100;
}

Tool usage:

execute: async ({ messages, testCase }) => {
  const toolCalls = messages
    .flatMap((m) => (m as any).toolInvocations ?? [])
    .map((inv: any) => inv.toolName);
  const expected = testCase.expected_tools ?? [];
  if (expected.length === 0) return 100;
  const matched = expected.filter((t) => toolCalls.includes(t));
  return (matched.length / expected.length) * 100;
}

LLM-as-judge:

execute: async ({ provider, messages, testCase, config }) => {
  const response = messages[messages.length - 1]?.content ?? "";
  const judgePrompt = `
Rate the following response on accuracy, completeness, and clarity (0-100).

Expected: ${testCase.expected_output}
Actual: ${response}

Respond with ONLY a number from 0 to 100.
  `.trim();

  const text = await provider.generateSync({
    prompt: judgePrompt,
    languageModel: judgeLanguageModel,
  });

  const score = parseInt(String(text).trim(), 10);
  if (isNaN(score)) return 0;
  return Math.max(0, Math.min(100, score));
}

execute must return a number strictly in [0, 100]. If it returns a value outside that range, run() throws "Eval function <name> must return a score between 0 and 100, got <value>". Clamp your score before returning when you cannot guarantee the range.

Config params

config

object[]

Optional array of runtime parameters. Administrators set values for these params in the platform UI when configuring an eval run. Values are passed to execute in the config argument as a Record<string, any>.

type ExuluEvalConfigParam = {
  name: string;
  description: string;
};

config[].name

string

required

Parameter name, used as the key in the config record passed to execute.

config[].description

string

required

Description shown to the administrator in the platform UI.

config: [
  { name: "keywords", description: "Comma-separated keywords that must appear in the response." },
  { name: "judge_agent_id", description: "Agent ID to use as the LLM judge." },
  { name: "min_length", description: "Minimum expected character count for the response." },
]

Unlike ExuluTool config params, eval config params have no type or default field in the current source — they are declared as { name: string; description: string } only. The value type and default must be handled inside execute itself (for example, using optional chaining and a fallback).

Queue

queue

Promise<ExuluQueueConfig>

required

Queue configuration for running evals as background jobs. In the current source, queue is a required field in the ExuluEvalParams interface — though if you pass undefined at runtime (in JavaScript), the constructor still assigns it and the class treats it as optional (public queue?: Promise<ExuluQueueConfig>). For robustness, always pass a queue config when registering production evals.

import { ExuluQueues } from "@exulu/backend";

const evalQueue = ExuluQueues.register("eval_runs", { worker: 2, queue: 10 }).use();

const myEval = new ExuluEval({
  id: "quality_check",
  name: "Quality check",
  description: "LLM-as-judge quality scoring.",
  llm: true,
  execute: async ({ provider, messages, testCase }) => {
    // ... scoring logic
    return score;
  },
  queue: evalQueue,
});

ExuluQueues registration requires an Enterprise Edition license. Without a valid EXULU_ENTERPRISE_LICENSE, app.create() throws when it detects registered queues.

Complete example

import { ExuluEval, ExuluQueues } from "@exulu/backend";

const evalQueue = ExuluQueues.register("eval_runs", { worker: 2, queue: 10 }).use();

export const responseQualityEval = new ExuluEval({
  id: "response_quality",
  name: "Response quality",
  description: "Scores the agent's response for accuracy and completeness using an LLM judge.",
  llm: true,

  execute: async ({ provider, messages, testCase, config }) => {
    const response = messages[messages.length - 1]?.content ?? "";

    const judgePrompt = `
You are an expert evaluator. Rate the agent response below on a scale of 0-100.

Test case: ${testCase.name}
Expected output: ${testCase.expected_output}
Actual response: ${response}

Criteria:
- Accuracy (40 pts): Does the response contain the correct information?
- Completeness (30 pts): Does it address all aspects of the expected output?
- Clarity (30 pts): Is it well-structured and easy to understand?

Respond with ONLY a number from 0 to 100.
    `.trim();

    const text = await provider.generateSync({
      prompt: judgePrompt,
      languageModel: resolvedJudgeModel,
    });

    const score = parseInt(String(text).trim(), 10);
    if (isNaN(score)) return 0;
    return Math.max(0, Math.min(100, score));
  },

  config: [
    { name: "judge_agent_id", description: "Agent ID to use as the LLM judge." },
  ],

  queue: evalQueue,
});

Identity

LLM flag

Execute function

Common scoring patterns

Config params

Queue

Complete example

Next steps

API reference

Evals (UI side)

​Identity

​LLM flag

​Execute function

​Common scoring patterns

​Config params

​Queue

​Complete example

​Next steps

API reference

Evals (UI side)

Identity

LLM flag

Execute function

Common scoring patterns

Config params

Queue

Complete example

Next steps