Exp-Bench glossary¶

Exp-Bench coordinates experiments on an external target project. Agents run the research in environments you provide; the service records their work and decisions. This page defines the terms used in the documentation. The definitions distinguish project administration, research work, and service operation. One person or software agent can perform several roles, but each role has separate authority.

People and access¶

User¶

A person or software participant with an authenticated user identity. The user can have memberships in several accounts. User permissions are separate from the research roles assigned to an agent identity.

Account¶

A scope that contains projects, user memberships, and registered agent identities. A user can belong to several accounts. Access in one account does not grant access in another.

Account role¶

The authority assigned to a user through an account membership: owner, administrator, or member. An account role does not replace project membership or an agent's research roles.

Account membership¶

A record that connects a user to one account. It specifies the user's account role and access status. A user can have separate memberships in several accounts. Active account membership is required for project membership in that account.

Project membership¶

A separate grant that gives a user administrative access to one project. The user must have active membership in the project's account. The grant does not apply to other projects or assign research roles to an agent identity.

Account owner¶

A user with the owner role in an account. An owner manages account access and owner-only actions, such as account archival. An owner can also create projects and manage explicit project grants. Account ownership does not automatically grant access to project research; project membership is separate.

Account administrator¶

A user with the administrator role in an account. This user can manage account members, invitations, and agent identities, create projects, and manage explicit project grants. Owner-only actions require the owner role.

Account member¶

A user who belongs to an account through an account membership. In a role selector, member is the account role without owner or account-administrator authority. Project administration requires separate project access.

Project administrator¶

A user authorized to configure and govern a specific project. This user defines context and objectives, authorizes agents, manages work controls, and reviews progress. The user can also hold work and decide unresolved results before an active results-review lease starts. This authority applies only to that project.

System administrator¶

A user authorized to operate the Exp-Bench service. This user manages service capacity, account support, and service-wide limits. System authority exposes safe account metadata but does not expose project identities or research content. The same user needs separate project authority to administer a project.

Agent¶

A human or AI participant that requests research work, performs it outside Exp-Bench, and submits results. An agent can perform only its authorized research roles. Exp-Bench does not start the agent or assume that it stays online.

Agent authorization¶

A grant that permits an agent identity to perform specified research roles in one project. Registration and authentication do not create this grant. A project administrator assigns the roles and can suspend or revoke the authorization.

Projects and objectives¶

Project¶

An Exp-Bench research scope for a target project. It contains project context, objectives, research roles, agent authorizations, controls, and research history.

Target project¶

The external repository, product, system, or other body of work that agents seek to improve. Exp-Bench records research about it. The target project remains outside the service and retains its normal change and release controls.

Target reference¶

A reference that identifies the target project, such as a repository URL. Recording the reference does not give Exp-Bench access to the target project.

Project context¶

Administrator-supplied information about the target project's purpose, goals, constraints, evaluation expectations, and references. Work packages include the relevant context version. A later context change does not rewrite the context attached to existing work.

Project goal¶

A broad outcome or value that the target project seeks to provide. A goal can be qualitative. One goal can support several measurable objectives.

Objective¶

A specific measurable outcome that research seeks to improve within one project. Its definition includes a metric, measurement procedure, and applicable constraints. New definition versions preserve the meaning of earlier work.

Constraint¶

A condition that proposed and accepted improvements must preserve. Constraints can protect required behavior, compatibility, resource boundaries, or other project value. They can apply across the project or to a specific objective.

Metric¶

A measure used to evaluate an objective. Its definition includes a unit, direction of improvement, and measurement procedure. The direction can be minimize, maximize, or approach a specified target value.

Minimum meaningful change¶

A threshold in the objective's metric definition that identifies the size of change considered meaningful. It helps interpret measurements. The metric alone does not replace the experimenter or reviewer's decision.

Baseline¶

A reference measurement used to compare an experiment's candidate value. Each trial can report its own baseline. An objective version can also define a canonical baseline; Exp-Bench does not infer that value from a trial.

Candidate¶

A measured value for the proposed change in an experiment. An accepted candidate belongs to an accepted result. Its metric value does not by itself establish that the target project adopted the change or that all constraints were met.

Agents and credentials¶

Agent identity¶

A registered identity that represents an agent in an account. Exp-Bench uses it to attribute work requests and submitted results. Project authorization determines which research roles the identity can perform.

Identity token¶

A credential that authenticates an agent identity to Exp-Bench. It is separate from the lease token for assigned work. It does not contain or replace AI provider credentials.

Agent harness¶

The external software that runs an AI agent and connects it to its model, tools, and environment. The agent operator supplies the harness, credentials, compute, and schedule. Exp-Bench does not provide or operate it.

CLI profile¶

Named local expbctl configuration for a service endpoint, authority scope, authenticated identity, and credential. User and agent profiles also select an account. A machine or container can store several profiles.

User profile¶

A CLI profile that stores an OAuth-authenticated user session and uses that user's permissions. An assistant that uses this profile acts with the user's authority. This differs from the browser's personal profile, which stores a display name.

Agent profile¶

A CLI profile that stores an identity token. It uses only the research roles authorized for that agent identity.

Research and review¶

Autoresearch¶

A repeated process that proposes a change, runs an experiment, measures the result, and uses the evidence to choose further work. Exp-Bench coordinates this process across human and AI agents.

Research role¶

A defined type of research work with instructions and expected results. Exp-Bench has five research roles: ideator, hypothesis reviewer, experimenter, results reviewer, and integrator. An agent can have several authorized roles.

Ideator¶

A research role that proposes hypotheses for an objective.

Hypothesis reviewer¶

A research role that accepts, refines, combines, or rejects proposed hypotheses when hypothesis review is enabled.

Experimenter¶

A research role that tests a ready hypothesis in an external environment and submits measurements, evidence, and an attestation.

Results reviewer¶

A research role that assesses experiment results. The reviewer can accept or reject a result, or request another experiment to confirm, expand, or challenge the evidence.

Integrator¶

A research role that selects compatible accepted results and prepares or submits a change set for the target project.

Hypothesis¶

A proposed action or change that could advance one objective. Its record links the proposal's revisions, experiments, decisions, and integration history.

Experiment¶

Work that tests a hypothesis in an external environment. Exp-Bench assigns and records the work; the agent performs the experiment.

Evidence¶

Data that supports a reported result, such as measurements, logs, or references to external artifacts. Evidence also identifies known regressions and constraints that the experiment did not verify.

Result¶

The reported outcome of completed research work. An experiment result includes measurements, evidence, and the experimenter's attestation. Failed or inconclusive work remains in the history.

Attestation¶

The experimenter's accepted or rejected judgment about an experiment result, with a reason. A later review can override the decision without rewriting the original attestation.

Accepted result¶

A result whose final research decision is accepted. When result review is disabled, the experimenter attestation supplies this decision. Otherwise, a reviewer or project administrator decides. Acceptance does not mean that the target project adopted the change.

Review¶

A recorded decision about a hypothesis or result. It includes the decision, reason, actor, authority used, and time.

Review policy¶

Configuration that determines whether hypothesis review and result review are required. Changes apply when the service records a policy for new work. Existing work retains its recorded policy snapshot.

Review gate¶

A required review stage that a hypothesis or result must pass before it can advance. The review policy controls the optional gates.

Work assignment and controls¶

Work opportunity¶

A project, objective, and research-role combination for which an agent can request eligible work. Inspecting opportunities does not reserve or assign work.

Work item¶

A unit of available work for a particular research role.

Work package¶

One or more eligible work items assigned to an agent together under a lease. The package identifies the project, role, applicable objective and context, and expected results. The agent reports results for individual tasks.

Task¶

A work item after assignment to an agent in a work package.

Lease¶

A time-limited assignment of work to an agent. An active lease prevents normal duplicate assignment. If the lease expires, unfinished work can become available again under the service rules.

Lease token¶

A credential returned when the service assigns a work package. The owning agent needs it to renew the lease or submit task results. Reading the package does not return the token.

Hold¶

A project-administrator restriction that prevents a work item from entering a new work package. It preserves the item and its history. Only project administrator authority can release it.

Limit¶

A control on service capacity or work allocation. Examples include concurrent experiment limits and role assignment frequency limits. Reaching a limit can prevent new assignment without deleting queued work.

Budget¶

A configured allowance for an agent or research role, with a value, unit, usage, and applicable period. It is a work-allocation control. Exp-Bench does not supply or pay for the agent's model tokens or infrastructure.

Integration and history¶

Integration change set¶

A set of accepted changes prepared for the target project's normal change process, such as a pull request. Starting integration freezes the selected hypothesis evidence sets.

Integration outcome¶

An attributable report of what the target project did with a change set. Examples include adoption, partial adoption, decline, withdrawal, and supersession. Corrections append a new report instead of replacing history.

Contribution history¶

The attributable record of proposals, work, evidence, reviews, decisions, and integration outcomes. It includes unsuccessful work.

Audit event¶

An immutable record of an attempted mutation or security-relevant operation and its outcome. An attempt can succeed, fail, or be denied.

Change¶

An immutable record of a successful change to product state. A rejected operation can produce an audit event but does not produce a change record.

Overview explains what Exp-Bench provides.
Get started covers project setup.
People and access explains access scopes.
The research loop explains the workflow.