feat: add Terminal-Bench adapter and headless agent CLI #198
Conversation
ThomasK33
force-pushed
the
thomask33/10-12-add_terminal_bench_integration
branch
from
10bb4bb to
a70399a
Compare
|
@codex review |
|
Codex Review: Didn't find any major issues. Delightful! ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. |
ThomasK33
force-pushed
the
thomask33/10-12-add_terminal_bench_integration
branch
from
a70399a to
feea514
Compare
ammario
left a comment
ammario
left a comment
There was a problem hiding this comment.
tbh i just gave it a cursory review -- plz just make sure urself that there are no regressions -- as an aside i think the whole "aiService" middleware file is unnecessary abstraction but we can fix that later
ThomasK33
force-pushed
the
thomask33/10-12-add_terminal_bench_integration
branch
2 times, most recently
from
c53fe34 to
fd4dddb
Compare
This commit introduces a headless agent runner and an adapter for the Terminal-Bench framework, enabling automated, programmatic evaluation of the agent's capabilities. The core of this change is a new `AgentSession` class that encapsulates the logic for managing a single workspace session. This refactors logic out of `ipcMain`, allowing the agent core to be used in environments without an Electron UI, such as the new headless CLI. Key components added: - `src/debug/agentSessionCli.ts`: A CLI for running an agent session headlessly. It can be driven programmatically and supports JSON output. - `benchmarks/terminal_bench/`: A Python adapter for Terminal-Bench that packages the cmux application, installs it in a task container, and runs it against benchmark instructions using the agent CLI. - `Makefile` target `benchmark-terminal` and `docs/benchmarking.md` to provide an easy entrypoint and documentation for running benchmarks. - Integration tests for the new `agentSessionCli` to ensure its correctness.
ThomasK33
force-pushed
the
thomask33/10-12-add_terminal_bench_integration
branch
from
fd4dddb to
6b30e43
Compare
I don't think there should be any breaking changes here; this is primarily a refactoring to separate the "agentic" components from the IPC layer, as the benchmarks assume files are modified in place. |
2 participants
This commit introduces a headless agent runner and an adapter for the
Terminal-Bench framework, enabling automated, programmatic evaluation of
the agent's capabilities.
The core of this change is a new
AgentSessionclass that encapsulatesthe logic for managing a single workspace session. This refactors logic
out of
ipcMain, allowing the agent core to be used in environmentswithout an Electron UI, such as the new headless CLI.
Key components added:
src/debug/agentSessionCli.ts: A CLI for running an agent sessionheadlessly. It can be driven programmatically and supports JSON output.
benchmarks/terminal_bench/: A Python adapter for Terminal-Bench thatpackages the cmux application, installs it in a task container, and
runs it against benchmark instructions using the agent CLI.
Makefiletargetbenchmark-terminalanddocs/benchmarking.mdtoprovide an easy entrypoint and documentation for running benchmarks.
agentSessionClito ensure itscorrectness.