Mapping instances/results onto EvalPort's TestCase/ResultSet schema (portable interchange)

[adhabnr-ux]

Hi — I maintain EvalPort, an open interchange format (JSON Schema + Python/TS SDKs) for portable LLM eval datasets: TestCase/Grader in, Result/ResultSet/GraderResult out. I went through evaluation/evaluation.py and evaluation/validation.py on main (not the python-only NeurIPS branch — I noticed those two diverge on purpose, and the multi-language/Windows harness on main is the one I'm mapping against) to work out a concrete, non-lossy mapping, rather than propose something generic.

What the harness actually produces, read straight from the two files:

• Instance dict: instance_id, docker_image, rebuild_cmds/test_cmds/print_cmds (lists, joined with ;), test_patch, patch/pred_patch, log_parser (falls back to legacy parser), PASS_TO_PASS/FAIL_TO_PASS (lists of test names), created_at.
• evaluation.py's per-instance report: {instance_id, resolved: bool, PASS_TO_PASS: {success: [...], failure: [...]}, FAIL_TO_PASS: {success: [...], failure: [...]}}.
• evaluation.py's aggregate results.json: submitted/success/failure/error/incomplete counts plus the corresponding *_ids lists.
• validation.py's ValidationResult: adds pre_patch_status/post_patch_status (per-test pass/fail/skip maps) alongside the same PASS_TO_PASS/FAIL_TO_PASS.

Proposed mapping (verified against schema/testcase.json, schema/grader.json, schema/resultset.json in EvalPort, not just the dataclasses):

• TestCase.id = instance_id. TestCase.metadata["swebench_live"] carries docker_image, rebuild_cmds, test_cmds, print_cmds, test_patch, log_parser, PASS_TO_PASS, FAIL_TO_PASS, created_at verbatim, so a consumer can round-trip back to something evaluation.py can re-run. (input would carry the task/issue text — I didn't assert a field name for that since it's not touched by either file I read; happy to fill it in against the actual HF dataset schema if this is worth pursuing.)
• TestCase.graders = a single {"id": "swebench_live_harness", "type": "custom", "params": {"handler": "swe-bench-live:docker-harness"}}. It has to be custom — grading here means spinning up docker_image, applying test_patch, running rebuild_cmds/test_cmds, and parsing print_cmds output with log_parser, which is out-of-band execution, not something the in-process code grader (Python/JS sandbox per grader.json) can represent honestly.
• Per-instance Result.passed = report["resolved"]. Result.grader_results = two GraderResults, pass_to_pass (score = len(success)/len(PASS_TO_PASS), passed = no regressions) and fail_to_pass (score = len(success)/len(FAIL_TO_PASS), passed = full resolution) — this keeps the real regression/resolution distinction instead of collapsing to one boolean, and reason can carry the actual failing test names for debugging.
• ResultSet.summary = {total: submitted, passed: success, failed: failure} from results.json; error/incomplete (which summary has no dedicated slot for) go to ResultSet.metadata["swebench_live"]["aggregate"] verbatim.

from openeval import TestCase, Grader, Result, GraderResult

def to_test_case(instance: dict) -> TestCase:
    return TestCase(
        id=instance["instance_id"],
        input=instance.get("input", ""),  # task/issue text — dataset-schema dependent, not in evaluation.py
        graders=[Grader(
            id="swebench_live_harness", type="custom",
            params={"handler": "swe-bench-live:docker-harness"},
        )],
        metadata={"swebench_live": {
            "docker_image": instance.get("docker_image"),
            "rebuild_cmds": instance.get("rebuild_cmds", []),
            "test_cmds": instance.get("test_cmds", []),
            "print_cmds": instance.get("print_cmds", []),
            "test_patch": instance["test_patch"],
            "log_parser": instance.get("log_parser", instance.get("parser", "")),
            "PASS_TO_PASS": instance["PASS_TO_PASS"],
            "FAIL_TO_PASS": instance["FAIL_TO_PASS"],
            "created_at": instance.get("created_at"),
        }},
    )

def to_result(report: dict) -> Result:
    p2p, f2p = report["PASS_TO_PASS"], report["FAIL_TO_PASS"]
    def rate(d): 
        total = len(d["success"]) + len(d["failure"])
        return (len(d["success"]) / total) if total else 1.0
    return Result(
        test_case_id=report["instance_id"],
        passed=report["resolved"],
        grader_results=[
            GraderResult(grader_id="pass_to_pass", type="custom", score=rate(p2p),
                         passed=len(p2p["failure"]) == 0, reason=str(p2p["failure"][:5]) or None),
            GraderResult(grader_id="fail_to_pass", type="custom", score=rate(f2p),
                         passed=len(f2p["failure"]) == 0, reason=str(f2p["failure"][:5]) or None),
        ],
    )

For precedent on the "wrap out-of-band execution as a custom grader, preserve the tool's native shape verbatim under a namespaced metadata key, don't fabricate a mapping to a grader type that needs params you don't have" approach, I'm following the pattern in EvalPort's lm-eval-harness-openeval-adapter — it's a real, tested adapter (21 tests against the actually-installed lm-eval package, not mocks) for EleutherAI/lm-evaluation-harness, another benchmark-runner-shaped tool rather than a single-metric library, which is the closer analog to SWE-bench-Live's harness than the RAG/agent-tracing adapters in the same repo.

Given evaluation.py/validation.py are a moving target across MultiLang/Windows/the frozen python-only branch, I'd rather check whether this is worth doing before writing a PR: would a to_openeval/from_openeval pair (as a small adapter, or a converter script alongside evaluation/) be something you'd want, and if so is main's multi-language harness the right target, or should I aim at the frozen Python splits instead? Genuinely happy to build it either way, or drop it if this isn't useful to you.

— Sahi, independent contributor (not affiliated with this project)

[Karina Kenan Li (njukenanli)]

Thanks for the detailed proposal. We definitely welcome this integration and would be happy to review a PR.

For TestCase.input, please use the SWE-bench-Live problem statement literally:

input=instance.get("problem_statement", "")

Please implement both conversion directions, to_openeval and from_openeval. You can put the converter in our repository under ./evaluation/EvalPort like this:

evaluation/
├── evaluation.py
├── validation.py
├── README.md
└── EvalPort/
    ├── README.md
    ├── converter.py
    └── test_converter.py

Please add a short section introducing EvalPort at the end of ./evaluation/README.md, and provide a detailed usage and conversion guide in ./evaluation/EvalPort/README.md.

The primary targets should be the schemas used by SWE-bench-Live/MultiLang and SWE-bench-Live/Windows on main.

The current evaluation/evaluation.py also retains backward compatibility with the old SWE-bench-Live/SWE-bench-Live Python task set. In particular, an old Python instance can be adapted using logic like the following, as specified in evaluation/evaluation.py:

rebuild_cmd = "" # dynamic languages like python do not require re-build/re-compile
test_cmd = (
    "cat > run_test.sh <<'CC_PROMPT'\n"
    + " ; ".join(instance.get("test_cmds", []))
    + "\nCC_PROMPT\n"
    + "bash run_test.sh > testlog.out 2>&1"
)
print_cmd = "cat testlog.out"

log_parser = r'''
from enum import Enum


def parse_log_pytest(log: str, test_spec: "TestSpec") -> dict[str, str]:
    """
    Copied from SWE-bench/swebench/harness/constants/__init__.py.
    """

    class TestStatus(Enum):
        FAILED = "FAILED"
        PASSED = "PASSED"
        SKIPPED = "SKIPPED"
        ERROR = "ERROR"
        XFAIL = "XFAIL"

    test_status_map = {}
    for line in log.split("\n"):
        if any(line.startswith(status.value) for status in TestStatus):
            if line.startswith(TestStatus.FAILED.value):
                line = line.replace(" - ", " ")
            test_case = line.split()
            if len(test_case) <= 1:
                continue
            test_status_map[test_case[1]] = test_case[0]
    return test_status_map


def parser(log: str) -> dict[str, str]:
    mapping = parse_log_pytest(log, None)
    for test, status in mapping.items():
        if "pass" in status.lower():
            mapping[test] = "pass"
        elif "skip" in status.lower():
            mapping[test] = "skip"
        else:
            mapping[test] = "fail"
    return mapping
'''

Users should follow the newer MultiLang/Windows schema. Compatibility with the old Python dataset is optional, but you are welcome to include it using the approach above.

Please include round-trip tests and validate the generated documents with EvalPort's actual schema validator. Also, if a Result contains GraderResults with the IDs pass_to_pass and fail_to_pass, the corresponding TestCase should declare graders with those IDs. Please preserve all native aggregate states, including empty_patch/empty_patch_ids, and preserve patch/pred_patch in the appropriate output fields or namespaced metadata so that the conversion remains non-lossy.

[adhabnr-ux]

Thanks for the detailed answer — locking in everything you specified:

• TestCase.input = instance.get("problem_statement", ""), as you gave it, rather than the placeholder I'd left unfilled.
• Both directions: to_openeval() and from_openeval(), not just export.
• In-repo at evaluation/EvalPort/ (converter.py, test_converter.py, README.md), plus a short EvalPort section appended to evaluation/README.md — I'll follow the file layout exactly as you laid it out.
• Primary target is main's MultiLang/Windows schema, with the old Python task set handled via the rebuild_cmd/test_cmd/print_cmd/log_parser adaptation logic you pasted from evaluation.py, included as optional backward-compat rather than the primary path.
• Grader-id/GraderResult-id consistency: TestCase.graders will declare pass_to_pass and fail_to_pass explicitly by those exact IDs, matching what the corresponding Result.grader_results use — I'll make sure the two are never allowed to drift.
• empty_patch/empty_patch_ids and patch/pred_patch preserved — I'll carry empty_patch/empty_patch_ids into ResultSet.metadata["swebench_live"]["aggregate"] alongside the other aggregate counts, and put patch/pred_patch into the per-instance Result.metadata["swebench_live"] (or TestCase.metadata["swebench_live"] for the source patch, Result for the predicted one) so nothing about the actual code change is lost.
• Round-trip tests validated against EvalPort's real schema validator (openeval.validate.validate_suite()/validate_result_set()), not just structural assertions.

I'll open the PR once this is built and tested against real instance/report data from both the MultiLang/Windows harness and, where feasible, the legacy Python path — thanks for giving me a concrete target instead of a vague "sounds good."

— Sahi