creating many tasks from a single container image with RepoLaunch

[Daniel Zayas (danielzayas)]

Congratulations on SWE-bench Goes Live! My name is Daniel Zayas. During my personal time, I contribute to open source projects for scaling SWE benchmark generation.

Question

For SWE-bench-Live, what are your thoughts/feelings on a future RepoLauch version that supports create many task instances from a single docker image (like SWE-smith)?

Background

"SWE-bench relies heavily on manual effort for constructing test instances and setting up executable environments". - https://arxiv.org/html/2505.23419v2

"Constructing instances for SWE-bench-like task instances involves substantial human labor: identifying appropriate issue-resolution pairs, locating relevant tests, configuring runnable environments, composing test commands, and validating the full workflow. For instance, it take about one year for Multi-SWE-bench zan2025multi to create 1,632 benchmark instances with 68 expert annotators." - https://arxiv.org/html/2505.23419v2

SWE-smith versus RepoLauch

I (Daniel Zayas (@danielzayas)) contribute to open source applications for scaling SWE task instance generation. Automating the creation of containerized environments has been challenging historically. My favorite technical visions help address this challenge are:

1. SWE-smith creates one containerized environment for a repository (e.g. astroid version b114f6b5) then creates many task instance for the one container image (e.g. 81 tasks for one astroid version b114f6b5 image). SWE-smith was used to generate the SWE-bench/SWE-smith dataset, which contains over 59000+ tasks.
2. SWE-bench-Live's RepoLauch launch/run.py creates one containerized environment per task (and pushes it to a remote container repository if push=True in runtime.py). The SWE-bench-Live/SWE-bench-Live dataset contains over 1890+ tasks.

Challenges Creating Containerized Environments

I've experienced multiple challenges when trying to create one containerized environment per task:

• dependencies changes over time (and past version may not be available)
• build tools and processes change over time (and past build tool versions may not be available)

SWE-bench-Live's curation/crawl_repo.py accepts a minimum --cutoff-date argument and curation/swe_task_crawling/fetch_pulls.py respects this cutoff date. That argument can help minimize the affects of those challenges, but it has two downsides:

• it limits the PR-Issue pairs that can be use to create new task instances
• it does not directly address the challenges (nor necessarily prevent them)

[Karina Kenan Li (njukenanli)]

Yes you are right. Currently our setting -- one image for one instance is too heady.
However, checking which commits can rely on the same base image is not easy. Even with dependency file and cutoff date the dependencies still cannot be determined definitely. Some dependencies are installed in the OS, not in the virtual env of the project, which cannot be decided from the dependency file.
Previous works like SWE-Smith involves human effort for such verification. The automation of sharing images across instances requires much effort.
But, yes, it is in our future works and todo list.
Thank you very much for your continuous interest in out project!

[Daniel Zayas (danielzayas)]

Great news! I agree that the SWE-smith versus SWE-bench-Live approaches each have their respective benefits/downsides. In SWE-smith, older dependency stacks may need separate profiles/images, which is also supported. I'm happy to hear that you'd like RepoLaunch to support both approaches!

Adding support for this is something I'm interested in contributing PRs for. However, a) it's quite a large change and b) you many have other ideas/thoughts/feelings on contributions that may be more urgent or more valuable. For contributing PRs to the project, would you recommend this feature, other ideas from my email, or other?

[Daniel Zayas (danielzayas)]

👋 Hi Karina Kenan Li (@njukenanli) ! I'm writing to follow-up on my question from 2025-12-12! ⬆️

[Karina Kenan Li (njukenanli)]

Hi Daniel,

Once again, thank you for your enthusiasm.
We are considering about this these days.
We think 2 types of image "merging" might be feasible:

1. To build image for a new commit A, try to check out commit from already built images of the same repo with the nearest 2 commits before and after commit A, and run rebuild and test command again. If rebuild returns 0 exit code and for test, most, say, 80% of tests still pass, then simple checkout is enough, and we commit the container to a new image with only a newly added checkout layer.
2. For those simple checkout does not resolve -- dependencies have diverged, we may run agent on the already built images of the same repo with the nearest commit to reinstall PART of the changed dependencies, but this will add heavier layers on the top. One lighter but harder way is to prompt LLM to select which bottom layers of the built image can be reused, start a container on those bottom layers that can be reused, and run agent on the container.

We welcome contributions to this image merging project.
For modifications that can be generalized to building any repositories, commit to RepoLaunch.
For modifications that are only applicable for SWE-bench-Live task creation, commit to SWE-bench-Live.

Best regards

[Karina Kenan Li (njukenanli)]

Your suggestion to share a docker image for multiple different issues (or commits) of the same repo is implemented and recent dataset updates adopt this strategy. See pull request 52: #52.