View on GitHub

General Program Synthesis Benchmark Suite

Thomas Helmuth, Lee Spector



T. Helmuth and L. Spector. General Program Synthesis Benchmark Suite. In GECCO '15: Proceedings of the 17th annual conference on Genetic and evolutionary computation. July 2015. ACM.


Recent interest in the development and use of non-trivial benchmark problems for genetic programming research has highlighted the scarcity of general program synthesis (also called "traditional programming") benchmark problems. We present a suite of 29 general program synthesis benchmark problems systematically selected from sources of introductory computer science programming problems. This suite is suitable for experiments with any program synthesis system driven by input/output examples. We present results from illustrative experiments using our reference implementation of the problems in the PushGP genetic programming system. The results show that the problems in the suite vary in difficulty and can be useful for assessing the capabilities of a program synthesis system.

Full paper

PDF of full paper

Supplementary materials

Tech Report

We published a tech report, which expands the paper by adding significant details necessary to implement the benchmarks.


PDF of slides presented at GECCO 2015


Repository of datasets for each problem. Each dataset contains input and output examples, and can be sampled for multiple evolutionary or program synthesis runs.

Source code

An archive of the source code for Clojush, the Clojure version of Push used to run our experiments, can be found here. The most recent version of Clojush can be found in its GitHub Repository.