Benchmarks

Two types of benchmarks have been performed on a large set of scheduling problems

• extensive benchmarks against IBM CPLEX CP Optimizer
• benchmarks against the state-of-the-art

Benchmarks against the state-of-the-art

Benchmarks against the state of the art are available as a separate GitHub site

In the benchmarks you will find

• a mathematical description of the problem and its main variants
• the instance files (usually > 1000) classified in easy, medium, hard or open
• a description of the most common formats
• the best known solutions from published results or engines (Cplex, CP Optimizer, OR-tools, OptalCP)
• list of relevant publications
• subsets of problems of interest (large, small but still open)

The most advanced state-of-the-art benchmark as of today are

• Jobshop benchmark
• Openshop benchmark

The very first version of the benchmark was built on top of the outstanding work of Naderi, Ruiz and Roshanaei Mixed-Integer Programming versus Constraint Programming for shop scheduling problems : New Results and Outlook which compares CPO, Cplex and Gurobi on a benchmark of 6623 instances over 17 benchmarks with a timeout of 1h.

Because running multiple solvers on thousand of instances takes time, this benchmark is still under construction. We are working to add other type of problems (rcpsp and workforce scheduling)

Benchmarks against IBM ILOG CP Optimizer

Detailed benchmarks against CP Optimizer are available here

• Both solvers are using 4 workers (threads)
• CP Optimizer is using the default setting.
• OptalCP uses half of the workers by LNS (with default propagation settings) and the second half by Failure-Directed Search (with maximum propagation settings)

Openshop benchmark is an exception. It is too easy and so it is solved with only 2 workers for both solvers.

Benchmark	Results	Modeling features
Jobshop	Results	endBeforeStart, noOverlap
Jobshop with Operators	Number of operators: 4, 5, 6, 7, 8, 9, 10	endBeforeStart, noOverlap, pulse, cumulLe
Flexible Jobshop	Results	alternative, endBeforeStart, noOverlap
Jobshop TTJobshop with sequence-dependent Transition Times	Results	endBeforeStart, noOverlap with transition times
Openshop	Results	noOverlap
RCPSPResource Constrained Project Scheduling Problem	Results	endBeforeStart, pulse, cumulLe
MMRCPSPMulti-Model Resource Constrained Scheduling Problem	PSPLIB, MMLIB50, MMLIB100	alternative, pulse, cumulLe, non-renewable resources (presenceOf, times, sum, le)
RCPSP-CPRRCPSP with Consumption and Production of Resources	Results	endBeforeStart, pulse, cumulLe, reservoir (stepAtStart, stepAtEnd, cumulNeg, cumulSum, cumulGe, cumulLe)