image courtesy of US EPA

About the benchmark problems . . .

This web-site has been created to facilitate the distribution of a set of benchamrk problems for simulation-based multilayer sorptive barrier design. The idea of converting these problems into benchmark problems came out of a workshop on "Derivative-Free Hybrid Optimization Methods for Solving Simulation-Based Problems in Hydrology" sponsored by the American Institute of Mathematics.

Download Benchmark Problems
Click the link above to download a zip file containing the sorptive design problems along with the necessary simulation engines and input files. Also included in the zip file is readme file that explains how to interface the optimizer of your choice with the benchmark problems.
 
The tables below contain lists of previously published solutions to the benchmark problems.

Note: only the Nighthawk simulation engine is distributed with the benchmark problems.

Click here for an Excel spreadsheet version of the tables.

Please contact L. Shawn Matott (lsmatott@buffalo.edu) to request table additions.
Previously published solutions using the Nighthawk simulation engine. . .
Organic
Contaminant		 Optimal
Cost
($/m2)		 100-year
Exit Mass
(g/m2)		 Coded Layer Properties
(L1, L2, L3, L4, L5, L6)		 Search
Algorithm		 Model
Runs		 Num.
Trials		 Solution Range [Reference] and Comments
1,2-DCB 16.65 2.34E-07 13 10 13 7 14 14 peHD-DDS 351 9 16.65 to 24.19 [4] Alternative layer encoding (cost-based)
1,2-DCB 16.65 2.34E-07 13 10 13 7 14 14 HD-DDS 529 9 16.65 to 24.19 [4] Alternative layer encoding (cost-based)
1,2-DCB 16.65 2.34E-07 13 10 13 7 14 14 HD-DDS 1194 9 16.65 to 16.84 [4] Alternative layer encoding (cost-based)
1,2-DCB 16.77 1.29E-06 10 8 13 13 14 14 BGA 1051 9 16.65 to 24.19 [4] Alternative layer encoding (cost-based)
1,2-DCB 16.83 4.43E-06 7 9 9 8 9 14 SA 5122 9 16.65 to 24.19 [4] Alternative layer encoding (cost-based)
1,2-DCB 16.95 2.12E-06 9 8 9 9 8 14 pePSO 661 9 16.83 to 41.56 [4]
1,2-DCB 16.95 2.12E-06 9 8 9 9 8 14 PSO 1051 9 16.83 to 41.56 [4]
1,2-DCB 24.19 3.84E-06 13 9 7 7 9 8 BGA 1051 9 16.94 to 24.31 [4]
1,2-DCB 24.20 2.39E-06 7 9 8 8 7 7 pePSO 1020 9 16.84 to 24.31 [4] Alternative layer encoding (cost-based)
1,2-DCB 24.20 2.39E-06 7 9 8 8 7 7 PSO 1051 9 16.84 to 24.31 [4] Alternative layer encoding (cost-based)
1,2-DCB 24.20 2.39E-06 7 9 8 8 7 7 SA 5122 9 16.65 to 24.49 [4]
Benzene 16.54 1.12E-13 13 13 13 10 14 14 Exhaustive 28561 n/a n/a to n/a [3] Alternative isotherms used in model
Benzene 16.66 1.48E-20 13 13 10 7 14 14 Exhaustive 28561 n/a n/a to n/a [3] Alternative isotherms used in model
Benzene 25.22 2.38E-07 13 13 10 10 14 14 Exhaustive 28561 n/a n/a to n/a [3] Alternative isotherms used in model
Benzene 41.62 4.90E-06 7 10 10 10 13 14 HD-DDS 1523 9 41.62 to 41.74 [4] Alternative layer encoding (cost-based)
Benzene 41.74 1.93E-06 13 10 10 10 8 14 BGA 10101 9 41.62 to 41.98 [4]
Benzene 41.74 1.93E-06 13 10 10 10 8 14 BGA 10101 9 41.62 to 41.74 [4] Alternative layer encoding (cost-based)
Benzene 41.75 3.76E-06 10 7 10 10 7 14 pePSO 8205 9 41.74 to 49.33 [4]
Benzene 41.75 3.76E-06 10 7 10 10 7 14 PSO 10101 9 41.74 to 49.33 [4]
Benzene 41.86 2.16E-06 10 10 13 10 9 14 SA 5122 9 41.62 to 49.33 [4] Alternative layer encoding (cost-based)
Benzene 42.72 2.94E-06 10 10 11 7 14 14 peHD-DDS 415 9 41.62 to 50.31 [4] Alternative layer encoding (cost-based)
Benzene 42.72 2.94E-06 10 10 11 7 14 14 HD-DDS 570 9 41.62 to 50.31 [4] Alternative layer encoding (cost-based)
Benzene 49.33 5.84E-07 7 10 7 10 13 10 pePSOd 2677 9 41.74 to 49.56 [4] Alternative layer encoding (cost-based)
Benzene 49.33 5.84E-07 7 10 7 10 13 10 SA 3309 9 41.86 to 49.57 [4]
Benzene 49.33 5.84E-07 7 10 7 10 13 10 PSO 3364 9 41.74 to 49.56 [4] Alternative layer encoding (cost-based)
TCE 8.79 4.29E-06 9 8 8 7 14 14 peHD-DDS 262 9 8.79 to 16.26 [4] Alternative layer encoding (cost-based)
TCE 8.79 4.29E-06 9 8 8 7 14 14 HD-DDS 456 9 8.79 to 16.26 [4] Alternative layer encoding (cost-based)
TCE 8.79 4.29E-06 9 8 8 7 14 14 HD-DDS 1137 9 8.79 to 16.26 [4] Alternative layer encoding (cost-based)
TCE 8.79 4.29E-06 9 8 8 7 14 14 SA 2152 9 8.79 to 16.26 [4] Alternative layer encoding (cost-based)
TCE 8.79 4.29E-06 9 8 8 7 14 14 BGA 2551 9 8.79 to 16.26 [4] Alternative layer encoding (cost-based)
TCE 8.90 2.35E-06 9 7 8 9 14 14 BGA 2551 9 8.79 to 16.37 [4]
TCE 16.26 7.34E-07 7 9 7 7 7 14 pePSO 2278 9 16.26 to 23.74 [4]
TCE 16.26 7.34E-07 7 9 7 7 7 14 pePSO 3181 9 8.79 to 23.85 [4] Alternative layer encoding (cost-based)
TCE 16.26 7.34E-07 7 9 7 7 7 14 PSO 3547 9 8.79 to 23.85 [4] Alternative layer encoding (cost-based)
TCE 16.26 7.34E-07 7 9 7 7 7 14 PSO 4520 9 16.26 to 23.74 [4]
TCE 23.73 2.10E-06 13 9 7 13 7 7 SA 2152 9 8.79 to 23.86 [4]
Previously published solutions using the Mouser simulation engine. . .
Organic
Contaminant		 Optimal
Cost
($/m2)		 100-year
Exit Mass
(g/m2)		 Coded Layer Properties
(L1, L2, L3, L4, L5, L6)		 Search
Algorithm		 Model
Runs		 Num.
Trials		 Solution Range [Reference] and Comments
1,2-DCB 11.13 2.05E-05 9 9 9 9 14 14 GA+PSO n/a n/a n/a to n/a [1] (infeasible if using 5x10-6 exit constraint)
1,2-DCB 16.66 2.91E-06 7 10 13 13 14 14 PSO 1051 10 16.66 to 41.50 [2] Tuned algorithm settings
1,2-DCB 16.72 1.36E-06 13 8 9 9 9 14 BGA 2101 10 16.66 to 16.83 [2] Nominal algorithm settings
1,2-DCB 16.72 9.11E-07 13 9 8 9 9 14 BGA 1051 10 16.66 to 24.19 [2] Tuned algorithm settings
1,2-DCB 16.72 1.80E-06 7 9 9 9 7 14 SA 5127 10 16.66 to 24.37 [2] Tuned algorithm settings
1,2-DCB 16.72 4.17E-06 7 7 9 9 9 14 PSO 2101 10 16.66 to 24.20 [2] Nominal algorithm settings
1,2-DCB 16.72 1.65E-06 7 9 8 9 8 14 RGA 5101 10 16.72 to 16.84 [2] Tuned algorithm settings
1,2-DCB 24.08 3.37E-06 13 9 7 8 8 7 RGA 2101 10 16.72 to 24.19 [2] Nominal algorithm settings
1,2-DCB 24.31 3.69E-08 13 9 8 9 8 7 SA 2072 10 17.24 to 32.19 [2] Nominal algorithm settings
Benzene 34.27 3.27E-06 9 10 10 10 14 14 BGA 7201 10 34.27 to 41.86 [2] Tuned algorithm settings
Benzene 34.38 3.27E-06 9 10 10 10 14 14 GA+PSO n/a n/a n/a to n/a [1] (used exit constraint of 0 g/m2)
Benzene 37.49 8.22E-06 10 10 9 10 14 14 PSO 10101 10 34.27 to 41.74 [2] Tuned algorithm settings
Benzene 41.75 3.95E-06 13 10 10 8 10 14 PSO 2101 10 41.74 to 49.45 [2] Nominal algorithm settings
Benzene 41.86 6.93E-07 9 10 13 10 10 14 BGA 2101 10 34.27 to 49.79 [2] Nominal algorithm settings
Benzene 41.97 1.38E-06 7 9 10 10 10 14 RGA 4201 10 41.75 to 49.45 [2] Tuned algorithm settings
Benzene 42.20 3.85E-07 10 9 10 10 9 14 SA 2577 10 41.86 to 49.91 [2] Tuned algorithm settings
Benzene 49.33 3.54E-06 10 10 13 8 10 13 RGA 2101 10 41.86 to 49.68 [2] Nominal algorithm settings
Benzene 49.39 3.70E-07 9 13 10 10 13 10 SA 2072 10 41.86 to 50.78 [2] Nominal algorithm settings
TCE 8.66 4.71E-06 13 9 7 9 14 14 PSO 5101 10 8.66 to 16.37 [2] Tuned algorithm settings
TCE 8.70 3.54E-06 13 9 8 8 14 14 GA+PSO n/a n/a n/a to n/a [1] (used exit constraint of 0 g/m2)
TCE 8.78 4.89E-06 9 9 8 13 14 14 BGA 2101 10 8.66 to 16.26 [2] Nominal algorithm settings
TCE 8.78 7.90E-07 7 7 9 9 14 14 BGA 1101 10 8.66 to 16.26 [2] Tuned algorithm settings
TCE 16.14 1.59E-06 13 9 7 7 7 14 PSO 2101 10 8.67 to 23.84 [2] Nominal algorithm settings
TCE 16.26 3.40E-06 8 13 7 7 9 14 RGA 2551 10 8.79 to 16.26 [2] Tuned algorithm settings
TCE 16.26 2.53E-06 7 7 13 9 8 14 SA 2152 10 16.15 to 23.86 [2] Tuned algorithm settings
TCE 16.38 7.69E-09 7 8 7 8 8 14 RGA 2101 10 16.27 to 16.72 [2] Nominal algorithm settings
TCE 16.43 1.02E-08 7 7 9 7 8 14 SA 2072 10 9.13 to 24.08 [2] Nominal algorithm settings
Table definitions . . .
Organic Contaminant - The type of pollutant that the barrier is being designed to contain. Different pollutants have different sorptive properties so each pollutant corresponds to a different design problem.

Optimal Cost - The optimal cost reported by a given solution method, or the median optimal cost if multiple trials were performed.

100-year exit mass - The predicted cumulative amount of pollutant mass exiting the liner after 100 years of transport simulation. The prescribed performance criteria is <5x10-6 g/m2.

Coded layer properties -  Specifies the optimal configuration of the barrier layers in terms of coded values. See readme file for mapping of coded values to corresponding layer composition.

Simulation engine - The modelling program used for computing the 100-year exit mass. Use of the Mouser engine is not supported, only the Nighthawk engine is distributed with the benchmark problems.

Search algorithm - The optimization algorithm. Abbreviations are as follows:
  • BGA - Binary-coded Genetic Algorithm
  • Exhaustive - All possible layer configurations were evaluated
  • GA+PSO - Genetic Algorithm followed by Particle Swarm Optimization
  • PSO - Particle Swarm Optimization
  • RGA - Real-coded Genetic Algorithm
  • SA - Simulated Annealing
  • pePSO - pre-emption enabled PSO
  • DDS - Dynamically Dimensioned Search
  • peDDS - pre-emption enabled DDS

Model runs - Number of evaluations of the simulation engine required by a single trial of the search algorithm.

Num. Trials - Number of trials of the optimization search algorithm. Many search algorithms have stohchastic behavior and multiple applications of the same algorithm may yield different results. The performance of such algorithms can be assessed by examining the range and central tendency of results generated from multiple trials.

Solution range - The minimum and maximum cost reported by the search algorithm if multiple trials were performed.
 
References . . .
[1] Bartelt-Hunt, S.L., T.B. Culver, J.A. Smith, L.S. Matott, and A.J. Rabideau, Optimal Design of a Compacted Soil Liner Containing Sorptive Amendments, Journal of Environmental Engineering, 132(7), 769-776, 2006.

[2] Matott, L.S., S.L. Bartelt-Hunt, A.J. Rabideau, and K.R. Fowler, Application of Heuristic Optimization Techniques and Algorithm Tuning to Multi-layered Sorptive Barrier Design, Environmental Science & Technology, 40(20), 6354-6360, 2006.

[3] Matott, L.S., A.J. Rabideau, and K. Bandilla, Incorporating Nonlinear Isotherms into Robust Multilayer Sorptive Barrier Design, Advances in Water Resources, 23(11), 1641-1651 2009.

[4] Matott, L.S., B. Tolson, and M. Asadzadeh, A Benchmarking Framework for Simulation-based Optimization of Environmental Models, Environmental Modelling & Software, submitted October, 2010.