BatchRunner swaps content of variable_params columns, only in random proportion of executions

[peterdekker]

Describe the bug
When using BatchRunner, and calling get_model_vars_dataframe(), the contents of the columns in the output of two variable parameters are swapped: the column heads stay the same, but matches with the parameter values of another parameter. I cannot tell yet whether it is just the columns of the DataFrame that is wrong, or also the batch run with the wrong values.

The remarkable thing is: when I execute my script several times, with the same code, it only happens in a number of executions. Maybe the behaviour of the bug is dependent on a random value?

Expected behavior
I expect the BatchRunner model vars dataframe to give back the variable parameter values that correspond with a certain parameter. Instead, the parameter values of one parameter are swapped with the parameter values of another parameter.

To Reproduce
I do not have a minimum working example. I have some code which depends on my specific model:

from mesa.batchrunner import BatchRunner
from constants import HEIGHT, WIDTH, PROPORTION_L2, SUFFIX_PROB, CAPACITY_L1, CAPACITY_L2, \
                        DROP_SUBJECT_PROB, DROP_OBJECT_PROB, MIN_BOUNDARY_FEATURE_DIST
from Model import Model

fixed_params = {
    "height": HEIGHT,
    "width": WIDTH,
    "proportion_l2": PROPORTION_L2,
    "suffix_prob": SUFFIX_PROB,
    "capacity_l1": CAPACITY_L1,
    "capacity_l2": CAPACITY_L2,
    "drop_subject_prob": DROP_SUBJECT_PROB,
    "drop_object_prob": DROP_OBJECT_PROB,
    "min_boundary_feature_dist": MIN_BOUNDARY_FEATURE_DIST
}

variable_params = {"proportion_l2":[0.0, 0.2, 0.5], "capacity_l1": [0.0,1.0,2.0,5.0]}

for k in variable_params:
    del fixed_params[k]

stats = {"global_filled_prefix_l1": lambda m: m.global_filled_prefix_l1,
         "global_filled_suffix_l1": lambda m: m.global_filled_suffix_l1,
         "global_filled_prefix_l2": lambda m: m.global_filled_prefix_l2,
         "global_filled_suffix_l2": lambda m: m.global_filled_suffix_l2}

batch_run = BatchRunner(
    Model,
    variable_params,
    fixed_params,
    iterations=1,
    max_steps=1,
    model_reporters=stats
)

batch_run.run_all()
cols = list(variable_params.keys()) + list(stats.keys())
run_data = batch_run.get_model_vars_dataframe()
run_data = run_data[cols]
print(run_data)
run_data.to_csv("evaluation.tsv", sep="\t")

It gives the following output:

12it [00:02,  5.13it/s]
    proportion_l2  capacity_l1  global_filled_prefix_l1  global_filled_suffix_l1  global_filled_prefix_l2  global_filled_suffix_l2
0             0.0          0.0                      0.0                      0.0                      0.0                      0.0
1             1.0          0.0                      0.0                      0.0                      0.0                      0.0
2             2.0          0.0                      0.0                      0.0                      0.0                      0.0
3             5.0          0.0                      0.0                      0.0                      0.0                      0.0
4             0.0          0.2                      0.0                      0.0                      0.0                      0.0
5             1.0          0.2                      0.0                      0.0                      0.0                      0.0
6             2.0          0.2                      0.0                      0.0                      0.0                      0.0
7             5.0          0.2                      0.0                      0.0                      0.0                      0.0
8             0.0          0.5                      0.0                      0.0                      0.0                      0.0
9             1.0          0.5                      0.0                      0.0                      0.0                      0.0
10            2.0          0.5                      0.0                      0.0                      0.0                      0.0
11            5.0          0.5                      0.0                      0.0                      0.0                      0.0

In the output, the values of ther parameters capacity_l1 and proportion_l2 have been swapped, if you compare them how these have been initialized.

Additional context
The bug occurs on two different Ubuntu machines, with mesa 0.8.7 or 0.8.8. The bug doesn't occur in mesa 0.8.6.

[misken]

I came here to report the very same behavior. I was getting strange behavior of BatchRunner get_model_vars_dataframe when passing a list of non-integers for variable_params. Decided to test on the Money Model to make sure it wasn't something in my model. Added a new model parameter initial_wealth and then ran via:

fixed_params = {"width": 10,
               "height": 10}

# The following leads to column names being swapped (N ends up as col name for
# the initial wealth values and vice versa) in the dataframe returned by get_model_var_dataframe()
variable_params = {"N": range(10, 30, 10),
                  "initial_wealth": [1.0, 1.5, 2.0]}

#If I use this version and simply swap the dict entries, the column names weren't swapped
#variable_params = {"initial_wealth": [1.0, 1.5, 2.0],
#                   "N": range(10, 30, 10)}

batch_run = BatchRunner(MoneyModel,
                        variable_params,
                        fixed_params,
                        iterations=5,
                        max_steps=100,
                        model_reporters={"Gini": compute_gini})
batch_run.run_all()

run_data = batch_run.get_model_vars_dataframe()
print(run_data.head(20))

Results in:

initial_wealth    N  Run      Gini  width  height
0               10  1.0    0  0.580000     10      10
1               10  1.0    1  0.620000     10      10
2               10  1.0    2  0.540000     10      10
3               10  1.0    3  0.580000     10      10
...

[YannickJadoul]

The use of set seems to be the issue, here.
This seems to have been introduced by #651

This (reasonably naive) diff seems to do the trick, when I test @misken's reproducing example:

diff --git a/mesa/batchrunner.py b/mesa/batchrunner.py
index 0c43c58..45467b8 100644
--- a/mesa/batchrunner.py
+++ b/mesa/batchrunner.py
@@ -257,10 +257,10 @@ class FixedBatchRunner:
         column as a key.
         """
         extra_cols = ["Run"] + (extra_cols or [])
-        index_cols = set()
+        index_cols = []
         for params in self.parameters_list:
-            index_cols |= params.keys()
-        index_cols = list(index_cols) + extra_cols
+            index_cols.extend(key for key in params.keys() if key not in index_cols)
+        index_cols = index_cols + extra_cols
 
         records = []
         for param_key, values in vars_dict.items():

I don't fully understand the subtleties of the code (and the dict(zip(index_cols, param_key)) that follows), so I'm not yet submitting this as PR. But as far as I can tell, this fixes the issue.

[YannickJadoul]

Should be fixed by #979, if anyone wants to check or review ;-)

[tpike3]

@YannickJadoul Thanks Yannick I will get it reviewed.

[peterdekker]

Sorry for a possibly bold question, I don't know about the release schedule of mesa, but would it be possible to release a new version of mesa, also on PyPI, which includes this bugfix? Right now, one has to use the old version 0.8.6 or check out the master branch. The last available release contains the faulty behaviour, which might confuse users.