It looks like in cbook.boxplot_stats it is automatically generating labels if none are set based upon incrementing numbers

    if labels is None:
        labels = [str(i) for i in range(1, ncols+1)]
    elif len(labels) != ncols:
        raise ValueError("Dimensions of labels and X must be compatible")

Then in the axes.bxp it uses the position (pos) only if a label isn't set but it is automatically generated above so the position in this case will always be ignored.

            # try to find a new label
            datalabels.append(stats.get('label', pos))

Thanks for tracking that down. I have possible fix up.

@kaufman Can you verify that #3566 fixes the problem?

Yes, it fixes the problem.