BUG: Fix numpy.random.dirichlet returns NaN for small 'alpha' parameters. #14924

trendelkampschroer · 2019-11-17T22:02:42Z

BUG: Fix numpy.random.dirichlet returns NaN for small 'alpha' parameters.

The PR introduces a second code branch into the 'dirichlet' method of 'Generator'. This branch will be executed whenever the maximum of all 'alpha' parameters for the dirichlet distribution is smaller than one. In this case the random vector will be generated by the stick breaking approach, cf. https://en.wikipedia.org/wiki/Dirichlet_process.

When the 'alpha' smallness condition is not triggered, i.e. 'alpha.max() >= 1.' the existing code path is executed.

Closes #5851.

…r small alpha (and beta) values

WarrenWeckesser · 2019-11-19T01:52:49Z

I haven't reviewed the code line by line, but something is not correct. In the following large sample, the mean of the last component is not close to the theoretical expected value:

In [49]: alpha = np.array([0.15, 0.15, 0.2])

In [50]: x = rng.dirichlet(alpha, size=1000000)

In [51]: alpha/alpha.sum()   # Expected values
Out[51]: array([0.3, 0.3, 0.4])

In [52]: x.mean(axis=0)   # Sample mean
Out[52]: array([0.30014439, 0.30044703, 0.42944236])

The sample mean of the last component is far from the expected value of 0.4. The difference is too big to be accounted for by sample variation. If you plot a histogram of x[:,2], you'll see that it is quite different from the expected beta distribution.

numpy/random/_generator.pyx

WarrenWeckesser

We need a test that would have caught the bug that I described above. For example, something like

    @pytest.mark.slow
    def test_dirichlet_with_moderately_small_alphas():
        """For a large sample, compare the sample mean to the expected mean."""
        alpha = np.array([0.15, 0.15, 0.2])
        expected_mean = alpha / alpha.sum()
        rng = np.random.default_rng(11335577)  # Or no seed?
        x = rng.dirichlet(alpha, size=20000000)
        sample_mean = x.mean(axis=0)
        assert_allclose(sample_mean, expected_mean, rtol=1e-3)

(It could be added to the other dirichlet tests in TestRandomDist, in which case it needs a self argument, and the random seed could be self.seed.)

The rationale for deciding which method to choose should be documented. Here are some notes along those lines:

The old code generates variates from gamma variates, where each alpha value is used as the shape parameter of a gamma distribution. The gamma variates are normalized to have a sum 1. If all the alpha values are sufficiently small, there is a high probability that all the gamma variates will be 0. When that happens, the normalization process ends up computing 0/0, giving nan.

The condition for using the new branch, which is based on beta variates, is alpha_arr.max() < 1.0. Is this a good choice? Why?

It is certainly the correct choice for an example such as alpha = [1e-3, 1e-4, 1e-3]. But what about something like alpha = [1.25, 1e-3, 1e-7]? In the current version of the PR, the old code will be used.

In the old code, the random variates are normalized gamma variates, and the problem occurs when all parameters are so small that all the gamma variates are 0, breaking the normalization code. The PDF for the gamma distribution has the term x**(alpha - 1), so if alpha > 1, the probability of the random variate being very close to 0 is extremely small, and effectively we can be sure it will never be 0. So the sum of the gamma variates will also never be 0, and we won't ever compute 0/0.

We could lower the threshold from 1.0 to something moderately smaller, and still still be confident that the gamma variate associated with the maximum alpha value will (effectively) never be 0, but deciding by how much we can safely lower the threshold will require some more work. I'm not sure we need to do that.

The motivation for lowering the threshold is performance. The old code tends to be faster than the new method. For example,

In [106]: alpha = np.array([0.15, 0.15, 1.0])   # max is 1, so old code is used.

In [107]: %timeit rng.dirichlet(alpha, size=100000)
27.6 ms ± 747 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [108]: alpha = np.array([0.15, 0.15, np.nextafter(1.0, 0)])  # max < 1, so new code is used.

In [109]: %timeit rng.dirichlet(alpha, size=100000)
38.3 ms ± 1.26 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In this case, the new code take about 1.4 times as long. If folks are interested in avoiding the performance hit as much as possible, we'll have to work on choosing a lower threshold.

WarrenWeckesser · 2019-11-20T00:09:11Z

numpy/random/_generator.pyx

-                    val_data[i + j] = val_data[i + j] * invacc
-                i = i + k
+
+        # Small alpha case: Use stick-breaking approach with beta random variates


PEP 8: Keep line lengths 79 characters or less (here and elsewhere).

@WarrenWeckesser : Thanks a lot for the extensive review. So far I have been able to address the following:

Adding the suggested test

Fixing the formatting issues.

As for documenting the rationale in the code change: Thanks a lot for your suggestions. I haven't found time to do a formal analysis giving the probability of generating gamma variates which sum to zero for a given vector of alpha parameters. That could potentially help to determine a new threshold for the switch between the two code branches.

If I remember correctly from the past discussion the point of adding the new code branch is to ensure that the old code branch is never called when there is positive probability of returning NaN. I think this is why we ended up with one being the threshold in the first place, but I'd need to backtrack the argument again to be 100% sure.

I'm fine with keeping the threshold at 1. @bashtage, @rkern, any opinions?

Can you expand the comments in the code with a sentence or two about why there are two methods? E.g. the normalized gamma variates method is faster than the "stick breaking" method, but it generates nans when all the gamma variates happen to be 0, and that can happen when all the alpha values are sufficiently small.

Will do that, as soon as possible.

I have added an inline docstring that provides the rationale for the switch to the stick-breaking approach.

As for the choice of the threshold, the probability for a gamma RV to be smaller than the smallest postive floating precision number can be computed using the regularized incomplete gamma function

import numpy as np from scipy.special import gammainc delta = np.finfo(np.float).tiny delta params = [1.0e-5, 1.0e-4, 1.0e-3, 1.0e-2, 1.0e-1, 1.] probs = [gammainc(p, delta) for p in params] probs >>>[0.9929467992914456, 0.9316650395588774, 0.4927171386203597, 0.0008432274068068666, 1.8046532550200846e-31, 2.2250738585072626e-308]

With these numbers one could argue that if alpha.max() >= 0.1 the probability of generating all zeros is still so tiny that it can be neglected and the threshold can be reduced to 0.1

Not sure if that is small enough though?

Nice. I was working through similar reasoning, using np.finfo(np.float).tiny as the threshold for "numerically zero", and came to roughly the same conclusion. With the revised threshold condition max(alpha) > 0.1, a conservative upper bound on the probability of generating all "tiny" gamma variates is 1.8e-31, so I think a threshold of 0.1 should be safe. For comparsion, here are some other values for max(alpha) and the associated upper bounds for the probabilty that all the gamma variates are "tiny"

max(alpha) p ---------- --------- 0.05 4.3e-16 0.1 1.8e-31 0.2 3.2e-62 0.25 1.3e-77 0.5 1.7e-154 1.0 2.2e-308

@bashtage, what do you think? Does the threshold 0.1 sound reasonable?

Thanks a lot for the great comments and suggestions!

I have changed the threshold from 1. to 0.1 to:

Not unneccesarily 'break the stream' compared to the legacy method of generation

Use the method with a higher performance whenever the probability of failure is acceptable
From the previous discussion I understand that a probabilty of failure < 1.8e-31 is acceptable.

I have also extended (the now quite extensive) inline docstring for the strick breakin approach in the dirichlet method. Could you double check whether this is a) the right place to document this and b) if the 'verbosity level' is ok.

In order to move this forward I'd need some guidance

in decoding the output of the failing CI, at the moment I have no clue what I need to change in order for the build to pass

about how the individual commits in this PR need to be edited for integration into master

Setting the threshold to 0.1 (so the estimate of the upper bound of the probability that the gamma variates method produces nan is 1.8e-31) seems fine, but to be honest, I don't have a rigorous or even heuristic argument for what we should consider acceptable. How about this... On my computer, dirichlet with alpha=[0.05, 0.5, 1.5] generates approximately 7 million samples per second. After running for a million years (single-threaded), I would generate only about 2.2e20 samples, so the probability that I would get all 0s in a sample is still negligible.

Don't forget to also update the unit test.

I have updated the unit tests to respect the updated threshold.

numpy/random/_generator.pyx

numpy/random/tests/test_generator_mt19937.py

…cters line length

…e mean is close to the exact mean.

numpy/random/tests/test_generator_mt19937.py

numpy/random/_generator.pyx

numpy/random/tests/test_generator_mt19937.py

…ck-breaking method over the standard approach

bashtage · 2019-11-27T11:14:02Z

0.1 or 0.2 since with this threshold there should be 0 chance of the problem given the resolution of a 53 bit float. Happy either way.

…

On Wed, Nov 27, 2019, 06:09 Warren Weckesser ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In numpy/random/_generator.pyx <#14924 (comment)>: > @@ -4150,17 +4156,32 @@ cdef class Generator: i = 0 totsize = np.PyArray_SIZE(val_arr) - with self.lock, nogil: - while i < totsize: - acc = 0.0 - for j in range(k): - val_data[i+j] = random_standard_gamma(&self._bitgen, - alpha_data[j]) - acc = acc + val_data[i + j] - invacc = 1/acc - for j in range(k): - val_data[i + j] = val_data[i + j] * invacc - i = i + k + + # Small alpha case: Use stick-breaking approach with beta random variates Nice. I was working through similar reasoning, using np.finfo(np.float).tiny as the threshold for "numerically zero", and came to roughly the same conclusion. With the revised threshold condition max(alpha) > 0.1, a conservative upper bound on the probability of generating all "tiny" gamma variates is 1.8e-31, so I think a threshold of 0.1 should be safe. For comparsion, here are some other values for max(alpha) and the associated upper bounds for the probabilty that all the gamma variates are "tiny" max(alpha) p ---------- --------- 0.05 4.3e-16 0.1 1.8e-31 0.2 3.2e-62 0.25 1.3e-77 0.5 1.7e-154 1.0 2.2e-308 @bashtage <https://github.com/bashtage>, what do you think? Does the threshold 0.1 sound reasonable? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14924?email_source=notifications&email_token=ABKTSRMXQQJOSQPS4VNFZILQVZIOFA5CNFSM4JOL7UU2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCNE2SBI#discussion_r351226725>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKTSROLWABAS6A4XM37YALQVZIOFANCNFSM4JOL7UUQ> .

WarrenWeckesser · 2019-11-30T19:18:39Z

numpy/random/_generator.pyx

+                                        alpha_csum_data[j + 1])
+                        val_data[i + j] = acc * v
+                        acc *= (1. - v)
+                    val_data[i + j + 1] = acc


This warning in the travis CI log indicates a real issue:

numpy/random/_generator.c:19904:12: warning: ‘__pyx_v_j’ may be used uninitialized in this function [-Wmaybe-uninitialized]

It corresponds to the Cython line

val_data[i + j + 1] = acc

that occurs after the j for-loop. The variable j is presumed to have been set to the final value from the for-loop. But if k is 1, then range(k - 1) is empty, the for-loop does not execute, and j is not set. The simple fix is to replace that line with

val_data[i + k - 1] = acc

Thanks for the explanation. This is fixed now.

WarrenWeckesser · 2019-11-30T19:23:00Z

numpy/random/_generator.pyx

+                    val_data[i + j + 1] = acc
+                    i = i + k
+
+        # Standard case: Unit normalisation of a vector of gamma random


I'm not sure about this style of commenting the else clause. It seems more natural to put the comment after the else, e.g.:

else: # Standard case: Unit normalisation of a vector of gamma random # variates

I realize that you also don't do this in the big block of comments before the if statement, but there you are explaining the reason behind this if-else statement, so it seems OK.

I have updated the large comment block and I have moved the small comment blocks into the if-else cases as suggested.

WarrenWeckesser · 2019-11-30T19:57:27Z

numpy/random/_generator.pyx

-                    val_data[i + j] = val_data[i + j] * invacc
-                i = i + k
+
+        # Small alpha case: Use stick-breaking approach with beta random variates


Setting the threshold to 0.1 (so the estimate of the upper bound of the probability that the gamma variates method produces nan is 1.8e-31) seems fine, but to be honest, I don't have a rigorous or even heuristic argument for what we should consider acceptable. How about this... On my computer, dirichlet with alpha=[0.05, 0.5, 1.5] generates approximately 7 million samples per second. After running for a million years (single-threaded), I would generate only about 2.2e20 samples, so the probability that I would get all 0s in a sample is still negligible.

Don't forget to also update the unit test.

WarrenWeckesser

While looking into the compiler warning, I noticed a backwards compatibility break in how alpha=[] is handled. Before this PR:

In [68]: rng.dirichlet([], size=3)                                                               
Out[68]: array([], shape=(3, 0), dtype=float64)

With this PR:

In [14]: rng.dirichlet([], size=3)                                              
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-14-661d8cc9cdb5> in <module>
----> 1 rng.dirichlet([], size=3)

_generator.pyx in numpy.random._generator.Generator.dirichlet()

~/mc37numpy/lib/python3.7/site-packages/numpy-1.18.0.dev0+ed19dea-py3.7-macosx-10.7-x86_64.egg/numpy/core/_methods.py in _amax(a, axis, out, keepdims, initial, where)
     28 def _amax(a, axis=None, out=None, keepdims=False,
     29           initial=_NoValue, where=True):
---> 30     return umr_maximum(a, axis, None, out, keepdims, initial, where)
     31 
     32 def _amin(a, axis=None, out=None, keepdims=False,

ValueError: zero-size array to reduction operation maximum which has no identity

The code needs to check for alpha having size 0 before attempting to call the max() method.

WarrenWeckesser · 2019-12-01T15:40:58Z

One more thing: a release note about the change in the stream of variates produced by dirichlet is needed. Add an entry in https://github.com/numpy/numpy/tree/master/doc/release/upcoming_changes, e.g. 14924.change.rst, that explains the change.

WarrenWeckesser · 2020-03-11T14:40:52Z

@trendelkampschroer, are you interested in continuing to work on this? I think there are just a few small things to fix up.

trendelkampschroer · 2020-03-12T21:36:10Z

Thanks for bearing with me all the time. Yes, I will try to address the remaining points in the coming days.

…rd to the stick breaking approach when generating Dirichlet RVs

…threshold of 0.1.

trendelkampschroer · 2020-03-12T22:30:48Z

The updated code checks for k=len(alpha) > 0 before calling the max method. Would you be so kind as to point me to the appropriate release note for the changes in this PR. I guess the one you linked above is outdated.

Do I need to reformat/squash commits? Do I need to rebase on or merge with an upstream branch?

WarrenWeckesser · 2020-03-13T00:38:59Z

Thanks, @trendelkampschroer. I probably won't review the latest updates until tomorrow.

Would you be so kind as to point me to the appropriate release note for the changes in this PR.

Ah, my link was a github link. From the top level of the numpy source, the location for the release notes is doc/release/upcoming_changes. The file README.rst in that directory explains the format to use for the release note. Check out some of the existing notes for examples.

Do I need to reformat/squash commits? Do I need to rebase on or merge with an upstream branch?

No need to rebase. We'll squash the commits when we merge the PR. That means we'll edit the commit message to reflect just the final change that is merged. If you want complete control over the commit message, you can rebase and squash the commits yourself, and then force push to your feature branch, but that is not necessary.

numpy/random/_generator.pyx

…or.pyx Co-Authored-By: Eric Wieser <wieser.eric@gmail.com>

WarrenWeckesser · 2020-04-22T16:30:38Z

I've committed the small bug fix I noted, and the simplification that Eric suggested. If the tests all pass, I think this is ready to be (squashed and) merged.

WarrenWeckesser · 2020-04-22T22:12:46Z

Updated with a release note.

seberg

Thanks @trendelkampschroer and Warren for the fixup, looks good to me. I think we could move the cumsum to later. @WarrenWeckesser just feel free to merge if you are happy.

numpy/random/_generator.pyx

Co-Authored-By: Sebastian Berg <sebastian@sipsolutions.net>

numpy/random/_generator.pyx

WarrenWeckesser · 2020-04-28T04:18:47Z

Azure Linux PyPy3 test failure appears to be unrelated to this PR. It looks like it failed the same way in #16083

mattip · 2020-04-28T07:57:22Z

We should pin the PyPy version we test against. We are using a nightly build rather than an official release. While this is nice for PyPy, it occasionally, like in last night's build, messes up.

mattip · 2020-04-29T20:47:11Z

Thanks @trendelkampschroer and the other contributors

btschroer added 2 commits November 17, 2019 22:42

Add stick-breaking

94ffafa

Add tests demonstrating slowness for beta and dirichlet generators fo…

6a7e3f1

…r small alpha (and beta) values

trendelkampschroer mentioned this pull request Nov 17, 2019

BUG: Add np.random.dirichlet2 #5872

Closed

charris added 00 - Bug component: numpy.random labels Nov 18, 2019

charris changed the title ~~Feature/dirichlet stick breaking~~ BUG: Fix numpy.random.dirichlet returns NaN for small 'alpha' parameters. Nov 18, 2019

WarrenWeckesser reviewed Nov 19, 2019

View reviewed changes

numpy/random/_generator.pyx Outdated Show resolved Hide resolved

Fix bug in accumulation for dirichlet stick breaking

da48d01

WarrenWeckesser requested changes Nov 20, 2019

View reviewed changes

btschroer added 2 commits November 20, 2019 21:20

Add missing white space around binarz operators. Ensure max. 80 chara…

9ff5769

…cters line length

Add test for Generator.dirichlet which ensures that the large sampl…

2434c8a

…e mean is close to the exact mean.

WarrenWeckesser reviewed Nov 21, 2019

View reviewed changes

numpy/random/tests/test_generator_mt19937.py Outdated Show resolved Hide resolved

WarrenWeckesser reviewed Nov 21, 2019

View reviewed changes

numpy/random/_generator.pyx Outdated Show resolved Hide resolved

WarrenWeckesser reviewed Nov 21, 2019

View reviewed changes

numpy/random/tests/test_generator_mt19937.py Outdated Show resolved Hide resolved

btschroer added 4 commits November 21, 2019 07:00

Align sublists in array definition.

9da470a

Restrict line length to 80 characters

bd1f8fe

Remove the test for beta with small a and b

0569b3d

Add inline documentation motivating the rationale for chosing the sti…

e098ffe

…ck-breaking method over the standard approach

Switch from standard to stick-breaking method whenever alpha.max() < 0.1

ed19dea

WarrenWeckesser reviewed Nov 30, 2019

View reviewed changes

WarrenWeckesser requested changes Nov 30, 2019

View reviewed changes

WarrenWeckesser added this to the 1.19.0 release milestone Dec 1, 2019

btschroer added 3 commits March 12, 2020 22:43

Avoid array access with possible uninitialised loop variable j.

c258f36

Improve comment describing the rationale for a switch from the standa…

e121dfd

…rd to the stick breaking approach when generating Dirichlet RVs

Update test so that stick breaking code path gets triggered with new …

01ce2c4

…threshold of 0.1.

btschroer added 2 commits March 12, 2020 23:43

Add missing parenthesis.

53dd7c6

Change & to and.

ae089e7

WarrenWeckesser self-assigned this Mar 13, 2020

WarrenWeckesser reviewed Mar 13, 2020

View reviewed changes

numpy/random/_generator.pyx Outdated Show resolved Hide resolved

eric-wieser reviewed Mar 13, 2020

View reviewed changes

numpy/random/_generator.pyx Outdated Show resolved Hide resolved

WarrenWeckesser and others added 2 commits April 22, 2020 12:26

Fix off-by-2 error in numpy/random/_generator.pyx

b4a1cd9

Remove unnecessary call of ascontiguousarray in numpy/random/_generat…

988c45c

…or.pyx Co-Authored-By: Eric Wieser <wieser.eric@gmail.com>

WarrenWeckesser approved these changes Apr 22, 2020

View reviewed changes

DOC: random: add release note about numpygh-14924.

561112c

seberg approved these changes Apr 22, 2020

View reviewed changes

numpy/random/_generator.pyx Outdated Show resolved Hide resolved

numpy/random/_generator.pyx Show resolved Hide resolved

WarrenWeckesser and others added 2 commits April 22, 2020 21:14

Move cumulative sum to later (part 1)

ea6cf00

Co-Authored-By: Sebastian Berg <sebastian@sipsolutions.net>

Move cumulative sum to later (part 2)

803605a

Co-Authored-By: Sebastian Berg <sebastian@sipsolutions.net>

WarrenWeckesser reviewed Apr 23, 2020

View reviewed changes

numpy/random/_generator.pyx Outdated Show resolved Hide resolved

WarrenWeckesser reviewed Apr 28, 2020

View reviewed changes

numpy/random/_generator.pyx Show resolved Hide resolved

WarrenWeckesser added 2 commits April 27, 2020 23:34

declare csum for upcoming commit

f52e491

Eliminate some redundant array creation

6b181c8

WarrenWeckesser mentioned this pull request Apr 28, 2020

CI: PyPy3 tests on Azure failed for a couple recent PRs #16092

Closed

mattip merged commit 44e26f2 into numpy:master Apr 29, 2020

rkern mentioned this pull request Aug 10, 2021

np.random.dirichlet returns np.nan for small alpha #19637

Closed

WarrenWeckesser mentioned this pull request Jan 2, 2022

rand_distr: Fix dirichlet sample method for small alpha. rust-random/rand#1209

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Fix numpy.random.dirichlet returns NaN for small 'alpha' parameters. #14924

BUG: Fix numpy.random.dirichlet returns NaN for small 'alpha' parameters. #14924

trendelkampschroer commented Nov 17, 2019 •

edited by charris

Loading

WarrenWeckesser commented Nov 19, 2019

WarrenWeckesser left a comment

WarrenWeckesser Nov 20, 2019

trendelkampschroer Nov 20, 2019

trendelkampschroer Nov 20, 2019

WarrenWeckesser Nov 21, 2019

trendelkampschroer Nov 21, 2019

trendelkampschroer Nov 26, 2019 •

edited

Loading

WarrenWeckesser Nov 27, 2019

trendelkampschroer Nov 30, 2019 •

edited

Loading

WarrenWeckesser Nov 30, 2019

trendelkampschroer Mar 12, 2020

bashtage commented Nov 27, 2019 via email

WarrenWeckesser Nov 30, 2019 •

edited

Loading

trendelkampschroer Mar 12, 2020

WarrenWeckesser Nov 30, 2019

trendelkampschroer Mar 12, 2020

WarrenWeckesser Nov 30, 2019

WarrenWeckesser left a comment •

edited

Loading

WarrenWeckesser commented Dec 1, 2019

WarrenWeckesser commented Mar 11, 2020

trendelkampschroer commented Mar 12, 2020 •

edited

Loading

trendelkampschroer commented Mar 12, 2020 •

edited

Loading

WarrenWeckesser commented Mar 13, 2020

WarrenWeckesser commented Apr 22, 2020

WarrenWeckesser commented Apr 22, 2020

seberg left a comment

WarrenWeckesser commented Apr 28, 2020

mattip commented Apr 28, 2020

mattip commented Apr 29, 2020

BUG: Fix numpy.random.dirichlet returns NaN for small 'alpha' parameters. #14924

BUG: Fix numpy.random.dirichlet returns NaN for small 'alpha' parameters. #14924

Conversation

trendelkampschroer commented Nov 17, 2019 • edited by charris Loading

WarrenWeckesser commented Nov 19, 2019

WarrenWeckesser left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trendelkampschroer Nov 26, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trendelkampschroer Nov 30, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bashtage commented Nov 27, 2019 via email

WarrenWeckesser Nov 30, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WarrenWeckesser left a comment • edited Loading

Choose a reason for hiding this comment

WarrenWeckesser commented Dec 1, 2019

WarrenWeckesser commented Mar 11, 2020

trendelkampschroer commented Mar 12, 2020 • edited Loading

trendelkampschroer commented Mar 12, 2020 • edited Loading

WarrenWeckesser commented Mar 13, 2020

WarrenWeckesser commented Apr 22, 2020

WarrenWeckesser commented Apr 22, 2020

seberg left a comment

Choose a reason for hiding this comment

WarrenWeckesser commented Apr 28, 2020

mattip commented Apr 28, 2020

mattip commented Apr 29, 2020

trendelkampschroer commented Nov 17, 2019 •

edited by charris

Loading

trendelkampschroer Nov 26, 2019 •

edited

Loading

trendelkampschroer Nov 30, 2019 •

edited

Loading

WarrenWeckesser Nov 30, 2019 •

edited

Loading

WarrenWeckesser left a comment •

edited

Loading

trendelkampschroer commented Mar 12, 2020 •

edited

Loading

trendelkampschroer commented Mar 12, 2020 •

edited

Loading