Fuzz testing in Elm
I will assume that you are familiar with the general concept of fuzz testing aka generative testing aka property based testing. If you are not familiar with it yet, you can get a nice intro from the elm-test readme or in case you have access to frontend-masters please take a look at the fuzz testing part of Richard’s Course. It is short and simple and is enough to get you started.
Fuzz testing all the time
Now that the background should be clear. There is only one thing I want to mention in this post: Use fuzz testing, ideally in every project! Even if you think it is not worth to add it to a certain project. I don’t mean use it for everything but prefer it over unit testing as the goto tool and you will quickly be surprised by its value.
My experience in Elm
I added tests and fuzz testing to Carna just to get used to it. Then surprisingly my tests failed for an unexpected input. You can find the actual test that failed here. My tests are really rudimentary but they still found multiple bugs that where quiet interesting.
Unexpected String.toInt behavior
In Carna I am asking for numeric values like weight or height and parse them to Int.
(Now they are Float, but that is not important here)
Thus the value is represented as a
Result String Int. That is also what is tested
at the line above. Then my tests failed for the input “+”. The reason:
> String.toInt "+" Ok NaN : Result.Result String Int
I think I would have never thought about using “+” or “-“ as a input value that
would cause an
Ok result. Especially I would not have thought that the state
Ok NaN was even possible.
What I was expected is something that happens when you use
> String.toFloat "+" Err "could not convert string '+' to a Float" : Result.Result String Float
This bug was not severe because
toString NaN just resulted in my input fields
having the content “NaN” if you enter “+” which is not is better than crashing
but not what I wanted.
I asked about the behavior in the elm slack and got quick and helpful feedback there as well as on twitter. So thanks to the great, friendly and helpful community. ❤️❤️❤️
This was only the first issue in a list of bugs my fuzz tests found. Afterwards I switch to regex matches instead of relying on toInt never creating invalid Ok states, and I forget to allow decimal points for floats, because I used the same pattern for Int and Float validation, but my fuzz tests found that quickly. And there were more, but I think my point is clear.
Combine with test watch mode
I realized that elm-test watch mode is great to have a higher number of generated
tests cases. By default 100 inputs are generated per fuzz-invocation.
You can change that value with the
--fuzz option of elm-test. However this can
quickly become annoying if you are executing your tests manually and wait for the
result. However in CI many people already use higher values. But you can also
increase the number locally when combining it with the recently added watch mode
that you can start with
--watch. Because the tests start immediately after saving
the file so they already run when you switch to the console (if you do that at all).
I use this line here for local development:
elm-test --watch --fuzz 10000
Since the numbers the fuzzers are biased towards values that are typical error cases higher values helped my to find more edge cases quicker. For more details take a look at elm-test Fuzz.frequency.
I guess if you have way more fuzz tests you might go down to 2000 or 1000 but this is still much more than 100 and it did not annoy me at all yet.
I really think everybody should consider adding fuzz testing to their Elm projects.
Even in my case of a private side project it was very valuable because there are
not only edge cases that you are not aware of, but you might also find bugs
in libraries or tools you use. This was for example the case for riak which had
a high priority bug discovered via generative testing.
In addition I also recommend this article from basho
for further reading and do not forget to increase your
--fuzz value in case
you use watch mode and on CI.