Original Presentation: https://fuzz-prettier-go-template.glitch.me/, Source: https://glitch.com/edit/#!/fuzz-prettier-go-template See: https://blog.glitch.com/post/changes-are-coming-to-glitch/, https://blog.keith.is/blog/the-end-of-glitch-even-though-they-say-it-isnt/, https://livelaugh.blog/posts/on-important-changes-coming-to-glitch/.
Original Presentation: https://jasikpark.github.io/fuzz-prettier-go-template/, Source: https://github.com/jasikpark/fuzz-prettier-go-template.
Current implementation at: https://github.com/jasikpark/fuzz-prettier.
This is a writing down of a talk I gave, trying to serialize this into a blog post.
I wanted to use NiklasPor/prettier-plugin-go-template at work, but it was having quite a few issues running against our templates.
My original repo is at https://github.com/jasikpark/fuzz-prettier-plugin-go-template
My good friend was telling me about this new testing technique he was researching and exploring called āfuzzingā where you squint and pretend that throwing lots of different inputs at a program is equivalent to proving there are no bugs. Similar to how machine learning will replace all programs as we know today š
I wanted to try my hand at writing a fuzzer but I had little reason to fuzz a rust program, though I love my little collection of RIIR cli tools, I turned my eye towards the prettier code formatter. Precisely, the golang html/template plugin.
Someone took that ālanguageā and wrote an AST parser for it, then wrapped a prettier printer around that output. It worked great! Except when it didnāt. And it was hard to reduce what the actual cause of the bug was, to find the root cause.
I ended up exploring tools like jsfuzz bought by Gitlab, but ended up abandoning them because I couldnāt get around OOM presumably from prettier have too many globals hang around.
I used radamsa, a cli fuzzer to produce files for prettier to format, then format again. I eventually ended up with a script that found a bug if either the output of the first and second format was different, or if the first format succeeded and the second format errored.
the problem was that the generated crash cases were often enormous in size
I researched a method to reduce these failure cases and ended up finding delta (8) which is a very dumb āgive me a script that tests for the behavior and iāll binary search the lines of code to find the bugā
it works great!
this also was helped by the existing thorough unit test fixtures that I could use to feed radamsa as seeds, so that it could alter the seeds for its purposes, like a GMO corn seed š
I found about 7 different idempotency bugs that imo greatly improved the reliability of the formatter, especially since i can now continually run the fuzzer without immediately running into a crash.
the important thing is that fuzzers donāt prove the non-existence of bugs, they just sniff out the ones they can find. but fuzzing uses computers for what theyāre solid at: churning through god-forseen numbers of data and producing value that would otherwise be discovered by the people that use them. build a funny lil fuzzer! test the tools you use day-to-day
golang and rust have great language support for fuzzing as a first class testing setup, and you can get ok dx with fast-check for Typescript. Itās technically property testing, which uses the same principles but with less time running, so theyāll slowly maybe explore the space.
having good invariants you test for is important too to make the crashes recognizable and useful.
thx for listening lol
interests:
- fuzzing
- house š” music
- css/web stuff
- bringing fuzzing to JS land