What this is
Instead of relying on generated test reports from Jenkins which are flaky and unintuitive
to work through for runs with lots of tests and/or many failures, this tool takes the console
output from a test run and surfaces all the information you should need to work through your
tests, debugging them, fixing them, and rerunning them.
Your tests
All your tests are here. You can filter them by status, search through them and push them away
by ignoring them. Group them together based on similar failure reasons, the test class they
come from, how they compare against a secondary run (like a nightly run), and more.
Debug a failing test
Want to debug a failing test? Click a test ID and see everything about why it failed. View
its failure reason and any associated stacktrace/traceback, view multiple failure reasons
if the test was flaky, and the Saucelabs link (for compatible tests) is just a click away.
Sort tests with custom statuses
Fixed a test in your local environment and want to move it to the side? Make a custom status
and update the status of your test to move it out of the "Failed" pile and into its own
area, where you can still revisit it. Knock down your failed tests pile, and make more custom
statuses for tests that failed beyond your control, such as environmental failures.
Compare with a baseline run
Not sure what's just failing for you and what's failing for everyone else? Click
"Load baseline run..." at the bottom of the summary. Import the console output of a secondary run you started, or a nightly run
to see how your tests compare against it.
Rerun your tests
Ready to test your fixes? In the same place, click "Run tests...". The Jenkins job side
pre-fills all of your previous Jenkins job arguments with whatever you used for this test
run. Select which tests you want to run, or pick "Rerun this run's tests" to start the same
run again. The Local ./py-tester side builds a py-tester command for the same tests to run on
your own machine. "Run this test..." in a test's "More options" menu opens it for that one test.
When the rerun is done, click "Import rerun results..." and give it the rerun's console log or
py-tester output. You see what changed before anything does, and choose what to apply.
Save and share
Or ready to take a break? You can save your work as a breakdown file (.mtrb),
and reopen it later from within the tool to start right back where you left off. All your
work is saved here, your custom statuses, tags, any notes you've made, including the original
console output you fed the tool for your main test run and a baseline run if you imported it,
so you'll never lose it. You can even share these files with others - everything they need is
within one .mtrb file, and they'll see exactly what you see.
If you have any questions or suggestions, reach out to Tom Wilkinson.