You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
is the test suite something that could be factored out/generalized? #40
You've done some incredible stuff in there making it possible to generate dumps, use dump_syms, and then load dumps again. I've been wanting a tool that is two binaries, minidump-harness and futility.
futility
(it seems like src/bin/test.rs is basically this)
The futile utility application, providing a "crash suite"
takes a simple cli arg choosing which func to call that will die in an interesting way
we can build for whatever platform
run dump_syms on
minidump-harness
(it seems like tests/ is kinda this)
Orchestrates building/analyzing/torturing futility and storing/diffing the results. it would serve three roles:
regression tests (running in ci)
getting a particular intermediate result of the pipeline for inspection
enabling us to quickly test/diff a change across our entire pipeline
The first is straight-forward and you mostly have that setup (although #30 still needs to be done). The second is more subtle. You can see my initial attempts at this problem with my socc-pair utility which largely automates diffing/benching a local version of minidump-stackwalk against what mozilla has in production. This is great for testing if I fixed a bug in production, but it's still more manual than I'd like.
The Dream
$ minidump-harness run base-trial
<spews a bunch of shit into a dir called base-trial>
<now i make some changes like [patch]ing local changes to rust-minidump/symbolic/dump_syms...>
$ minidump-harness run patched-trial
<spews a bunch of shit into a dir called patched-trial>
$ minidump-harness diff base-trial patched-trial
<spits out a report of what changed>
The Design
minidump-harness run $DIR would:
build futility, putting the target in $DIR/target
run dump-syms on it, putting the symbols in $DIR/symbols
run every crash suite it has, putting the dumps in $DIR/dumps/$suite.dmp
run minidump-stackwalk --cyborg --verbose=trace --symbols-path=$DIR/symbols on each minidump, putting:
json in $DIR/results/$suite.json
human in $DIR/results/$suite.txt
logs in $DIR/results/$suite.log.txt
minidump-harness diff $DIR1 $DIR2 would:
spit out a diff of particular details (default would probably be what socc-pair does?)
but ideally would streamline other diff/filter queries like "did the human output change" or "just tell me about this suite"
probably also want to be able to cargo-insta (snapshot) some of these results?
Right now you have some of these things, but a lot of the test implementations are in the harness and kind of manually adhoced. I don't fully understand your design, and all the constraints there are. I'm happy to do a lot of legwork here, but you've clearly done a lot stuff that's worth using/extending instead of me kicking off a green-field project.
I think something like this would be great. The MacOS dump_syms stuff I added was loosely based on the end-to-end testing I did in https://github.com/EmbarkStudios/crash-handling/tree/main/minidumper-test though there I at the moment just doing the basic "does this minidump contain the expected crash reason" without checking the symbolicated stack traces for the thread(s) also match, but that was something I wanted to add eventually.
I guess the one big difference to me is that the dream scenario you specified above feels a bit too manual for me, but maybe this is just a different workflow for doing changes before making automated tests?
yeah i made the dream slightly manual just so we could do absolutely whatever we want and extract only the parts we care about for individual runs.
i.e. "this old dump_syms version had this problem, I want a sym file showing that for my tests, so I will [patch] symbolic back to that version and do minidump-harness run and grab the sym file it spits out for my test-suite"
making automated tests I think would look closer to cargo insta snapshots, where we have a copy of the outputs we care about saved in one dir, and the tests just minidump-harness run and diff against the snapshot.
Ok here's a rough diagram of what a full deployment architecture looks like.
I've broken the architecture into 3 major sections:
your build
your client (user)
your server (you)
The client and server are further broken up into separate processes.
Rough atlas:
solid arrows: code invoking
dashed arrows: data flow
diamond boxes: data (files/structs)
square boxes: specific rust-minidump/crash-handling crates
round boxes: abstract services that we don't provide but you "need" to glue things together. I have labeled some of them with mozilla-specific names because those are what I know, but a lot of these could be labeled "Sentry". Many of these are the things a test harness would need to fill in.
Note that Mozilla (and Sentry?) don't use any of the "client" stuff right now except for minidump-writer in some specific situations. Migrating to minidump-writer completely is desirable for Mozilla (kill breakpad), but the other stuff is less clearly important to us.
Activity
Gankra commentedon Jul 7, 2022
cc @mstange and @Jake-Shadle
Jake-Shadle commentedon Jul 7, 2022
I think something like this would be great. The MacOS dump_syms stuff I added was loosely based on the end-to-end testing I did in https://github.com/EmbarkStudios/crash-handling/tree/main/minidumper-test though there I at the moment just doing the basic "does this minidump contain the expected crash reason" without checking the symbolicated stack traces for the thread(s) also match, but that was something I wanted to add eventually.
I guess the one big difference to me is that the dream scenario you specified above feels a bit too manual for me, but maybe this is just a different workflow for doing changes before making automated tests?
Gankra commentedon Jul 7, 2022
yeah i made the dream slightly manual just so we could do absolutely whatever we want and extract only the parts we care about for individual runs.
i.e. "this old dump_syms version had this problem, I want a sym file showing that for my tests, so I will
[patch]symbolic back to that version and dominidump-harness runand grab thesymfile it spits out for my test-suite"making automated tests I think would look closer to
cargo instasnapshots, where we have a copy of the outputs we care about saved in one dir, and the tests justminidump-harness runand diff against the snapshot.Gankra commentedon Jul 7, 2022
basically it would be useful as a
Gankra commentedon Jul 14, 2022
Ok here's a rough diagram of what a full deployment architecture looks like.
I've broken the architecture into 3 major sections:
The client and server are further broken up into separate processes.
Rough atlas:
Note that Mozilla (and Sentry?) don't use any of the "client" stuff right now except for minidump-writer in some specific situations. Migrating to minidump-writer completely is desirable for Mozilla (kill breakpad), but the other stuff is less clearly important to us.
graph TD; subgraph build cargo dump_syms http-symbol-sender(http-symbol-sender) binary{.exe/.pdb} sym{.sym} end subgraph client subgraph app YOUR_ACTUAL_APP(YOUR ACTUAL APP) minidumper::Client crash-handler end subgraph crash-monitor minidumper::Service minidump-writer http-dump-sender(http-dump-sender) context{CrashContext} dump{.dump} end end subgraph server subgraph crash-server http-dump-receiver(http-dump-receiver) minidump-processor minidump structMinidump{Minidump} report{report.json} end subgraph symbol-server tecken(tecken) http-symbol-receiver(http-symbol-receiver) end subgraph crash-stats socorro(socorro) end end cargo -.-> binary binary -.-> dump_syms binary -.-> deploy(deploy) dump_syms -.-> sym sym -.-> http-symbol-sender crash-handler ---> minidumper::Client minidumper::Client ---> minidumper::Service minidumper::Service ---> minidump-writer minidump-writer -.-> dump dump -.-> http-dump-sender minidumper::Service ---> http-dump-sender minidumper::Service -.-> context context -.-> minidump-writer http-dump-receiver ---> minidump-processor http-dump-receiver ---> minidump minidump -.-> structMinidump structMinidump -.-> minidump-processor tecken -. .sym .-> minidump-processor minidump-processor -.-> report report -.-> http-dump-receiver http-dump-receiver -. report.json .-> socorro http-symbol-sender -. .sym .-> http-symbol-receiver http-dump-sender -. .dump .-> http-dump-receivergraph source