The first profile on a production mesh is allowed to be embarrassing. The unforgivable part is rewriting the flux kernel before that profile exists.
A short true-to-life sequence#
We had a finite-volume code that "did not scale past 16 nodes". The MPI traces looked busy. Halo exchanges were the obvious villain.
We timed three runs on the same 8 nodes:
| Variant | Walltime |
|---|---|
| Production, full VTK every step | 41 min |
| VTK every 50 steps | 18 min |
| No VTK, same compute | 11 min |
The halo exchanges were fine. The parallel filesystem was taking a metadata beating from per-rank VTK. The "MPI problem" was I/O.
What we changed#
Not the Riemann solver. A single writer rank, larger intervals, and a binary format. Scaling to 64 nodes started to look like a solver again, at which point a real MPI issue (an unnecessary
Allreduce in a residual check) was visible.Rules we kept#
- One change per experiment
- The same mesh and the same node type
- A timer around output, exchange, and update, not only the whole executable
- No optimisation sprint during a conference deadline