← all cases C1 · 2023-11-30 C2 · 2024-01-21 C3 · 2024-12-16 synthesis

Synthesis Three cases, one fair baseline — the cross-case reading

CONVECT / Project Sharktopus · draft for WRF User Meeting 2026

Revision 2026-07 (blockage-mask lineage). 12–40% of the radar obs assimilated in every cycle sat inside the radar's blind sectors; all four experiments were re-run with the beam-blockage mask active (erratum E9). Numbers below are control → radar masked, with the unmasked radar value in parentheses where it changed materially.

Cycled radar 3D-Var robustly improves the organized overnight system (C2-morning) — a result that survives beam-blockage QC. In afternoon/evening regimes the DA over-forecasts (the mask cuts that by ~⅓). The apparent 1-km event recovery in C3 did NOT survive the mask: it was carried by echo assimilated inside blind sectors.

Three cases spanning two calendar years and three convective regimes, all run with a fair same-lineage control (no cold-start mixing with the DA signal — lesson AU3-5 from the C2 audit). Comparisons within each case (control vs radar) are apples-to-apples; cross-case comparisons should account for the config differences listed in the caveats.

C3 d03 POD — control misses, radar recovers
C3 d03, unmasked lineage: DA appeared to recover the missed nocturnal event (POD 0→0.385 at ≥1 mm/h). With the blockage mask the recovery collapses to 0.08 — the signal came from blind-sector echo (see revision note).

The four experiment windows

ExperimentDateDA cycles Event windowd03 boxClimo <z>
C2 · morning2024-01-2100–03 Z (4) 03–12 Z · nocturnal floodover radar +0.40σ
C2 · afternoon2024-01-2106–09 Z (4) scattered afternoonover radar +1.00σ
C12023-11-3012–15 Z (4) 16–21 Z · weak afternoon (max ~0.7 mm/h) NE radar — 10 stations +0.32σ (DJF climo / Nov 30)
C32024-12-1614–17 Z (4) 18–20 Z · nocturnal line, 64 dBZ, Arraial over radar — 16 stations +0.64σ

Key numbers (control → radar)

Metric C2 · morning C2 · afternoon C1 C3
d03 POD ≥1.0 —— 0→0.25 (0→0 unmasked) 0.00→0.08 (0.385 unmasked — did not survive)
d03 POD/CSI ≥2.5 0.04→0.24 / 0.04→0.20 (0.26/0.20 unmasked) 0.13→0.06 / 0.08→0.05 0→0.33 / 0→0.25 (0→0 unmasked) 0.00→0.10 / 0→0.04 (1 hit)
d02 POD/CSI ≥2.5 (gain — see C2 site) 0.11→0.17 / 0.05→0.04 0.25→0.30 / 0.05→0.05 (0.16 unmasked) 0.10→0.01 (0.06 unmasked)
d02 BIAS ≥2.5 →~1 (corrects under-forecast) rises (over-forecast) 4.4→5.8 (4.9 unmasked) 2.52→0.97 (via POD collapse — not a gain)
21-h accum MAE d02 improves 11.3→25.8 mm (36.4 unmasked: mask cuts over-forecast ~⅓) 20.5→19.9 mm (14.6 unmasked; gain gone) 9.8→8.5 mm (10.4 unmasked; now improves)
21-h accum MAE d03 improves strongly worsens 6.6→6.1 mm 8.1→22.0 mm (corr →−0.06)
MODE centroid d02 −17% (−45% at peak)†— ~neutral 92.7→113.3 km (worsens)
MODE centroid d03 23.5→29.9 km (19.4 unmasked — gain did not survive)— 51.6→47.1 km (10.6 unmasked; 1 h — fragile) ~neutral

Reading the results honestly

  1. The clear, robust gain is C2-morning (organised nocturnal system, 00–03 Z). It is the only case where POD/CSI/BIAS/MODE/accumulation all improve together with a decent sample size.
  2. Beam blockage QC matters as much as the DA (new, 2026-07): masking the blind sectors (54–71°, 254–262°, 279–339°) removed 13–42% of the reports per volume. The C2-morning gain survives, C1 flips to a clear station gain (POD ≥2.5: 0→0.33, FAR 1.00→0.50 — blind-sector clutter was poisoning its analysis), the afternoon over-forecast shrinks by ~⅓, and the C3 "recovery" collapses. Fine-scale MODE centroid gains do not survive anywhere.
  3. Consistent signature of afternoon/evening windows (3 windows, 2 dates): DA injects moisture/instability → over-forecast of light-to-moderate rain; hourly point scores neutral-to-worse. (The displacement-tolerant examples previously cited here — C1 accum −29%, C1 centroid 51.6→10.6 km, C3 corr 0.17→0.33 — did not survive the blockage mask and are withdrawn.)
  4. C3 d03 event "recovery" is withdrawn: the control still misses the event entirely, but the radar POD 0→0.385 at ≥1 mm/h collapses to 0→0.08 with the mask. C3 had the highest blind-sector fraction of assimilated obs (31–40% per cycle) and the event sat in the W→NW terrain sector. Part of that echo may be REAL (clean short range before the terrain; attenuated returns through partial blockage) — a range-aware mask (azimuth×range, from macae_cbb.nc) is the proposed path to re-evaluate C3 defensibly.
  5. Information age alone does not explain the pattern: C3 had lead times of 1–3 h and still only gained on d03/event-recovery. The convective regime appears decisive: organised nocturnal convection (C2) responds robustly; scattered afternoon convection over-forecasts.
  6. Design lesson (C1): the d03 box fell in a gauge-poor region (10 stations, event maximum displaced). Station verification was empty there. In weak cases the picker should consider observation density. MODE (vs radar) partially covered the gap but with only 1 matched hour.

Caveats on comparability

Per-case pages

C2 nest decision overview
C2 · 2024-01-21

The audited flagship case

2 windows (morning + afternoon), full placement/innovation/verification site, audit-revised. Organised nocturnal flooding event.

Open case 2
C1 nest decision
C1 · 2023-11-30

Afternoon weak convection

Gauge-sparse d03 box. With blockage QC the radar flips to a clear station gain (POD ≥2.5: 0→0.33). Design lesson on observation density.

Open case 1
C3 POD d03
C3 · 2024-12-16

The recovery that did not survive blockage QC

Unmasked, DA seemed to recover the missed event (POD 0→0.385 at ≥1 mm/h); with the blockage mask it collapses to 0.08. Candidate for re-evaluation with a range-aware mask. Arraial do Cabo, 64 dBZ.

Open case 3