How does SWIM compare with Australia's established bushfire simulators?
ATS reconstructed nine historical Australian bushfires using the evaluation approach developed for Phoenix RapidFire, Spark, Australis and Prometheus. The results provide encouraging evidence of spatial prediction skill, and an important foundation for evaluating probabilistic risk and consequences.
Australia already has sophisticated fire prediction tools. The real question is what additional evidence SWIM brings.
Australian fire agencies have invested heavily in wildfire spread simulation. Established tools such as Phoenix RapidFire and Spark help Fire Behaviour Analysts understand fire propagation under complex combinations of weather, fuels and terrain. Their outputs support planning and operational decisions across jurisdictions.
Any new model should face the same scrutiny as the models already in service. Claims based on attractive maps or isolated success stories are not sufficient. It must be possible to test the predicted footprint against observed behaviour, compare it with a meaningful baseline, and discuss the circumstances in which the model might fail.
This motivated ATS to evaluate its Stochastic Wildfire Impact Model (SWIM) against a prominent Australian historical benchmark. We wanted to answer two connected questions: Is SWIM's spatial prediction informative, and can that prediction be translated into better estimates of likelihood, timing and consequence?
Built on an established Australian fire-simulator evaluation framework
The starting point was a landmark Bureau of Meteorology study, commissioned within an Australian fire-agency collaboration, which compared the fire progression predicted by several simulators across ten historical Australian fires. Its peer-reviewed analysis was published by Fox-Hughes and colleagues in 2024; the full technical evaluation report was issued in 2017.
Phoenix RapidFire
Versions 4.06, 4.07, 4.08 and 5.00. A widely used simulator built around Australian fire behaviour formulations and a perimeter-propagation framework.
4 historical versionsSpark
Basic, Vesta and McArthur workflows within the then-tested Spark framework. The workflow choice changes the fire-behaviour formulation.
3 historical workflowsAustralis & Prometheus
Australis 1.5.6 was included; Prometheus 6.1.0.7 was evaluated for a single Tasmanian case, not across the full ten.
Coverage varies by modelThe original evaluation used agency-selected historical fires, mapped progression observations, gridded weather and model-specific inputs, and investigated sensitivity to input uncertainty. The researchers concluded that no simulator clearly dominated across cases. Read Fox-Hughes et al. (2024).
The benchmark's ten historical fires
Events and jurisdictions from the 2024 paper's Table 2. These are the original study cases, not a newly assembled or cherry-picked ATS collection.
† ATS excluded Wye River because it did not identify a suitable satellite overpass after the original evaluation interval. Thus, nine of the ten cases were used for the ATS numerical comparison.
How ATS reconstructed the historical simulations
ATS followed the original evaluation's case selection and threat-score concept, while reconstructing fire progression using available VIIRS satellite observations. The steps below explain the analytical chain and where it differs from the original work.
Use the original cases
Retain the ten Australian benchmark fires and their published simulation windows. Exclude Wye River from numerical scoring because an appropriate post-window satellite observation was unavailable.
Bracket with satellite passes
For each remaining event, identify the nearest VIIRS fire-detection observations on either side of the study window, then establish a reconstructable before-and-after interval.
Re-run SWIM
Simulate progression using SWIM's spatially resolved stochastic spread model, including fuel, terrain and weather effects. The internal analysis included high-percentile footprint visualisations.
Score spatial overlap
Compare the reconstructed SWIM footprint with satellite-derived observed fire detections using intersection over union, then relate its scores to reported historical model results.
What is a threat score?
The threat score, also called the Jaccard coefficient or intersection over union, measures how closely two geographical footprints overlap. It rewards correct agreement and penalises both areas predicted to burn that were not observed and areas observed to burn that the model missed.
1.0 means perfect geometric agreement; 0.0 means no overlap. It is useful because an excessively broad prediction cannot obtain a perfect score merely by enclosing the true fire, and a narrow prediction is penalised for missed area.
Threat score is a spatial diagnostic, not the percentage of fires correctly forecast and not a probability of a structure burning. It also does not by itself assess arrival time, probability calibration, false-alarm costs, or warning quality.
Why shape matters
Where a fire travels, not just how much area it burns, determines which roads, communities and assets face danger. An area-matched circle is a useful check against a model that gets the total area approximately right but the direction and footprint wrong.
SWIM showed encouraging spatial agreement on the historical fire set
ATS's internal summary found an average threat score of 0.38 for SWIM's nine reconstructed events, compared with an approximately 0.28 population median across the historical simulator evaluation and a 0.29 mean for a simple ignition-centred, area-matched circle.
Threat score comparison 0 = no overlap / 1 = perfect overlap
In seven of the nine reconstructed events, ATS reports that SWIM's threat score exceeded the median score of the simulator configurations tested for that event in the earlier evaluation.
For five of the nine cases, ATS reports a SWIM threat score above the leading simulator configuration from the historical comparison for that case. This is a cross-study comparison, not evidence of superiority over modern releases.
Why include an area-matched circle?
Consider an elementary model that draws a circle around the ignition, chosen to cover the same area as the observed event. Even without understanding fuels, slopes or wind, it will overlap a portion of the real fire. If a sophisticated spread model cannot substantially improve upon that circle, the added complexity may not be producing useful spatial information.
In ATS's comparison, the circle averaged 0.29 and SWIM averaged 0.38. This is a useful diagnostic: SWIM's predicted geometry appears to carry additional signal beyond an ignition-centred area assumption.
Why is the 36% result not an accuracy claim?
The relative uplift comes from (0.38 − 0.28) ÷ 0.28 ≈ 36%. It is an increase in the threat-score statistic versus the earlier benchmark figure; it does not mean that SWIM predicts 36% more fires correctly, nor that its real-world operational accuracy is 36% higher than Spark or Phoenix.
The distinction matters because the numerical comparison uses a SWIM mean and an older-simulator median calculated against different observational sources.
Case-level reproducibility. The public version of this study should ultimately include a downloadable nine-row appendix containing each event's SWIM threat score, circle score, historic median and best configuration, exact start/end observations, footprint selection rule, and calculation code or method. Those case-level values were not provided in the correspondence used to prepare this mockup and have therefore not been invented.
What this comparison establishes, and what it does not
The credibility of a validation study depends as much on describing its mismatches as on reporting its results. Several limitations are central to interpreting ATS's findings.
Different observed-fire references
The original Australian evaluation used mapped fire perimeters supplied by fire agencies. ATS's reconstruction used VIIRS satellite fire observations with nominal detection scales of hundreds of metres. Active-fire detections do not reproduce every burned cell or the exact final fire boundary, especially where clouds, timing, small patches or canopy obscure heat.
Approximate temporal alignment
Satellite overpasses are discrete, while the original study employed specifically defined fire-progression intervals. Bracketing a simulation window with the nearest available observations can shift the represented fire area, even if the model physics were unchanged.
High-resolution output versus coarse observations
SWIM's roughly 20-metre spatial grid can preserve fine-scale unburned features. Coarser satellite observations cannot always validate such detail. This may penalise SWIM, but it can also produce other biases. The net direction and size of the effect must be quantified rather than assumed.
Not a current-generation comparison
Historical Phoenix and Spark configurations tested in 2017 are not synonymous with their 2026 capabilities. In particular, newer Spark Operational releases support ensemble forecasting and advanced scenarios. ATS has not established that SWIM outperforms Spark Operational 1.4 or the latest Phoenix implementation.
Limited number of cases
Nine reconstructed fires span diverse Australian conditions but are too few to establish robust performance over every fuel type, weather regime, region, suppression setting or rare extreme. The original authors themselves cautioned against sweeping conclusions from ten cases.
Different validation questions
Spatial overlap does not by itself test arrival-time accuracy, uncertainty calibration, burn severity, spotting performance, false-alarm burden, warning decisions or estimated losses. Those require additional ground truth and dedicated evaluation designs.
SWIM turns the spread forecast into a probability-and-consequence analysis
Spatial prediction is one component of the operational problem. ATS developed SWIM to analyse multiple plausible fire developments and connect them with people, assets, geography and economic consequences, within a broader risk-management workflow.
What might happen to the fire?
For specified weather, fuel and ignition conditions, a simulation depicts plausible movement, intensity or timing. The historically benchmarked configurations primarily tested this geometric spread component.
- Predicted burned footprint and evolving fronts
- Rate and direction of travel
- Potential arrival at a location
What are the potential consequences, and how uncertain are they?
SWIM's stochastic spread outputs are intended to support cell-level likelihoods, percentile scenarios and consequence distributions that help compare operational and mitigation choices.
- Probability of impact and time-to-target
- Exposure of structures, communities and infrastructure
- Tail outcomes and monetised risk
- Comparison of risk-reduction interventions
Stochastic propagation
At approximately 20-metre hexagonal resolution, SWIM represents many plausible pathways of fire progression using local conditions including fuels, slope, weather and dryness. Rather than treating a single predicted perimeter as inevitable, the output can be summarised as percentiles and location-specific impact possibilities.
Consequence layers
Predicted hazard can be overlaid on building footprints, population, transportation links and infrastructure. Exposure estimates can then be combined with consequence and vulnerability assumptions to derive loss scenarios. These are model estimates requiring region-specific ground truth, especially for casualties, asset damage and monetary impacts.
Action-oriented analysis
Users can explore points or polygons, evolving active fire perimeters, time-to-point or area, fire-break scenarios and evolving weather conditions. ATS also offers 2D and 3D visualisations to help specialists and decision-makers interpret forecasts; visual sophistication should not be confused with independently measured predictive skill.
Risk and uncertainty
For planning and risk reduction, a relevant result may be the range of consequences under alternative weather assumptions, treatments or ignition points, not merely the central fire footprint. The system can support P50, P90, P95 and other scenario summaries, provided the percentile definitions and calibration are disclosed for each output.
Where a validated probabilistic model can add decision value
The benchmark evaluates a core modelling component. The decision value comes from showing how reliable, appropriately calibrated outputs can be used in the situations fire agencies face.
Prioritise exposed communities
Identify settlements and corridors where multiple plausible fire progressions intersect areas of high population or structural exposure, and plan treatments or preparedness measures accordingly.
Estimate time to consequence
Explore the range of times at which fire could reach a road, communications site or community. Any alert or evacuation use should be reviewed and authorised by the responsible agency.
Examine high-consequence tails
Use plausible adverse scenarios alongside central scenarios when considering staging, access, asset protection and limited firefighting resources.
Compare interventions
Evaluate how fuel treatments, fire breaks, asset hardening, defensible space or other measures change expected loss, conditional on sound assumptions.
Quantify asset exposure
Translate geographic fire pathways into potential impacts on electricity, transport, water, telecommunications and strategically important facilities.
Support independent cross-checks
Use a separate modelling approach to challenge a single forecast and identify where different assumptions or underlying data produce materially different risk estimates.
The next standard: a common-input, current-generation evaluation
ATS views this historical benchmark as an informative first test, not the final validation. A more decisive agency-focused assessment would preserve the transparency of the Fox-Hughes framework and extend it to calibrated probabilistic and operational outcomes.
Proposed evaluation protocol
- Independent event selection. Include multiple states and territories, diverse fuels and weather, fire sizes and well-documented fire progression periods.
- Common references. Compare all predictions against the same agency-derived, time-stamped observed footprints wherever possible.
- Matched initial conditions. Standardise ignition, wind and weather, fuels, terrain and forecast issue time, while documenting differences in each simulator's requirements.
- Current versions. Include current Spark Operational and available Phoenix versions rather than treating archived releases as today's benchmarks.
- Multiple diagnostics. Score threat score, false alarms and misses, bearing, forward spread, arrival time, prediction latency and uncertainty calibration.
- Operational impact. Ask Fire Behaviour Analysts to evaluate analyst effort, interpretability, warning support, reproducibility and decisions affected by the model.
What a meaningful result would look like
The result should not be a one-number winner. Agencies need to know which system performs best for which task, under what conditions, and at what operational cost. A current-generation evaluation should ask whether adding SWIM as an independent model, risk layer or scenario analysis improves decisions relative to existing practice.
Crucially, probability outputs should be validated as probabilities: across many cases, do locations given a nominal 90% impact probability experience fire approximately 90% of the time? For consequences, separate hazard footprint accuracy from damage and cost-model accuracy.
For jurisdictions pursuing nationally coordinated capability, a parallel pilot or technical comparison can build evidence without asking an agency to abandon its established Fire Behaviour Analyst workflows.
Put SWIM to the test on the fire scenarios that matter to your agency.
ATS welcomes technical reviews of the study methodology, collaborative evaluation on documented historical events, and live demonstrations of probabilistic bushfire spread, consequence forecasting and mitigation analysis. Discussion of a limited-scope proof of concept can begin without committing to adoption.
Source material
The published Australian study and the underlying technical report establish the original simulator framework. The SWIM figures shown above come from ATS's own retrospective analysis discussed in September 2026, and should be treated as ATS-reported until the full computational audit and case-level results are independently reproduced.
- Fox-Hughes, P., et al. (2024). “An evaluation of wildland fire simulators used operationally in Australia.” International Journal of Wildland Fire, 33, WF23028. DOI 10.1071/WF23028.
- Bureau of Meteorology (2017). An Evaluation of Fire Spread Simulators Used in Australia, Final Report v1.81, prepared with AFAC/NSW RFS involvement. Original technical report (PDF).
- ATS internal benchmark summary (September 2026). Nine-fire SWIM retrospective reconstruction, mean threat score 0.38, historical comparator median approximately 0.28, circle control mean 0.29, seven of nine exceeding historical case median, five of nine exceeding historical case best. Underlying per-fire score table not supplied for this case-study mockup.
- AFAC (2025–2026). Spark Operational 1.3 and 1.4 product updates. Ensemble capability and current release.
Attribution: Phoenix RapidFire, Spark and Australis are identified here for historical scientific comparison; their mention does not imply endorsement, partnership or direct participation in ATS's reconstruction. This ATS research case study is separate from the independent 2017/2024 studies.