Why do treatment means have 'na' instead of mean comparison letters?
Why do treatment means have 'na' instead of mean comparison letters?
Starting in ARM 2024.1, AOV reports may use 'na' instead of mean comparison letters for certain assessment columns. This is accompanied by a footnote stating "Mean separation letters are 'na' (not applicable) when error variance is 0." There are two scenarios where this can occur.
Scenario 1: All observations are the same value
The first scenario is when an entire assessment has no variation in the observations, commonly all 0s or 100s when there is no pest presence or zero damage to the crop, for example.
In this situation there is 0 variability in the data set, and when there is no variance, ARM cannot perform an Analysis of Variance! Technically, the sum of squares calculation is 0, and thus the error mean square is 0. So no F statistic can be calculated for AOV (and so is reported as 'NaN' for "not a number"). Because no further analysis can be performed, ARM displays "na" to make it clear that no statistical conclusions can be drawn from this data because a statistical analysis was not performed.
Scenario 2: Differences among treatments but no differences across replicates
The second scenario is when an assessment has no variation across replicates within treatment, even though numerical differences exist between different treatments. For example, a "percent control" rating where the check is 0, but all other treatments completely controlled the pest, so all observations are 100. Or a visual assessment was performed that aggregates the entire treatment across replicates; we will use a general phytotoxicity example here:
There is variation across treatments in this situation, but 0 variability within each treatment. In the AOV table below, although the Sum of Squares is not zero, the error mean square is once again 0, and so there is no result from AOV:
So, like in Scenario 1 above, ARM displays 'na' to indicate that no statistical conclusions can be drawn from this data because a statistical analysis was not performed.
Another way to look at this scenario is to consider that each treatment reall has only one observation. Although there is replication in the study, for this particular assessment the observation was identical for every experimental unit (or like in our example the rating is an average across all replicates, and simply entered into ARM as replicated data). Thus, this assessment is like a one-replicate study or an observational rating, which cannot be analyzed using statistics.
A couple of points of emphasis:
- The 'na' does not mean "no significance." It means "we cannot calculate statistical significance."
- This does not mean the data has no value, just that statistical conclusions are not possible for this assessment.
Can we change this back?
No. ARM reports are now clearer and more accurate in the above scenarios, and we cannot, in good faith, return to a statistically misleading output.
This change has prompted many questions from ARM users, both for their own understanding as well as their clients. The conversations GDM has had with clients on this topic have been positive. Thus, we are confident the authorities and other clients will come to understand the situation, too, and trial results will be shared more accurately and understood better as a result of this change. In fact, a regulatory group brought this issue to our attention in the first place!
In summary, statistical conclusions cannot be drawn from the type of ratings described above. ARM reports now make this clear and evident, carrying on the tradition that ARM ensures the presentation of the research data is not misleading.