Changes in Revised ISO 16140

Method comparison study for qualitative methods. The first important change is the introduction of the concept of a “paired” and an “unpaired” study. A paired study is a study where both the reference and the alternative methods use the same first step in the enrichment protocol, e.g. pre-enrichment of Salmonella in Buffered Peptone Water (BPW) for 16 hours to 20 hours at 37 degrees Celsius. In case of an unpaired study, the first step for the reference and alternative methods are not identical, e.g. the use of a specific enrichment broth other than BPW for the alternative method. In this case separate test portions (from the same sample) need to be tested for the reference and the alternative methods. This seems a technical change but it has a major impact on the interpretation of the results. In a paired study, it is easy to determine what is a false-negative result (= result that was positive by the reference method but negative by the alternative method). When the reference method gives a positive result the alternative method should also have a positive result as it is proven that the sample contains the target microorganism e.g. Salmonella. But in an unpaired study, there is no direct link anymore between the result obtained with the reference and alternative methods as they use different test portions. This is especially the case when the contamination level of the samples is very low. For qualitative methods the level of contamination should be very low in order to demonstrate that the methods work as intended. The heterogeneity caused by the low levels leads to differences between test portions as some test portions will not contain the target organism anymore and thus cannot be found positive.

For the interpretation of results, a new concept is introduced as well. The evaluation is based on so called Acceptability Limits (AL). These ALs are based on evaluation of previous validation studies and expert opinion. They are not based on formal statistical tests, like the [Symbol]2 test in ISO 16140:2003. This test was in practice not capable of detecting differences between the reference and alternative methods that were, according to many microbiologists, important differences. The ALs are based on the number of positive and negative deviations observed in the validation study, and maximum acceptable values for paired and unpaired studies are determined.

Another important difference compared to ISO 16140:2003 is the introduction of the Relative Level of Detection (RLOD) concept. In this new concept, the two methods are compared using the same (artificially contaminated) samples. The level of contamination should be very low so that not all samples tested will be positive for the target organism. By testing many replicates (20) at this low level, differences in the number of positives found by the reference and alternative methods can be observed. The observed difference is then compared to a preset AL. The actual level of contamination of the samples is not used in the evaluation of the data because the accurate determination of a very low level of contamination (sometimes lower than 1 colony forming unit per test portion) is difficult to determine. The experimental design of the RLOD study is also fully in line with the Probability of Detection used by AOAC International. The statistical analysis of the results is however different, being based on a generalized linear model developed by WG 2, Statistics of ISO/TC 34/SC 9.

Graph 1. Example of the Bland-Altman plot for the validation of quantitative methods.

Image credit: ISO

Quantitative methods. TThe “relative trueness” study replaces the former “linearity and accuracy” study. The experimental design of this part of the validation study is not changed compared to ISO 16140:2003, but the evaluation of the data is. Linear regression is no more carried out but the data are plotted in two different ways, as a scatter plot and as a Bland–Altman plot. The scatter plot gives an impression of the linearity of the results. In the Bland-Altman plot (see Graph 1) the difference between the alternative-method (log10 transformed) result and the reference-method (log10 transformed) result is plotted for each individual sample. This plot also contains the data points for samples where one of the two methods gave a result that was outside the quantification limit of the method (e.g. counts lower than 10 on a plate). Rules for the quantification limits of reference methods are described in ISO 7218, Microbiology of food and animal feeding stuffs—General requirements and guidance for microbiological examinations. Based on the two plots, the data are interpreted and visually outlying results should be investigated in order to explain the cause of this variation.