10 Challenges in Certifying Small-Scale (IoT) Hardware Random Number Generators
171
The standalone generators (Quantis, Comscire and ChaosKey entries in
Table 10.2) are substantially faster, making data collection trivial by comparison.
However, this does not mean that samples of appropriate size were tested.
SP800-90B states that an entropy source must provide 1,000,000 bits of sequential output for testing [449]. Concatenation of smaller sequences is tolerable if
contiguous output to that size is not possible, but is undesirable. 1000 such
sequences must be concatenated, according to NIST guidelines. SP800-22 extends
these requirements by recommending that 100 samples of the aforementioned size
are tested to validate the results [508]. AIS-31 and AIS-20 do not stipulate minimum
sample sizes. John Walker states that, in their default configuration, the Diehard tests
should be run over at least 100 MB of data [568].
With this in mind, the test reports of several TRNG manufacturers can be more
thoroughly analyzed. IDQ states that their Quantis devices pass the Diehard and
NISTSP800-22 batteries with no failure. 1 SP800-22 tests were conducted over 1000
samples of 1,000,000 bits in length. A significance level of 1% was maintained
throughout this process. Diehard was used over a single sample of 1 · 10 9 bits. Our
own tests confirm that IDQ’s report of no failures is true, even for larger samples
(ours were 2.1 GB in size). In this case, IDQ is a good example of a test protocol
that is in line with the recommendations of test developers.
Comscire’s PQ32MU, a QRNG that uses shot-noise as an entropy source, is a
different story. Their NIST-Diehard report 2 shows that the number of tests has been
reduced. The reduced sample size is one issue, but reducing the number of tests
can result in the loss of certain capabilities. Unless the removed tests are wholly
redundant, it is likely that their removal will impact the capability of the battery to
detect certain types of non-randomness. The insufficient sample size is cited as the
reason for excluding those tests. Comscire only tests this QRNG using 2 samples;
one of 8 · 10 7 bits and another of 1 · 10 6 bits. This is drastically below the suggested
sample size for Diehard. Even though these samples meet the requirements of NIST
SP800-90B in the most basic sense, they still fall short of SP800-22’s additional
recommendations requiring the testing of at least 100 samples. Considering the ease
with which samples can be generated from standalone RNGs such as these, it is
surprising that a more robust test process is not used.
10.4 Appropriate Selection of Tests
The correlation between tests in a battery, and as a whole if the evaluation
methodology involves multiple test batteries, must be considered. Statistical tests
have a limited range of issues that they are able to identify in the target RNG. Test
1 https://marketing.idquantique.com/acton/attachment/11868/f-004c/1/-/-/-/-/Randomness
%20Test%20Report.pdf.
2 https://comscire.com/files/cert/comscire-pq32mu-nist_diehard-validation-tests.pdf.
171
The standalone generators (Quantis, Comscire and ChaosKey entries in
Table 10.2) are substantially faster, making data collection trivial by comparison.
However, this does not mean that samples of appropriate size were tested.
SP800-90B states that an entropy source must provide 1,000,000 bits of sequential output for testing [449]. Concatenation of smaller sequences is tolerable if
contiguous output to that size is not possible, but is undesirable. 1000 such
sequences must be concatenated, according to NIST guidelines. SP800-22 extends
these requirements by recommending that 100 samples of the aforementioned size
are tested to validate the results [508]. AIS-31 and AIS-20 do not stipulate minimum
sample sizes. John Walker states that, in their default configuration, the Diehard tests
should be run over at least 100 MB of data [568].
With this in mind, the test reports of several TRNG manufacturers can be more
thoroughly analyzed. IDQ states that their Quantis devices pass the Diehard and
NISTSP800-22 batteries with no failure. 1 SP800-22 tests were conducted over 1000
samples of 1,000,000 bits in length. A significance level of 1% was maintained
throughout this process. Diehard was used over a single sample of 1 · 10 9 bits. Our
own tests confirm that IDQ’s report of no failures is true, even for larger samples
(ours were 2.1 GB in size). In this case, IDQ is a good example of a test protocol
that is in line with the recommendations of test developers.
Comscire’s PQ32MU, a QRNG that uses shot-noise as an entropy source, is a
different story. Their NIST-Diehard report 2 shows that the number of tests has been
reduced. The reduced sample size is one issue, but reducing the number of tests
can result in the loss of certain capabilities. Unless the removed tests are wholly
redundant, it is likely that their removal will impact the capability of the battery to
detect certain types of non-randomness. The insufficient sample size is cited as the
reason for excluding those tests. Comscire only tests this QRNG using 2 samples;
one of 8 · 10 7 bits and another of 1 · 10 6 bits. This is drastically below the suggested
sample size for Diehard. Even though these samples meet the requirements of NIST
SP800-90B in the most basic sense, they still fall short of SP800-22’s additional
recommendations requiring the testing of at least 100 samples. Considering the ease
with which samples can be generated from standalone RNGs such as these, it is
surprising that a more robust test process is not used.
10.4 Appropriate Selection of Tests
The correlation between tests in a battery, and as a whole if the evaluation
methodology involves multiple test batteries, must be considered. Statistical tests
have a limited range of issues that they are able to identify in the target RNG. Test
1 https://marketing.idquantique.com/acton/attachment/11868/f-004c/1/-/-/-/-/Randomness
%20Test%20Report.pdf.
2 https://comscire.com/files/cert/comscire-pq32mu-nist_diehard-validation-tests.pdf.