168
D. Hurley-Smith and J. Hernandez-Castro
Table 10.1 Standards applied in the testing of selected RNGs
Cost
Manufacturer
Device
e
Entropy source Certifications/tests
NXP
DESFire EV1
0.59
Not disclosed
CC EAL4+
NXP
DESFire EV2
1.25
Not disclosed
CC EAL5+
IDQ
Quantis 16M
2900 Beam splitter
NIST SP800-22, METAS, CTL
IDQ
Quantis 4M
1299 Beam splitter
NIST SP800-22, METAS, CTL
IDQ
Quantis USB 4M 990
Beam splitter
NIST SP800-22, METAS, CTL
Comscire
PQ32MU
1211 Shot noise
NIST SP800-90B/C
NIST SP800-22
Diehard
Altus Metrum ChaosKey
45
RBSJ a
FIPS 140-2
a Reverse biased semiconductor junction
Table 10.1 shows a selection of RNGs and their associated certifications. CC
EAL, METAS, CTL, and FIPS 140-2 are applicable as certifications from their
respective institutions. NIST SP800-22 indicates that the NIST methodology and
test battery were applied when testing the RNG in question (whether internally or
externally). Any RNG testing process requires a set of statistical tests. One of the
earliest examples of a statistical test battery for randomness is Marsaglia’s Diehard
battery [393]. NIST SP800-22 provides a more expansive series of tests developed
by Rukhin et al. [508]. The NIST battery contains 15 tests, which are evaluated
in terms of uniformity and proportion of p-values for each test. There has been
some criticism of the accuracy of these results. Marton and Suciu observed that
false alarms were common and that more tests that SP800-22 suggests can be failed
by otherwise robust RNGs [396]. NIST itself states that any failure is cause for
further investigation, but does not suggest any specific follow up procedures for
RNG testing. It is implied that further data collection and testing a larger number of
target devices are initial approaches to the problem.
Dieharder is an extension of Diehard, integrating the SP800-22 tests and the
original Diehard tests [116]. This brings the battery up to a total of 30 tests, with 76
variant tests in total. This battery requires a much larger body of test data than its
predecessors. To test a stream of data with no rewinds with every test in the suite,
one must collect 228 GB of data. This is far beyond the recommended parameters
suggested by NIST and CC. A 4 GB sample would rewind 57 times under the same
test conditions. If a sequence of repeats during a single execution of a given test,
type-1 errors may be introduced. The test may report such repetition as a violation
of its definition of randomness, and identify the sequence as non-random, when in
fact it was just insufficiently large. This highlights the importance of appropriate
data collection.
TestU01, developed by L’Ecuyer and Simard, is more of an RNG developer’s
toolbox than test battery [362]. However, it incorporates 5 different test batteries:
Alphabits, Rabbit, Small Crush, Crush, and Big Crush. Each battery has a differing
D. Hurley-Smith and J. Hernandez-Castro
Table 10.1 Standards applied in the testing of selected RNGs
Cost
Manufacturer
Device
e
Entropy source Certifications/tests
NXP
DESFire EV1
0.59
Not disclosed
CC EAL4+
NXP
DESFire EV2
1.25
Not disclosed
CC EAL5+
IDQ
Quantis 16M
2900 Beam splitter
NIST SP800-22, METAS, CTL
IDQ
Quantis 4M
1299 Beam splitter
NIST SP800-22, METAS, CTL
IDQ
Quantis USB 4M 990
Beam splitter
NIST SP800-22, METAS, CTL
Comscire
PQ32MU
1211 Shot noise
NIST SP800-90B/C
NIST SP800-22
Diehard
Altus Metrum ChaosKey
45
RBSJ a
FIPS 140-2
a Reverse biased semiconductor junction
Table 10.1 shows a selection of RNGs and their associated certifications. CC
EAL, METAS, CTL, and FIPS 140-2 are applicable as certifications from their
respective institutions. NIST SP800-22 indicates that the NIST methodology and
test battery were applied when testing the RNG in question (whether internally or
externally). Any RNG testing process requires a set of statistical tests. One of the
earliest examples of a statistical test battery for randomness is Marsaglia’s Diehard
battery [393]. NIST SP800-22 provides a more expansive series of tests developed
by Rukhin et al. [508]. The NIST battery contains 15 tests, which are evaluated
in terms of uniformity and proportion of p-values for each test. There has been
some criticism of the accuracy of these results. Marton and Suciu observed that
false alarms were common and that more tests that SP800-22 suggests can be failed
by otherwise robust RNGs [396]. NIST itself states that any failure is cause for
further investigation, but does not suggest any specific follow up procedures for
RNG testing. It is implied that further data collection and testing a larger number of
target devices are initial approaches to the problem.
Dieharder is an extension of Diehard, integrating the SP800-22 tests and the
original Diehard tests [116]. This brings the battery up to a total of 30 tests, with 76
variant tests in total. This battery requires a much larger body of test data than its
predecessors. To test a stream of data with no rewinds with every test in the suite,
one must collect 228 GB of data. This is far beyond the recommended parameters
suggested by NIST and CC. A 4 GB sample would rewind 57 times under the same
test conditions. If a sequence of repeats during a single execution of a given test,
type-1 errors may be introduced. The test may report such repetition as a violation
of its definition of randomness, and identify the sequence as non-random, when in
fact it was just insufficiently large. This highlights the importance of appropriate
data collection.
TestU01, developed by L’Ecuyer and Simard, is more of an RNG developer’s
toolbox than test battery [362]. However, it incorporates 5 different test batteries:
Alphabits, Rabbit, Small Crush, Crush, and Big Crush. Each battery has a differing