178
D. Hurley-Smith and J. Hernandez-Castro
Table 10.6 Dieharder, NIST and TestU01 results
Samples Dieharder NIST SP800-22 Alphabits Rabbit Small crush Crush
Device
#
passed
passed
passed
passed passed
passed
DESFire EV1 100
100
98
0
0
–
–
Quantis 16M 100
100
100
54
60
93
47
Post 16M
100
100
100
95
87
91
82
Quantis 4M
100
100
100
3
7
91
3
Post 4M
100
100
100
91
82
93
86
Quantis USB 100
100
100
3
21
89
3
Post USB
100
100
100
90
81
97
80
Comscire
PQ32MU
100
100
100
91
86
93
84
samples collected from one of each type of device. Ideally, more devices would be
tested, but the cost was a limiting factor (the cheapest device, a 4M, costs e900).
All devices pass Dieharder, while all but 2 EV1’s pass the SP800-22 tests. The
TestU01 toolkit has been used, with 4 of its statistical test batteries used to evaluate
all tested devices, including the EV1. Due to the sample size requirements of the
Crush tests, EV1 data has not been tested for either Crush test. Immediately, the
EV1 shows critical issues, failing the Alphabits and Rabbit batteries. The average
failure rate is 1 of 4 tests for Alphabits, and 5 of 16 tests for Rabbit. This shows
how the simple addition of a new test battery can instantly reveal weaknesses that
the better-known batteries cannot identify.
Raw Quantis samples, especially those of the 4M and its USB variant, also
perform very poorly on Alphabits and Rabbit. They also perform very poorly in
Crush, but a significant number of samples pass the Small Crush tests. This could be
because the Small Crush battery has many tests in common with SP800-22, leading
to a correlation between the results. Post-processing cleans up many of these issues,
but not completely. Most notably, Alphabits, Rabbit and Crush test results improve
dramatically, with the most drastic change being the jump from 3 passed tests for
the 4M under Alphabits, to 91 passes. This shows that appropriate use of a QRNG is
yet another factor to consider: improper use of a device may not be identified by the
more well-known test batteries and incorrect configuration can be as damaging as
any other form of non-randomness. The Comscire PQ32MU performs well on most
tests but struggles with the Rabbit and Crush tests.
Table 10.7 shows the results of Ent for the QRNG. DESFire results are not shown
to avoid repetition. A summary of the 100 samples tested shows that Post-processed
Quantis data, and the PQ32MU, passes the χ 2 and serial correlation tests with
no issues. All devices pass the other tests, hence their omission from this table.
However, the raw Quantis data fails the χ 2 test dramatically. Furthermore, the 4M
and its USB variant perform quite poorly on the serial correlation test at the bit
level. This emphasizes the need to test sequences across multiple block sizes to
identify issues that may occur at lower or higher orders of output. Unlike the EV1,
raw Quantis data does not provide an easily identifiable or consistent bias across
D. Hurley-Smith and J. Hernandez-Castro
Table 10.6 Dieharder, NIST and TestU01 results
Samples Dieharder NIST SP800-22 Alphabits Rabbit Small crush Crush
Device
#
passed
passed
passed
passed passed
passed
DESFire EV1 100
100
98
0
0
–
–
Quantis 16M 100
100
100
54
60
93
47
Post 16M
100
100
100
95
87
91
82
Quantis 4M
100
100
100
3
7
91
3
Post 4M
100
100
100
91
82
93
86
Quantis USB 100
100
100
3
21
89
3
Post USB
100
100
100
90
81
97
80
Comscire
PQ32MU
100
100
100
91
86
93
84
samples collected from one of each type of device. Ideally, more devices would be
tested, but the cost was a limiting factor (the cheapest device, a 4M, costs e900).
All devices pass Dieharder, while all but 2 EV1’s pass the SP800-22 tests. The
TestU01 toolkit has been used, with 4 of its statistical test batteries used to evaluate
all tested devices, including the EV1. Due to the sample size requirements of the
Crush tests, EV1 data has not been tested for either Crush test. Immediately, the
EV1 shows critical issues, failing the Alphabits and Rabbit batteries. The average
failure rate is 1 of 4 tests for Alphabits, and 5 of 16 tests for Rabbit. This shows
how the simple addition of a new test battery can instantly reveal weaknesses that
the better-known batteries cannot identify.
Raw Quantis samples, especially those of the 4M and its USB variant, also
perform very poorly on Alphabits and Rabbit. They also perform very poorly in
Crush, but a significant number of samples pass the Small Crush tests. This could be
because the Small Crush battery has many tests in common with SP800-22, leading
to a correlation between the results. Post-processing cleans up many of these issues,
but not completely. Most notably, Alphabits, Rabbit and Crush test results improve
dramatically, with the most drastic change being the jump from 3 passed tests for
the 4M under Alphabits, to 91 passes. This shows that appropriate use of a QRNG is
yet another factor to consider: improper use of a device may not be identified by the
more well-known test batteries and incorrect configuration can be as damaging as
any other form of non-randomness. The Comscire PQ32MU performs well on most
tests but struggles with the Rabbit and Crush tests.
Table 10.7 shows the results of Ent for the QRNG. DESFire results are not shown
to avoid repetition. A summary of the 100 samples tested shows that Post-processed
Quantis data, and the PQ32MU, passes the χ 2 and serial correlation tests with
no issues. All devices pass the other tests, hence their omission from this table.
However, the raw Quantis data fails the χ 2 test dramatically. Furthermore, the 4M
and its USB variant perform quite poorly on the serial correlation test at the bit
level. This emphasizes the need to test sequences across multiple block sizes to
identify issues that may occur at lower or higher orders of output. Unlike the EV1,
raw Quantis data does not provide an easily identifiable or consistent bias across