474
C. E. Budde et al.
literature case studies, enriched with RBOX elements and non-Markovian pdfs.
We estimated their UNREL 10 3 or UNAVA in increasingly resilient configurations.
To estimate these values we used various simulation algorithms: Standard
Monte Carlo (smc); Fixed Effort [19] for different number of runs performed
in each S k region (fe n for n = 8, 12, 16, 24, 32); restart [47] with thresholds
selected via a Sequential Monte Carlo algorithm [12] for different global splitting
values (rst n for n = 2, 3, 5, 8, 11); and restart with thresholds selected
via Expected Success [10], which computes splitting values independently for
each threshold (rst es ). fe n , rst n , and rst es , used the automatic isplit
framework based in our importance function, as described in Sec. 4.2.
An instance y is a combination of an algorithm algo, an rft, and a dependability metric. An rft is identified by a case study (CS) and a parameter (p),
where larger parameters of the rft CS p indicate smaller dependability values
p CS p . Running algo for a fixed simulation time, instance y estimates the value
p y = p CS p . The resulting ci (ˆ p y ) has a certain width ˆ p y ∈ [0, 1] (we fix the
confidence coefficient δ = 0.95). The performance of algo can be measured by
that width: the smaller ˆ p y , the more efficient the algorithm that achieved it.
The simulation time fixed for an rft may not suffice to observe rare events,
e.g. for smc. In such cases the FIG tool reports a “null estimate” ˆ
p y = [0, 0].
Moreover, the simulation of random events depends on the rng—and its seed—
used by FIG, so different runs may yield different results ˆ
p y . Therefore, for each
y we repeated n = 10 times the computation of ˆ
p y , to assess the performance of
algo in y by: (i ) how many times it yielded not-null estimates, indicated with a
bold number at the base of the bar corresponding to y (e.g. 8 10 in Fig. 5b);
(ii ) what was the average width ˆ p y , using not-null estimates only, indicated by
the height of the bar; and (iii ) what was the standard deviation of those widths,
indicated by whiskers on top of the bar. We performed n = 10 repetitions to
ensure statistical significance: a 95% ci for a plotted bar is narrower than the
whiskers and, in the hardest configuration of every CS , the whiskers of smc bars
never overlap with those of the best res algorithm.
Case studies. Our seven parametric case studies are: the synthetic models
DSPARE n and VOT m , with n ∈ {3, 4, 5} SBEs the first, m ∈ {2, 3, 4} shared
BEs the second, and one RBOX each; FTPP s [17], where we study one triad
with s ∈ {4, 5, 6} shared SBEs, using one RBOX for the processors and another
for the network elements; HECS o [43], with 2 memory interfaces, 4 RBOX (one
per subsystem), o ∈ {1, . . . , 5} shared spare processors, and 2o parallel buses;
and RWC u∈{4,...,7} [22, 21, 39], which combines subsystems RC v with one RBOX
and v ∈ {3, . . . , 6} SPAREs, and HVC w with another RBOX and w ∈ {2, . . . , 4}
shared SBEs. In total these are 26 rfts with pdfs that include exponential,
Erlang, uniform, Rayleigh, Weibull, normal, and log-normal distributions. In an
extended version of this work [9] we provide all details of our case studies.
Hardware. Experiments ran in two types of nodes in a SLURM cluster running
Linux x64 (Ubuntu, kernel 3.13.0-168): korenvliet nodes have CPUs Intel
® Xeon
®
E5-2630 v3 @ 2.40 GHz, and 64 GB of DDR4 RAM @ 1600 MHz; caserta has
CPUs Intel
® Xeon
® E7-8890 v4 @ 2.20 GHz, and 2 TB of RAM DDR4 @ 1866 MHz.
C. E. Budde et al.
literature case studies, enriched with RBOX elements and non-Markovian pdfs.
We estimated their UNREL 10 3 or UNAVA in increasingly resilient configurations.
To estimate these values we used various simulation algorithms: Standard
Monte Carlo (smc); Fixed Effort [19] for different number of runs performed
in each S k region (fe n for n = 8, 12, 16, 24, 32); restart [47] with thresholds
selected via a Sequential Monte Carlo algorithm [12] for different global splitting
values (rst n for n = 2, 3, 5, 8, 11); and restart with thresholds selected
via Expected Success [10], which computes splitting values independently for
each threshold (rst es ). fe n , rst n , and rst es , used the automatic isplit
framework based in our importance function, as described in Sec. 4.2.
An instance y is a combination of an algorithm algo, an rft, and a dependability metric. An rft is identified by a case study (CS) and a parameter (p),
where larger parameters of the rft CS p indicate smaller dependability values
p CS p . Running algo for a fixed simulation time, instance y estimates the value
p y = p CS p . The resulting ci (ˆ p y ) has a certain width ˆ p y ∈ [0, 1] (we fix the
confidence coefficient δ = 0.95). The performance of algo can be measured by
that width: the smaller ˆ p y , the more efficient the algorithm that achieved it.
The simulation time fixed for an rft may not suffice to observe rare events,
e.g. for smc. In such cases the FIG tool reports a “null estimate” ˆ
p y = [0, 0].
Moreover, the simulation of random events depends on the rng—and its seed—
used by FIG, so different runs may yield different results ˆ
p y . Therefore, for each
y we repeated n = 10 times the computation of ˆ
p y , to assess the performance of
algo in y by: (i ) how many times it yielded not-null estimates, indicated with a
bold number at the base of the bar corresponding to y (e.g. 8 10 in Fig. 5b);
(ii ) what was the average width ˆ p y , using not-null estimates only, indicated by
the height of the bar; and (iii ) what was the standard deviation of those widths,
indicated by whiskers on top of the bar. We performed n = 10 repetitions to
ensure statistical significance: a 95% ci for a plotted bar is narrower than the
whiskers and, in the hardest configuration of every CS , the whiskers of smc bars
never overlap with those of the best res algorithm.
Case studies. Our seven parametric case studies are: the synthetic models
DSPARE n and VOT m , with n ∈ {3, 4, 5} SBEs the first, m ∈ {2, 3, 4} shared
BEs the second, and one RBOX each; FTPP s [17], where we study one triad
with s ∈ {4, 5, 6} shared SBEs, using one RBOX for the processors and another
for the network elements; HECS o [43], with 2 memory interfaces, 4 RBOX (one
per subsystem), o ∈ {1, . . . , 5} shared spare processors, and 2o parallel buses;
and RWC u∈{4,...,7} [22, 21, 39], which combines subsystems RC v with one RBOX
and v ∈ {3, . . . , 6} SPAREs, and HVC w with another RBOX and w ∈ {2, . . . , 4}
shared SBEs. In total these are 26 rfts with pdfs that include exponential,
Erlang, uniform, Rayleigh, Weibull, normal, and log-normal distributions. In an
extended version of this work [9] we provide all details of our case studies.
Hardware. Experiments ran in two types of nodes in a SLURM cluster running
Linux x64 (Ubuntu, kernel 3.13.0-168): korenvliet nodes have CPUs Intel
® Xeon
®
E5-2630 v3 @ 2.40 GHz, and 64 GB of DDR4 RAM @ 1600 MHz; caserta has
CPUs Intel
® Xeon
® E7-8890 v4 @ 2.20 GHz, and 2 TB of RAM DDR4 @ 1866 MHz.