Applications of Cross-Entropy Method for Demand Based Network Reliability

Transkriptio

1 Tampere University of Technology Department of Information Technology Timo Pylvänäinen Applications of Cross-Entropy Method for Demand Based Network Reliability Master of Science Thesis Subject approved by the Department Council on 5..3 Examiners: Prof. Keijo Ruohonen Dr. Tech. Petri Korpisaari

2 Preface This thesis is the result of a research initiated in the summer of 3 in Datactica Ltd in Tampere. I would like to express my gratitude to my examiners professor Keijo Ruohonen and Dr. Tech. Petri Korpisaari. Also, I am indebted to Tuomo Valkonen for his suggestions regarding the variance analysis. Furthermore, I would like to thank my distinguished co-workers in Datactica Ltd. Finally, I would like to thank my friends and family for their continuing support throughout my studies. Tampere..3, Timo Pylvänäinen Matti Tapion katu F Tampere Tel: i

3 Contents Preface Contents Abstract Tiivistelmä List of Acronyms List of Symbols i ii iv v vi vii Introduction Network Reliability 3 3 Monte Carlo Methods 6 3. Crude Monte Carlo Variance of State Spaces Upper Bounds of Variance Expected Variance in Random Configurations Variance Reduction Techniques 8 4. Fishman s Sampling Plan Control Variates Method Importance Sampling Cross-Entropy Method Remarks on Variance Reduction Rare Event Estimation 7 5. Cross-Entropy Method in Rare Event Estimation Remarks on the Cross-Entropy Method Pair Connectivity Reliability Cross-Entropy Formulation ii

4 CONTENTS iii 6. Computing the Performance Metric Computing Reliability Metric Demand Based Network Reliability Demand Networks with Routing Calculating Traffic Loss for Fair Queue Performance Metric for Demand Networks Cross-Entropy Method for Demand Based Reliability Simulation Experiments and Results Simulation Environment Network Generation Cross-Entropy Monte Carlo Crude Monte Carlo Simulation Tests Simulation Test Simulation Test Simulation Test Simulation Test Results Results of Test Results of Test Results of Test Results of Test Conclusions Conclusions 57 References 59 A Implementation Details 6 A. Network Description A. Cross-Entropy Monte Carlo Estimator A.3 Crude Monte Carlo Estimator A.4 Systematic Evaluation

5 Abstract TAMPERE UNIVERSITY OF TECHNOLOGY Department of Information Technology Institute of Mathematics PYLVÄNÄINEN, TIMO: Applications of Cross-Entropy Method for Demand Based Network Reliability Master of Science Thesis, 6pp.; Appendix 4pp. Examiners: Prof. Keijo Ruohonen, Dr. Tech. Petri Korpisaari. Funding: Datactica Ltd. November 3 Keywords: network reliability, Cross-Entropy, Monte Carlo, variance reduction In the business world of today, it is vitally important for network providers to guarantee certain degree of reliability. In this thesis, a general network reliability problem is formulated. The solution to this problem can be estimated with Monte Carlo methods. The high accuracy requirements associated with the problem often lead a prohibitive number of samples. The accuracy of Monte Carlo methods depends on the variance of sampled random variable. Some properties of the variance of the network reliability formulation are proved. Methods to reduce the variance are introduced; the Cross- Entropy method in particular. To increase fidelity this thesis proposes to take into account traffic loss and routing. The Cross-Entropy method and importance sampling are applied to this particular formulation. An algorithm to compute traffic losses quickly is presented. The new formulation, which includes traffic loss and routing, does not readily lend itself to Cross-Entropy method. To overcome this problem, the routing and traffic loss is ignored in the distribution estimation step. The empirical results will show that the performance of the Monte Carlo method is improved regardless. iv

6 Tiivistelmä TAMPEREEN TEKNILLINEN YLIOPISTO Tietotekniikan osasto Matematiikan laitos PYLVÄNÄINEN, TIMO: Ristientropian soveltaminen vaatimuksiin perustuvan verkon luotettavuuden arvioinnissa Diplomityö, 6s.; liite, 4s. Tarkastajat: Prof. Keijo Ruohonen, TkT Petri Korpisaari. Rahoittajat: Datactica Oy. Marraskuu 3 Avainsanat: verkon luotettavuus, ristientropia, Monte Carlo, varianssin vähentäminen Nykypäivänä verkkopalveluiden tarjoajille on elintärkeää taata tietty luotettavuustaso. Tässä työssä formuloidaan yleinen verkon luotettavuusongelma. Tämän ongelman ratkaisua voidaan arvioida Monte Carlo -menetelmin. Ongelmaan liittyvät korkeat tarkkuusvaatimukset johtavat usein liian suuriin otosjoukkoihin. Monte Carlo -menetelmien tarkkuus on suoraan riippuvainen tutkittavan satunnaismuuttujan varianssista. Työssä todistetaan verkon luotettavuusongelmaan liittyvän satunnaismuuttujan varianssin ominaisuuksia. Joitakin menetelmiä varianssin pienentämiseksi esitellään. Todenmukaisuuden parantamiseksi tässä työssä ehdotetaan huomioitavaksi myös reititys ja mahdollinen hävikki verkossa. Ristientropiamenetelmää sovelletaan tämän ongelman ratkaisemiseksi. Työssä esitellään nopea algoritmi hävikin laskemiseksi. Reitityksen ja hävikin huomioiva formulointi ei kuitenkaan sellaisenaan sovi ristientropiamenetelmän käyttöön. Tämä ongelma kierretään jättämällä reititys ja hävikki jakaumanarviointivaiheessa huomiotta. Kokeelliset tulokset osoittavat, että tällöinkin ristientropiamenetelmä parantaa Monte Carlo -menetelmän tarkkuutta. v

7 List of Acronyms CMC Crude Monte Carlo method CE Cross-Entropy CE-MC Cross-Entropy Monte Carlo method vi

8 List of Symbols Reliability Analysis Notations D E G P (s) R S S A S U V W c i e i f p i q i r i r i s v i w i δ γ The demand graph for the network G The link set of graph G A graph describing a network topology Probability of state s The set of paths for the demands The set of possible states for the network The set of operational states The set of failed states The node set of graph G The set of links of D The capacity of link e i A link, an element of E The reliability function or the characteristic function of S A end node after traffic loss The probability that link e i is operational The probability that a link is failed, q i = p i The traffic rate of the demand w i The traffic rate of the demand w i at the end of the route Vector defining a state for a network A node, an element of V A demand, an element of W Maximum allowed traffic loss fraction The performance limit for the operational status of the network Monte Carlo Notation E[f] E[x] E π [f] E v [f] The expectation value of f The expectation value of the random variable x The expectation value of f over the distribution π The expectation value of f over a distribution defined by the parameter v vii

9 LIST OF SYMBOLS viii E[V (x c N )] The expectation value of the variance of x given c N Ê m [f] The Monte Carlo estimate of E[f] taken from m samples. N Number of states S N Ω The set of points in the N-dimensional simplex defined by the origo and the N orthogonal unit vectors of the standard Cartesian coordinate system The set of elementary events Φ The cumulative distribution function of the normal distribution N(, ) m The number of samples p i x x x i c N s i The probability of the value x i A random variable The sample mean of a random variable One of the possible values of the random variable x A configuration of a network with N possible states A random state sample µ The expectation value of a random variable µ x The expectation value of the random variable x σ The variance of an random variable cov(s, t) The covariance of the random variables s and t V(x) The variance of the random variable x V(x c N = c) The variance of x given the configuration Cross-Entropy notation M(s) The performance of the state s W (s, u, v) The ratio of the probability of the state s in the distributions defined by the parameters u and v f γ The characteristic function of the set of failed states defined by performance metric and the performance level γ g A distribution g π(s) π(s, v) v ˆv D(g, h) Exp(λ) The optimal choice for the distribution g A distribution of states A distribution of state defined by the parameter v The optimal choice for the parameter v An estimate of the optimal choice for the parameter v The Cross-Entropy distance between distributions g and h The parametric exponential distribution with mean λ

10 LIST OF SYMBOLS ix Results notation N sc N mc The number of samples in the stochastic counterpart The number of samples in the final estimation

11 Chapter Introduction Information has become, in many respects, a necessity of life in todays world. Many people earn their living in the business of generating and refining information. The Internet has revolutionized the way information is shared. The number of search engines is an indication of the importance of information in people s life today. The need for accurate and up to date information has become vitally important for many of the businesses today. Thus there is a lot of money in the data transfer business. The network operators are expected to provide uninterrupted data transfer services. Interruptions can be extremely expensive for both the network operator and the customer. A completely failsafe network is an impossibility, but it has become vitally important to make sure that networks fail with extremely low probability. In 96, a typical reliability requirement was a 4 hr mean time between failure. In the 99s the requirements were stated as 5 minutes of system-outage/year []. Network reliability analysis helps to achieve this goal. In the early days of reliability engineering, many analytical solutions were developed. These were based on highly simplifying assumptions.[8] The real problem, however, is so complex that it cannot be solved by analytical means. The network reliability problems are in general members of the family of NP-hard problems[]. With the advances in computing technology, people have now turned to approximate methods, simulation in particular. The basic simulation method is Monte Carlo sampling. In Monte Carlo sampling, the states of the network are generated according to the probabilities of the states. From only a few samples, the probability of a failed state is estimated as a fraction of failed states to the operational states. When the probability of a failed state is small, huge numbers of samples are required for accurate approximations. Importance sampling is a 5-year old technique, where the samples are not generated from the true distribution of the states, but from some other distribution.[] The sample mean is weighted accordingly to correct for the error that this introduces. This can significantly improve the performance of Monte Carlo sampling. The im-

12 CHAPTER. INTRODUCTION provement that importance sampling brings, depends highly on the new distribution that is chosen. Cross-Entropy, sometimes called relative entropy, was first introduced by Kullback and Leibler in 95[8]. It can be thought of as a measure of distance between two distributions. The optimal importance sampling distribution depends on an unknown scaling factor. Thus it is impossible to directly choose the optimal distribution. The Cross-Entropy distance, however, has somewhat unique analytical properties. It will be seen in this thesis, that it is theoretically possible to minimize the Cross-Entropy distance to the optimal distribution without knowing the unknown scaling factor. The application of Cross-Entropy measure in this context was fairly recently discovered by Rubinstein[4]. While in practice the actual analytical solutions are usually not available, this notion provides a framework where the importance sampling distribution can be estimated. The traditional network reliability measures usually consider the probability that a network is connected. Brunilde and François[6] point out, however, that routing is an important factor. Only considering connectivity can lead to an overestimation of the network reliability. One of the significant effects of routing is the unexpected creation of bottlenecks. In some cases, it is natural to accept a certain amount of traffic loss. This thesis formulates a network reliability formulation where routing is considered and the possibility for traffic loss is allowed. The applications of the Cross-Entropy method to this problem is studied. To the author s knowledge, such study has not been conducted previously in literature. Some algorithms are presented which can be used to quickly compute values required in the application of the Cross-Entropy method to the demand based network reliability. These algorithms constitute another contribution of this thesis, in addition to the study of application of the Cross-Entropy method to the demand based network reliability problem. The remainder of this thesis is organized as follows. Chapter formalizes the network reliability problem. Chapter 3 presents the basic Monte Carlo method and the problems associated with it. The performance of the basic Monte Carlo method relies on the variance of the underlying random variable. Some theorems about the variance of the network problem are proved in Chapter 3. In Chapter 4 the various variance reduction methods that can be applied to Monte Carlo methods are introduced. Especially the Cross-Entropy method is described in detail. Chapter 5 applies the Cross-Entropy method to a general rare event estimation problem. The network reliability problem is a special case of rare event estimation. In Chapter 6 the basic connectivity network reliability problem is studied. Chapter 7 extends the study in Chapter 6 to the case where routing and traffic loss is also considered. In Chapter 8 the developed methods are tested empirically. The thesis is concluded in Chapter 9.

13 Chapter Network Reliability A network in this thesis is represented as a graph. In practice, the nodes often represent routers of a data network. In such cases, these nodes are in charge of routing decisions. A network of routers forms a rather complex system which is hard to analyze. Especially the convergence and dynamic behavior of the network is analytically very difficult. Thus researchers and engineers usually rely on simulation to analyze various network and protocol designs. The performance of the network is bound by the physical resources available and the logic which controls these resources. That is, the network may not, for example, have sufficiently sophisticated routing algorithms to fully take advantage of the available capacity. Traditionally reliability analysis only focuses on the physical side. Often the logic of the network is completely neglected. This can give us, in a way, an upper bound on the performance. The network cannot, under any circumstances, perform better than the physical restrictions allow. It is possible, however, to also consider some of the non-physical properties of the network in reliability analysis. This thesis in particular takes into account the routing logic. Traditionally there have been two approaches to reliability analysis.[][6]. Connectedness: A measure of the probability of network connectedness is derived.. Capacity: A measure of the probability that the network capacity is sufficient to meet the demands is derived. Measures for the probability of connectedness include the probability that a given pair of nodes are connected, the expected number of communicating node pairs and the probability that the network is fully connected.[] As for meeting demands, one can imagine any number of measures, depending on what the demands are. Many researchers, however, seem to consider the probability that the network is physically capable of transferring the requested traffic without loss. This may entail finding optimal routing, which is a very difficult problem in itself. 3

14 CHAPTER. NETWORK RELIABILITY 4 Some have pointed out, that the analysis is often detached from reality.[8] The justification for this has been the computational intractability of the problems. Network reliability problems are almost without exception NP-hard problems.[8] It is, however, questionable whether this justifies analysis that leads to misleading results. Considering routing and traffic loss is a step closer to more realistic models. The reliability situation is best formulated as a graph theoretical problem. There are a number of possible formulations, which stem from different ideas of reliability. Many consider the probability that the network is fully connected. This thesis takes a more general approach. Many of the commonly used reliability measures can be seen as special cases of this. The following notations are used. G v i e i s P p i q i S f The graph describing the topology of the network. G = (V, E) where V is the set of nodes and E is the set of links A node, an element of V A link, an element of E The vector s = (s, s,..., s n ) denotes the state of the network. s i = when the link e i is up, otherwise s i = P is the probability measure for the states. P (s) is the probability of an individual state The probability that link e i is operational The probability that link e i is failed, q i = p i The set of possible states for the network A mapping f : S R. f can be thought of as the reliability measure for state s The reliability measure for the system is the expectation value of f: E[f] = s S f(s)p (s). (.) By the choice of f one can get meaningful reliability measures. One choice for f can be the characteristic function for the subset of operational states. That is, S is partitioned into operational states S A and failed states S U and s S U, f(s) = s S A. (.) Then E[f] gives the probability that the network is in an operational state. This gives a very meaningful and intuitive meaning for the reliability measure. The characterization of operational states can, of course, be fairly arbitrary. Characterizations based on connectivity or demands again give meaning to the reliability measure.

15 CHAPTER. NETWORK RELIABILITY 5 Notice that the formulation could be further generalized by allowing an arbitrary state space. Such a state space could, for example, reflect stochastic capacities of the links. For the sake of clarity, however, this generalization is omitted. At first glance the network G plays no role in the expectation value in (.). This, however, is not the case. While it is usually not explicitly stated, the function f clearly must depend on the properties of G. Suppose for example that f is the characteristic function (.). A state is considered operational, when there is a path between given nodes v i and v j. Then clearly, to compute f for a given state, knowledge of the topology of the network G is required. To simplify notation, however, any reference to G from f is omitted. From here on, it is implicitly assumed that f is somehow based on the network G. Notice also, that when the link failures are independent, P (s) can be readily computed. P (s) = p i i {i s i =} i {i s i =} q i. (.3)

16 Chapter 3 Monte Carlo Methods In general the exact computations of the network reliability measures are, or give rise to, NP-hard problems.[] This is already sufficient justification for approximation methods. Monte Carlo methods are further supported by the fact that they provide a rather general framework, which does not limit the choice of the reliability measure nor the characteristics of the network. Monte Carlo methods, in general, refer to a collection of means to approximate various values with the aid of random numbers. The applications of Monte Carlo methods include the approximation of π, computation of difficult integrals and the computation of the free energy of a system.[] The problems are formulated so that the expectation value for a function f of the random variable gives the required value. Reliability analysis is a particularly fortunate case, since typically the state space is finite and the distribution of states is well defined. Thus it is easy to generate pseudo random numbers which follow exactly the defined distribution. 3. Crude Monte Carlo The naive Monte Carlo method, sometimes referred to as the Crude Monte Carlo (CMC), goes as follows. Draw m samples s,..., s m from a known distribution of s. The estimate of E[f] is then Ê m [f] = m (f(s ) + f(s ) + + f(s m )). (3.) It follows from the law of large numbers that[] lim Ê m [f] = E[f], with probability. (3.) m The error E[f] Ê[f] m is a random variable. The central limit theorem states that when m is sufficiently large, the distribution of the error term approaches that of 6

17 CHAPTER 3. MONTE CARLO METHODS 7 Descriptive function Cumulative probability Logarithmic absolute error for systematic evaluation Log error Log error Fraction of probability space evaluated Logarithmic absolute error for CMC Number of states as a fraction of distinct states Figure 3.: Systematic evaluation of the state space can be understood as integration over the descriptive function. The accuracy of the CMC method does not develop quickly as the number of samples is increased. normal distribution.[] That is (E[f] Êm[f]) N(, σ ), in distribution, (3.3) m where σ is the variance of f(s). This tells something about the convergence of the sample mean. The traditional sales pitch for Monte Carlo methods is that their convergence is independent of the dimensionality of the random variable. In this particular case, this property is reflected in the fact that one basically needs equally many random samples to estimate the probability of a tail in coin tossing, as for estimating the network reliability for arbitrarily large network. Figure 3. compares the accuracy of a systematic approach to the CMC method. The network that this example is based on is depicted in Figure 3. along with the reliabilities of individual links. The network is considered operational, if there is a path between nodes s and t. The exact reliability, given by the systematic approach, for this network is.4787.

18 CHAPTER 3. MONTE CARLO METHODS 8.3 s t Figure 3.: A simple network. The numbers indicate the reliability of the link. The probability that nodes s and t are connected is The descriptive function at the top of Figure 3. is the graph of the function f D (x) = f(s i ), when x B i, (3.4) where s i denote the possible states in order of ascending probability and i B i = {x P (s i ) < x j= i P (s i )}. (3.5) j= Normal integration over this function results in the expectation value: f D(x)dx = E s [f]. The descriptive function illustrates the structure of the state space. It shows how the operational sates are inconveniently mixed with the failed states. The width of the bars indicate the probability of a set of consecutive operational states. The systematic approach here is to evaluate most probable states first. The middle graph represents the logarithm of the absolute error for the systematic approach. The systematic approach leads to monotonically increasing estimate, thus the error is monotonically decreasing. The plot shows that almost all of the states need to be considered for accuracy of ±.. In the Figure 3., the graph of the systematic approach corresponds to f D (x). The x-axis in the two upper plots represents the portion of probability space covered. This connection nicely illustrates how the systematic approach is not so much dependent on which states are most probable. Rather, the most significant factors are the most probable operational states. In Chapter 4, it is seen how this fact can be exploited to our advantage. On the other hand, CMC reaches a relatively good accuracy with only a few samples. It is also apparent, however, that the accuracy of the CMC method does not increase quickly as the number of samples increases. The plot is drawn such, that the same number of samples was considered for the CMC and the systematic approach at each point. This causes the x-axis to be nonlinear with respect to the number of samples. The reliability measure is usually bounded. Thus the variance of f is bounded above. This makes it possible to compute an upper bound on the number of samples required for a given accuracy and confidence. The next section studies these upper bounds.

19 CHAPTER 3. MONTE CARLO METHODS 9 3. Variance of State Spaces Consider a system which can be in N possible states. Denote the states by s i, i =...N and S = {s i, i =...N}. Let x : S [a, b] be a mapping that associates to each state some bounded value. Then x can be considered a discrete random variable on its own. Suppose then, that a Monte Carlo simulator is used to estimate E[x]. It was seen earlier, that for large number of samples the distribution for the sample mean approaches the normal distribution with a variance of σ N, where σ is the variance of x. Thus, for m samples it is possible to estimate the probability P ( µ x < c), where µ is the true expectation value for the random variable x and x = m m i= x i, where x i are the m random samples. To estimate P ( µ x < c) it is assumed that m is large enough so that the distribution of the sample mean is very close to normal. The probability for the error to be less than c is then Φ ( ) ( ) cn σ Φ cn σ, where Φ is the cumulative distribution function of the normal distribution N(, ). Given some accuracy c, it is then possible to calculate the number of samples required to reach the accuracy c with a given probability. We refer to the probability of reaching the accuracy as confidence. The problem is, that the variance of the random variable x is unknown. It is, however, possible to make estimates on the magnitude of σ. 3.. Upper Bounds of Variance Since x is bounded in [a, b], it follows that µ = E[x] [a, b]. Then since N V(x) = (µ x(s i )) p i i= the variance is bounded above. N N (b a) p i = (b a) p i = (b a), (3.6) i= i= Thus the variance for the sample mean is less than (b a). Suppose, for instance, that N x is bounded in [, ]. Then the variance is less than. Then it is known that N approximately 9 samples will guarantee that for 99.9% confidence the sample mean is within unit of the true value. There is a stricter upper bound for the variance, which is almost equally intuitive, yet significantly more difficult to prove. The following lemma will be needed. Lemma 3.. Assume x < x < x 3 and x < µ < x, then (µ x ) x 3 x x 3 x (µ x ) + x x x 3 x (µ x 3 ). (3.7) Proof. This is of course equivalent to f(µ) = (µ x ) x 3 x x 3 x (µ x ) x x x 3 x (µ x 3 ). (3.8)

20 CHAPTER 3. MONTE CARLO METHODS The derivative with respect to µ is d dµ f(µ) = µ x rµ + rx qµ + qx 3 = µ (r + q)µ x + (rx + qx 3 ) = x + (rx + qx 3 ) (3.9) =, where r = x 3 x x 3 x and q = x x x 3 x and thus r + q =. This shows that the choice of µ is irrelevant. The choice µ = x yields f(x ) = (x x ) x 3 x x 3 x (x x ) x x x 3 x (x x 3 ) = (x x ) (x x )(x 3 x ) (3.), since x < x < x 3 implies x 3 x > x x. The following theorem gives an upper bound for the variance. Theorem 3.. Let N i= p i = and x i [a, b] for all i. Then where µ = N i= x ip i. N (µ x i ) p i 4 (b a), (3.) i= Proof. For N = it holds that N (µ x i ) p i = (p x + ( p )x x ) p + (p x + ( p )x x ) ( p ) i= = (x x ) (p p ) (b a) (p p ) 4 (b a). For clearly p p has a maximum at p =, and thus p p 4. (3.) For N > reorder the x i and p i if necessary such that x = min{x i, i =...N} and x N = max{x i, i =...N}. Now, if N >, then find x k, so that x < x k < x N. Then the contribution to the mean µ from these three is x p + x k p k + x N p N, which remains unchanged if x k is removed and its probability p k distributed for x and x N

21 CHAPTER 3. MONTE CARLO METHODS ( such that p x = p + p N x k k x N x and p N = p N + p k x N x k by the lemma it holds that x N x ) = p N + p k x k x x N x. Then, p k (µ x k ) p k x N x k x N x (µ x ) + p k x k x x N x (µ x N ). (3.3) Thus the left hand side of (3.) is increased and it follows that N (µ x i ) p i i= N i=,i k (µ x i ) p i. (3.4) The right hand side of this has N values, and thus by induction it follows that the theorem holds true for all N. Consider the example at the beginning of this section. By the theorem it holds that σ 5 for the variance of x. Thus it is now known that for 99.9% confidence the sample mean is within unit of the true value for only 76 samples. The upper bound in (3.) is a strict upper bound in the sense that the equality may hold. Especially when N = selecting p = p =.5, x = a, x = b yields a variance of exactly (b 4 a). Thus no lower upper bound exists for the general case. 3.. Expected Variance in Random Configurations The upper bound for the variance, in a way, represents the worst case scenario. From the point of view of the system developer, it is not known a priori what kind of configuration is to be used in a simulation. To get some idea of the number of samples required, the expected variance for the random variable x could be used. Here, configuration means some set of parameters for a simulator that defines a state space of N possible states and their probabilities as well as the associated values of x. Let us assume that all configurations resulting in N possible states are equally likely, e.g. uniformly distributed. Denote the configuration resulting in N states by c N. The configuration c N is a random variable and thus V(x c N ) as a function of a random variable is a random variable. The expectation value of the variance E[V(x c N )] is therefore meaningful. It is, of course, a function of N as well. The following lemmas are needed to derive E[V(x c N )]. Lemma 3.3. p p p P N i= p i N i= p i dp N dp N...dp = N (N + )!. (3.5)

22 CHAPTER 3. MONTE CARLO METHODS Proof. First it holds that P N i= p i N p i dp N...dp = i= N i= P N i= p i p i dp N...dp. (3.6) Then by Fubini s well known theorem[4] P N i= p i p i dp N...dp = p i dv, (3.7) S N where S N is the N-simplex that the iterated integral defines. That is, the simplex defined by the N orthogonal unit vectors and origo. Then by the symmetry of this volume, the value of the integration in equation (3.7) is independent of i and thus P N i= p i N p i dp N...dp = N i= p p Then by substituting p = ( p )a, p3 = ( p )a 3 etc. P N i= p i dp N...dp. (3.8) N p p a N p P N i= p i dp N...dp = P N i= a i = N ( p ) N da N da N..da dp p ( p ) N S N dv dp. (3.9) Now, S N dv is the volume of N -simplex, which is generally known to be. Thus by consecutive partial integration (N )! N p ( p ) S N N dv dp = p N (N )! ( p ) N dp N = (N )! N(N + )(N + ) = N (N + )!. (3.) Next, a similar lemma Lemma 3.4. p p p P N i= p i ( N p i ) dp N dp N...dp = i= (N + )! (3.). I am indebted to my colleague Tuomo Valkonen for the idea of using a symmetric volume to show that the terms are equal for all i.

23 CHAPTER 3. MONTE CARLO METHODS 3 Proof. Substitute p N = ( N i= p i)a N, to get P N i= p i N ( p i ) dp N...dp = P N i= p i i= ( = 3 N i= p i ( N i= P N i= p i p i )a N ) ( ( N i= N i= p i ) da N dp N...dp p i ) 3 dp N...dp. (3.) Then continue by substituting p N = ( N i= p i)a N, and so forth to get P N i= p i N ( p i ) dp N...dp = 3 4 ( p ) N+ dp N + i= (3.3) = (N + )!. These lemmas are needed to prove main result of this section: the expectation value for the variance as a function of N. At this point, it seems appropriate to make a comment on the assumptions. While at first glance it seems reasonable to assume all configurations are equally likely, a moments consideration, however, shows that this can hardly be the case. For example in a network simulator, a configuration where traffic loss is always % has to be unlikely compared to the average case. On the other hand, x is bounded in [a, b] and can have N possible values. Similarly the corresponding probability distribution must be defined within an N -simplex. Thus the configurations can be thought to be uniformly distributed in [a, b] N S N. Where S N is an N -simplex. Thus, the possible configurations lie within N -dimensional volume, the size of which can be evaluated. For such a volume, uniform distribution is computationally easy, while almost any other distribution seems difficult. For the uniform distribution, the probability density function is x Ω, Vol(Ω) f(x) = otherwise, where Ω is the volume in which the allowed values for x lie. So in this case (3.4) c N [a, b] N S N = Ω. (3.5) The N probabilities define the Nth probability, and so the simplex is in a lower dimension. It holds that Vol(Ω) = b a b p a = (b a) N (N )!, P N i= p i dp N...dp dx N...dx (3.6)

24 CHAPTER 3. MONTE CARLO METHODS 4 because the outer integrals form an N-dimensional hypercube with edge length of b a and the inner integrals are the volume of N -simplex. Equation (3.4) now becomes (N )! x Ω, (b a) f cn (x) = N otherwise. (3.7) Now V(x c N = c) = N N (x i p j x j ) p i, (3.8) i= j= where x i and p i are defined by c. Also, notice that p N is only defined implicitly by p N = N i= p i. The expectation value for the variance then is And so E[V(x c)] = Ω V(x c N = c)f cn (c)dc. (3.9) E[V(x c)] = b a b a Vol(Ω) P N m= p m N i= N (x i i= P N m= p m N p j x j ) p i f cn (c) dp N...dp dx N...dx = j= b a b a I i dx N...dx dp N...dp, (3.3) where N N I i = x i p i x i p i p j x j + p i j= j= k= N p j p k x j x k and c = [p,..., p N, x,..., x N ]. Now integrate each term separately. For the first term it holds that b a b a b x i p i dx N...dx = p i a b a x i dx i dx N...dx = p i 3 (b3 a 3 )(b a) N, (3.3) where the integral for x i is moved to innermost integral and it evaluates to 3 (b3 a 3 ), which is a scalar. Then the rest of the integrals obviously give (b a) N.

25 CHAPTER 3. MONTE CARLO METHODS 5 The second term becomes b ( b N ) x i p i x j p j dx N...dx = a p i a N j=,j i b p j a j= b a b x j x i dx i dx j dx N...dx 3 p i (b 3 a 3 )(b a) N = a 4 p i( p i )(b a ) (b a) N 3 p i (b 3 a 3 )(b a) N, (3.3) where the fact that N k=,k j p k = ( p j ) is used. Similarly with the first term, the two integrals containing a variable are integrated first and each evaluate to (b a ), giving the term 4 (b a ). The right hand term results from the terms left out of the summation, where k = j, this can be evaluated as the previous case. Finally using the same techniques the third term becomes b b N N p i a a j= k= N N b b p i j= k=,k j a a p i 4 (b a ) (b a) N Collecting these yields P N m= pm (b a) N + N i= p i p j p k x j x k dx N...dx = b a x j x k p j p k dx N...dx + p i b P N m= p i ( (b a ) 4 a N j= N b j= a b p j ( p j ) + p i 3 (b3 a 3 )(b a) N N I i dx N...dx dp N...dp = i= ( N i= (b3 a 3 )(b a) p i ( (b 3 a 3 )(b a) 3 (b a ) 4 a (b a ) 4 x jp j dx N...dx = N p j. (3.33) j= + (b3 a 3 )(b a) 3 ) + (b a ) 4 ) ) dp N...dp, (3.34) where the fact that N i= p i = is used. Notice also, that the summation N j= p j from the third term of I i combines with the p i from second term. Collecting terms and moving constants out of the integral yields ( ) (b (b a) N 3 a 3 )(b a) (b a ) ( = K 3 ( P N m= pm P N m= pm ( 4 ( N i= N i= p i p i + ( ) N i= dp N...dp ) p i ) ) dp N...dp ). (3.35)

26 CHAPTER 3. MONTE CARLO METHODS 6..8 Variance Number of states Figure 3.3: The expected variance of the reliability measure is bounded above as the number of states increases. Finally applying Lemmas 3.3 and 3.4 yields E[V(x c N )] = K ( ) (N ) Vol(Ω) (N )! (N + )! (N + )! ( K(N )! = (b a) N (N )! N ) (N + )! ( K = ). (b a) N N + (3.36) The following theorem follows by substituting K back and simplifying. Theorem 3.5. E[V(x c N )] = (b a) ( ). (3.37) N + The growth of the expected variance is illustrated in Figure 3.3. Similarly V(x c N ) is a random variable and has a variance. The variance is expected to approach zero, as the number of states grows. This is supported by empirical studies. A straight forward proof of this, however, leads to complex integrations. The results give an idea of the expected number of samples required for a given accuracy. The random nature of the model, however, excludes the possibility of ever being entirely certain that such an approximate answer is even close to correct. Nothing can be done about this. In practice, the solution often converges much faster than the analysis may indicate. This is because of the nature of the natural distribution. The probability mass is densest near the expectation value. For example, if the the number of samples required for the accuracy of ± to a confidence of 99% is estimated to be N, then with 95% probability about.57n samples is sufficient for the accuracy. The upper bound of the variance is also likely to be a gross overestimation. Theorem 3.. indicates that the expected variance is over three times lower than the upper

27 CHAPTER 3. MONTE CARLO METHODS 7 bound. The assumptions which lead to the expected variance, however, may not hold for actual networks. The expected variance still remains high in terms of the number of samples required to gain a reasonably accurate answer. Especially in network analysis, the absolute accuracy must be extremely high. The number of samples required is proportional to, where c is the absolute error. Thus methods to reduce the variance are an absolute necessity for the application of Monte Carlo methods to the network c reliability problem. The next chapter will study such methods.

28 Chapter 4 Variance Reduction Techniques The results in the previous chapter can give us an estimate of the number of samples required. The latter result also explains why in practice significantly fewer samples seem to lead to accurate results. The computation of the reliability measure even for a single state can easily take several seconds in a large network. When the number of samples is still in the thousands, this leads to uncomfortably slow analysis. The variance of the CMC estimator Êm[f] is dependent on the number of samples as well as the variance of the random variable f(s). This chapter introduces methods to reduce the variance of the estimator, leading to more accurate estimations for the same number of samples. 4. Fishman s Sampling Plan Fishman s sampling plan[] is a method which can reduce the number of samples required when the reliability measure is terminal pair connectivity and state probability is based on independent link failure probabilities. That is, network is considered operational when there is a path between some selected pair of nodes s, t. It can be generalized to global connectivity, where the network is considered operational only when there is a path between every pair of nodes.[] Unfortunately, while the number of samples is reduced, the generation of samples becomes more complex in Fishman s sampling plan. The idea is to split the state space into three sets Ω (), Ω (), Ω (3). All states in Ω () are operational and all states in Ω (3) are failed. The operational status of states in Ω () is unknown. The partitioning of the state space is done so that the probabilities P (s Ω () ) and P (s Ω (3) ) can be easily evaluated. Monte Carlo methods are then used to approximate the probability for P (s Ω () ). The difficulty is in generating samples in Ω () according to the correct distribution. The partitioning into the three sets can be done on the basis of I disjoint minimal paths P,..., P I between the two terminals s and t, and J disjoint minimal cuts 8

29 CHAPTER 4. VARIANCE REDUCTION TECHNIQUES 9 C,..., C J separating s and t.[] Ω () contains any state of the network where one of the paths P j is available. That is, on at least one path, no link is down. Ω (3) contains any state where one of the cuts C i separates the nodes s and t. In other words, for some i all the links in C i are down. The probability P (s Ω () ) is easily calculated. For a single path P j, the probability that it is operational is p i, e i P j (4.) where p i is the probability for the link e i to be operational. Thus the probability that the path P j is failed is p i. e i P j (4.) Then, because the paths are disjoint, the probability that all paths are failed is. (4.3) j I e i P j p i Thus the probability that at least one path is operational is, (4.4) j I e i P j p i which is fast to compute. When p i are high, this can cover a significant portion of the state space. Similarly the probability that any cut C j is failed is ( p i ). (4.5) e i C j j J Now the reliability of the network is given by E[f] = P (S Ω () ) + f(x)p (s Ω () ) s Ω () = P (S Ω () ) + P (S Ω () )E[f Ω () ] (4.6) and the estimate of the reliability is Ê m [f] = P (S Ω () ) + m m P (S Ω() ) f(s i ), (4.7) where the samples s i are drawn according to the probabilities P (s Ω () ). That is, using the conditional distribution. i=

30 CHAPTER 4. VARIANCE REDUCTION TECHNIQUES The sampling from the set Ω () is not as straightforward as in CMC. The state of an edge e i which belongs to some path P j is constrained by the fact that the path must eventually fail. Thus the link states are no longer independent. The sampling algorithm is beyond the scope of this thesis. Interested reader is referred to []. The variance of this estimator is V(Êm[f]) = P (S Ω () ) V(Êm[f] Ω () ) (4.8) and when the probability P (S Ω () ) is small, the variance can be significantly less than the variance of the CMC estimator. This often implies that Ω () and Ω (3) cover a large portion of the state space and thus the state space decomposition allows deterministic evaluation of a large portion of the problem. Fishman s sampling plan performs better than the naive method only when the reliability of the individual links are high.[] High reliability leads to a favorable situation in terms of the above discussion, i.e. Ω () and Ω () cover a large portion of the state space. Performance is remarkably poor, however, when the link reliabilities are below.97.[] 4. Control Variates Method Ignore for a moment, for the sake of simplicity, the function f and consider the random variable s only. Suppose another random variable r is known, such that r and s are correlated, µ r = E[r] is known and that cov(r, s) and V(s) are easily computable. Then define t = s + b(r µ r ). (4.9) It then holds that µ t = E[s + b(r µ r )] = µ s + b(µ r µ r ) = µ s. (4.) But the variance of t is V(t) = V(s) bcov(s, r) + b V(r). (4.) Now let b = cov(s,r) V(r) and it follows that V(t) = V(s) cov(s, r) V(r) < V(s). (4.) Thus the variance is reduced. In fact, when r is always s then cov(s, r) = V(r) = V(s) and thus the variance of t is zero. Of course µ r is then µ s which was unknown in the first place and thus it is not possible to apply (4.9).

31 CHAPTER 4. VARIANCE REDUCTION TECHNIQUES In practice the control variates method can be applied as follows. When the random variable is f(s), it may be possible to find a function g which closely approximates f so that cov(g(s), f(s)) and E[g(s)] are easily computed. The correlation is highest when g is equal to f, but of course, this will not enable the easy computation of the required values. An appropriate selection of g will lead to an estimator with reduced variance. 4.3 Importance Sampling The original network reliability problem formulation leads to a discrete state space and thus a discrete random variable. It may, however, be beneficial to translate it to an equivalent problem, where the state space is continuous. This in mind, the importance sampling will now be introduced in the continuous case. The expectation value of f over distribution π is defined as E π [f] = f(s)π(s)ds. (4.3) Ω However, when the value of f is non-zero with a small probability, CMC methods require a large number of samples to produce an acceptable result in terms of relative error. An alternative is to draw samples based on an importance sampling density g, which differs from the true density. The estimate of E π [f] then becomes Ê m [f] = m m i= f(s i ) π(s i) g(s i ) (4.4) For this to be useful the fraction π(x i) g(x i ) samples from the distribution g. has to be easy to compute as well as drawing The expectation value of this new estimate is E[f]. This is readily seen because [ E g f(s i ) π(s ] i) = g(s i ) = Ω Ω f(s) π(s) g(s) g(s)ds f(s)π(s)ds = E π [f]. (4.5) The advantage of the importance sampling scheme over CMC depends heavily on the choice of the importance sampling distribution g. The optimal choice for g is g (s) = f(s)π(s), (4.6) E π [f]

32 CHAPTER 4. VARIANCE REDUCTION TECHNIQUES which leads to zero variance for the estimator. The obvious difficulty is that g depends on the scaling factor E π [f], which was unknown in the first place. In practice g is chosen from a known family of distributions to match the function f(s)π(s) as closely as possible. The distributions in the family should be easy to sample. The fundamental principle is to concentrate the sampling on the states which matter. This idea implicitly plays a role in Fishman s sampling plan as well. The states in Ω (3) all give a zero value for f. Thus they have no effect on the value of E π [f] and can be safely ignored. Instead of wasting samples on this space, the sampling is concentrated on the states that matter. 4.4 Cross-Entropy Method When the importance distribution g is selected from some family of distributions F, then a natural choice is one that minimizes the variance of the estimator. In other words which is equivalent to ( g = argmin V g f π ), (4.7) g F g [ ( g = argmin E g f π ) ]. (4.8) g F g Suppose the underlying distributions are from some parametric family F = {π(s, v)}. Then g differs from the original distribution by a single parameter v. Let the original distribution be π(s, u). Equation (4.8) then becomes where W (s, u, v) = π(s,u) π(s,v). v = argmin E v [ f W (u, v) ], (4.9) The major problem with this formulation is that the expectation value is taken with respect to the distribution which depends on the optimized parameter v. This can be avoided by change of measure: [ E v f W (u, v) ] = = Now selecting v = u yields π(s, u) f(s) π(s, v)ds π(s, v) π(s, u) π(s, u) f(s) π(s, v) π(s, v ) π(s, v )ds = E v [ f W (u, v)w (u, v ) ]. (4.) E v [ f W (u, v) ] = E u [ f W (u, v) ]. (4.)

33 CHAPTER 4. VARIANCE REDUCTION TECHNIQUES 3 The optimal choice for v is then [ v = argmin E u f W (u, v) ]. (4.) v In general there is no analytical solution to this. A workaround is to settle for solving its Monte Carlo estimate, or stochastic counterpart as it is sometimes called. That is ˆv = argmin v N N f(s i ) W (s i, u, v), (4.3) i= where s,..., s N are N identically and independently distributed samples generated from the original pdf π(s, u). Even this problem often proves to be too difficult to solve. It usually involves complicated numerical optimization. An alternative is to, instead of minimizing the variance, minimize some measure of distance to the optimal choice for g. A useful measure of distance between two densities g and h is the Kullback-Leibler distance, sometimes called cross entropy or relative entropy between g and h. It is defined as D(g, h) = E g [ g log e h ]. (4.4) It is not a true metric, since it is not symmetric. It does, however, have a number of nice qualities. It is always non-negative, zero only when g h and in case of discrete distributions has nice convexity properties. Now the distance from the optimal choice for g is D(g (s), π(s, v)) = log e (g (s)) g (s)ds log e (π(s, v)) g (s)ds. (4.5) Minimizing this with respect to v is equivalent to maximizing the last term. That is argmin D(g (s), π(s, v)) = argmax log e (π(s, v)) g (s)ds. (4.6) v v Substitute g from equation (4.6) to get argmin D(g (s), π(s, v)) = argmax v v = argmax v log e (π(s, v)) f(s)π(s, u) ds E π [f] log e (π(s, v)) f(s)π(s, u)ds = argmax E u [f(s) log e π(s, v)]. v (4.7) Again, a change of measure allows the expectation value to be taken with respect to an arbitrary parameter w. max v E u [f(s) log e π(s, v)] = max E w [f(s)w (s, u, w) log e π(s, v)]. (4.8) v

34 CHAPTER 4. VARIANCE REDUCTION TECHNIQUES 4 The estimate for the optimal parameter then becomes v = argmax E w [f(s)w (s, u, w) log e π(s, v)]. (4.9) v The optimal choice v can again be estimated with the stochastic counterpart ˆv = argmax v N N f(s i )W (s i, u, w) log e π(s i, v), (4.3) i= where s i are samples taken from the distribution π(s, w). Homem-de-Mello and Rubinstein[6] give an informal comparison of the variance minimization technique and CE-method. They point out that, in general, the solutions are not the same. The fast analytical solutions provided by the CE-method, however, make it a very useful tool. Suppose the summation term in equation (4.3) is concave and differentiable with respect to v. A function h is said to be concave if the following holds h(αx + ( α)y) αh(x) + ( α)h(y), for all x y R +, α (, ), (4.3) where R + is {x x i for all i}. If a function is concave, then its maximum value is found at the point where its gradient vanishes. This is easy to see. Suppose h(y) =. It holds that h(αx + ( α)y) = h(y + α(x y)) = h(y) + h(z) T (α(x y)) (4.3) h(y) + α(h(x) h(y)), where z [y, y + α(x y)]. It follows that h(z) T (x y) h(x) h(y). (4.33) Taking the limit as α yields h(x) h(y), for all x R +. (4.34) Thus with sufficient convexity properties, equation (4.3) can be solved by solving the following equation N N f(s i )W (s i, u, w) log e π(s i, v) =. (4.35) i= It turns out that this can often be done analytically. De Boer et. al[3] present two important cases when this is so.