Let us give a simple example. Suppose we use the crudest selection method, namely we blindly pick |S| members of the N samples and ask the probability that among these |S| selected designs there are at least k designs that will rank in the top-g of the performance order (recall the engineering design example in the earlier transparency). Blind Pick (BP) is equivalent to a selection method where the error variance of selection is infinity. It turns out this probability can be calculated in closed form. There are gCi ways of picking i elements out of the top-g designs and N-gCg-i ways picking the remaining (g-i) designs out the N-g samples; this is divided by the total number of ways of selecting g designs out of N samples NCg to arrive at the alignment probability. The curves shown are for N = 1000. As we can see for k=1, P ~ 1 for size of |G|=g=50, i.e., without any knowledge we can effect a 20:1 reduction and still be sure that at least one of the top-50 design is in the blindly picked subset.

For sigma2, we no longer have a closed form answer. And we need a model to calculate the alignment probability. For example, let us assume that

Performanceobserved = Performanceactual + error .................... (*)

which is known as the Thurstone Model. We shall take the observed top-|S| performances as the selected subset S and compute experimentally via simple simulation using Eq.(*) the alignment probability. However, it is also clear that this probability will be problem-dependent. We need to introduce some other concepts before we can characterizing this probability.