Bayesian Statistics. Debdeep Pati Florida State University. October 6, 2016

Transkript

1 Bayesian Statistics Debdeep Pati Florida State University October 6, 2016

2 Collapsed / Marginal Gibbs sampler (Escober & West, 1995) For density estimation,consider the DP mixture (DPM)model y i µ i, τ i N(µ i, τ 1 i ), θ i = (µ i, τ i ) P, P DP(αP 0 )( ) Not immediate clear how to conduct posterior computation One strategy relies on marginalizing out P to obtain ( ) α i 1 1 (θ i θ 1,..., θ i 1 ) P 0 + α + i 1 α + i 1 δ θ j j=1

3 Computation of (θ 1,..., θ n ) y Computational methods like the Gibbs sampler will require the conditional distributions of θ i y, θ i. conditional distribution of θ i given (y, θ i ) is proportional to N(y i, θ i )( j i δ θ i (dθ i ) + αg 0 (dθ i )) j = j i = j i N(y i ; θ i j N(y i ; θ i j )δ θ i (dθ i ) + αn(y i ; θ i )G 0 (θ i ) j )δ θ i (dθ i ) + αn(y i, G 0 ) N(y i; θ i )G 0 (θ i ) j N(y i, G 0 ) where N(y i, G 0 ) = N(y i ; θ i )G 0 (θ i )dθ i. The normalizing constant is j i N(y i; θ i j ) + αn(y i ; G 0 ) is available in closed form.

4 Computation of f (y n+1 y) Let y = (y 1,..., y n ) and θ = (θ 1,..., θ n ). f (y n+1 y) = f (y n+1 y, θ)f (θ y)dθ 1 M M f (y n+1 y, θ (t) ) t=1 f (y n+1 y, θ) = = = f (y n+1 y, θ, θ n+1 )f (θ n+1 θ, y)dθ n+1 f (y n+1 θ n+1 )f (θ n+1 θ)dθ n+1 α N(y n+1 ; θ n+1 )G 0 (θ n+1 )dθ n+1 + n + α 1 n N(y n+1 ; θ j ) (1) n + α j=1 Draw samples from the posterior θ y and plug in (1) at each step

5 Improved Collapsed Gibbs Sampler (Bush & MacEachern, 96) Let θ = (θ1,..., θ k ) denote the unique values of θ. Let S i = h if θ i = θh denote allocation of subject i to cluster h Let k ( i) is the number of unique values in θ ( i) and n ( i) h are the corresponding counts Gibbs sampler alternates between 1. Update the allocation S = (S 1,..., S n ) by sampling from multinomial with { n ( i) h N(y i ; θh ), h = 1,..., k( i) P(S i = h ) α N(y i ; θ)dp 0 (θ), h = k ( i) Update the unique values of θ by sampling (µ h, τ, 1 h ) = N(µ h, ˆµ h, ˆκ h τ 1 h )Ga(τ h, â τh, ˆb τh ) with parameters defined as in the finite mixture model case

6 Marginal Gibbs Sampler - Some Comments Only slightly more complicated the Gibbs sampling for finite mixture models Unless further collapsing is done, the chain might be sticky and prediction is more complicated # mixture components k represented in the sample of n subjects is unknown From the MCMC samples, we can estimate posterior distribution of k As subjects are added k will increase stochastically To estimate the predictive density of y n+1 use f (y) = k h=1 n h n + α N(y; θ h ) + α n + α N(y; θ)dp 0 (θ) averaged over MCMC iterations after burn-in.

7 Clustering & Label Ambiguity via the Dirichlet Process Clustering via the Dirichlet Process If we let y i f,with f assigned the prior described above, then y i N(µ Si, τ 1 S i ), S i k π h δ h h=1 where S i is a cluster index for subject i and (µ h, τ h ) P 0 independently. (π 1,..., π k ) Dir(α/k,..., α/k) {θ h = (µ h, τ h ), h = 1,..., k} are component specific parameters.

8 Estimating component specific parameters and Label Ambiguity Note that the labels {1,..., k} are treated as exchangeable in the above mixture model There is nothing in the prior or likelihood distinguishing mixture component (cluster) h and cluster h Hence, the true marginal posterior distribution of θ h (the parameters specific to component/cluster h) will be identical for all h {1,..., k}. Each of these marginals can be expected to be multi-modal

9 Problems with Label Ambiguity & MCMC Due to the multi-modality of the posterior distributions, the Gibbs sampler described above will have a tendency to get stuck for long intervals in local modes This stickiness depends strongly on the separation between the different components If the components are widely separated, then one may obtain an apparently unimodal posterior for each θ h and the Gibbs sampling trace plots may seem well behaved For example, if k = 2 with one component close to µ = 1 and one close to µ = 1, the samples of µ 1 may remain close to 1 while the samples of µ 2 remain close to 1 Is this evidence of convergence? Are we happy with this?

10 Label switching & MCMC No! We know in advance that the marginal posteriors of every θ h are identical Hence, if we observe MCMC chains that do not converge to the same stationary distribution, then we know these chains haven t converged Is this a problem if our focus is on estimating the density & not on inferences on component-specific parameters? Seemingly not, as the modes corresponding to permutations of the label indices all correspond to the same posterior on the induced density. However, what about if we are interested in mixture-component specific inferences? i.e., we like to know where the different components are located and report this.

11 Dealing with Label Switching It is very common to simply apply standard methods of summarizing the component-specific parameters - e.g., take posterior means & 95% credible intervals for each µ h - is this a good idea? No! This is a very bad idea, because unless weve gotten lucky and are stuck in one local mode/configuration of the cluster indices, then posterior summaries are completely meaningless In fact, if we had a large number of perfect samples from the true joint posterior, then posterior summaries of µ h would be identical to those for µ h One possibility is to relabel the mixture indices after running the MCMC algorithm in a post-processing step (Stephens, 2000; Jasra et al., 2005)

12 What about putting in order restrictions? To deal with label ambiguity, another very common strategy is to put on some identifying restriction to avoid a priori exchangeability For example, we could let µ 1 < µ 2 <... < µ k - any problems with this approach? When θ h has dimension greater than one, it is typically not clear how to define an appropriate constraint For example, it may be the case that the means are the same for different components but only the variances differ Difficult to implement in general

13 Approaches to clustering There is commonly interest in clustering observations into groups Suppose we have y i R p, for i = 1,..., n, we may want to group subjects that have similar y values There is a very rich literature on clustering via distance-based methods without a likelihood specification From a Bayes perspective, model-based clustering is more natural (Banfield & Raftery, 93; Fraley & Raftery, 98)

14 Model based clustering Let y i k h=1 π hk(y; θ h ), for some parametric kernel K (typically Gaussian), for i = 1,..., n. The n subjects allocated to at most k clusters, with each mixture component corresponding to a different cluster Suppose we fit the finite mixture model using the EM algorithm to obtain an MLE ˆπ h, ˆθ h, h = 1,..., k, with k the number of components estimated using BIC Conditionally on the estimated parameters, we obtain P(S i = h y i, ˆπ, ˆθ) = ˆπ hk(y i ; ˆθ h ) k l=1 ˆπ lk(y i ; ˆθ l ) with the optimal allocation corresponding to the h that maximizes these probs

15 Clustering - Comments Allocating all the subjects to clusters in this manner, we obtain a partition of {1,..., n} into k n k clusters The index on the different clusters is not important - the grouping of the subjects is the focus Note that the choice of kernel K can have a big impact on the estimated number of clusters & the allocation to clusters In fact, the definition of a cluster is inherently determined entirely by the kernel - if we have a flexible enough kernel, then subjects can always be allocated to a single cluster

16 Pitfalls & Limitations of Clustering From a statistical perspective, new clusters are introduced to accommodate lack of fit in the parametric model K( ). Clearly this is hugely sensitive to K & it is not clear that clusters obtained from a statistical procedure correspond to scientifically meaningful clusters Scientifically, clusters are often viewed as corresponding to different mode in a multi-modal distribution, with clusters well defined if these modes are well separated Each mixture component does not correspond to a different mode - the relationship between the number of components, the component-specific parameters & the number of modes is complex even for multivariate normal distributions (Ray & Lindsay, 05)

17 Robust Clustering Even focusing on multivariate normals, the clusters can be sensitive to parameterization of the covariance Clustering based on normals with diagonal covariance may lead to too many clusters - from the viewpoint of sparsity of modeling & scientific interpretability of the clusters Li, Ray & Lindsay (07, JMLR) propose an approach for clustering via mode identification using kernel density estimation & a modal EM algorithm Would be interesting to develop a np Bayes version of their approach - e.g., modeling K h (the kernel specific to component h) as an unknown unimodal density

18 How to Estimate Clusters from the MCMC Draws? Medvedovic & Sivaganesan (2002) propose to apply standard clustering methods (e.g., hierarchical agglomerative clustering) to a distance matrix obtained using the posterior probabilities of pairwise clustering Dahl (2006) proposes a simple approach to obtain a clustering estimate based on the MCMC output using least squares distances from the posterior probability that two subjects are clustered

19 Modal Clustering Note that each MCMC iteration produces one clustering One possibility is to estimate the clustering probabilities as the proportion of samples in which that clustering is drawn, and then use the MAP as the optimal clustering under 0-1 loss # possible clusterings in n subjects grows exponentially via Bell number (e.g., > for n = 200) Hence, it is very difficult to get accurate estimates of the posterior clustering probabilities & the MAP will have a low posterior probability anyway

20 Dahl (2006) Cluster Estimation Method Dahl (2006) proposed a useful alternative to ad hoc clustering based on the MCMC results & MAP Let ˆπ = {ˆπ ij } denote the n n matrix with elements corresponding to the estimated pairwise posterior probabilities of clustering subjects iand j Dahl proposes to choose the least-squares clustering c LS c LS = argmin c {c1,...,c B } n i=1 j=1 n (δ ij (c) ˆπ ij ) 2 where δ ij (c) = 1 if subjects i and j are in the same cluster under clustering c & 0 otherwise We just calculate the least squares distance for each MCMC iteration & choose the best of these iterations

21 Zhang et al (2014+) Cluster Estimation Method Let F B denote the space of all membership matrices, as a subset of symmetric n n matrices with restrictions: (1) B(i, j) = {0, 1} for all i, j = 1,..., n; (2) B(i, ) = B(j, ) and B(, i) = B(, j) if i-th observation and j-th observation are in the same cluster. Obtain posterior samples {B (i), i = 1,..., M} The final matrix B is obtained by calculating the extrinsic mean of the posterior samples defined as follows: Find the mode of the number of clusters k 0 based on the samples B (1),..., B (M). Calculate the Euclidean mean and project it onto the membership matrix space: 1. Euclidean mean: let B = 1 M M t=1 B(t). 2. Projection: Project the Euclidean mean onto the space of membership matrix by a thresholding operation B = threshold( B, t ) where t is the largest threshold such that B has k 0 clusters.