Variational Upper Bounds for Probabilistic Phylogenetic Models

Transkript

1 Varatonal Upper Bounds for Probablstc Phylogenetc Models Ydo Wexler and Dan Geger Dept. of Computer Scence, Technon - Israel Insttute of Technology, Hafa 32000, Israel {ywex,dang}@cs.technon.ac.l Abstract. Probablstc phylogenetc models whch relax the ste ndependence evoluton assumpton often face the problem of nfeasble lkelhood computatons, for example for the task of selectng sutable parameters for the model. We present a new approxmaton method, applcable for a wde range of probablstc models, whch guarantees to upper bound the true lkelhood of data, and apply t to the problem of probablstc phylogenetc models. The new method s complementary to known varatonal methods that lower bound the lkelhood, and t uses smlar methods to optmze the bounds from above and below. We appled our method to algned DNA sequences of varous lengths from human n the regon of the CFTR gene and homologous from eght mammals, and found the upper bounds to be apprecably close to the true lkelhood whenever t could be computed. When computng the exact lkelhood was not feasble, we demonstrated the proxmty of the upper and lower varatonal bounds, mplyng a tght approxmaton of the lkelhood. 1 Introducton Most organsms share a great deal of ther genetc code wth other forms of lfe. Phylogenetc tree models are used to assocate the genetc makeup of dfferent organsms accordng to ther genetc varaton. A node on phylogenetc trees corresponds to a pece of genetc code n a sngle organsm, and the branches and the relatve branch lengths measure the relatve dstance from each organsms genes to the others. The greater the dstance, the more the gene sequence has changed between one organsm and the other. The classcal phylogenetc models of Neyman (1971) and Felsensten (1981) make several assumptons regardng how evoluton occurs n the trees, from whch the most strngent assumpton s that evoluton takes place ndependently n dfferent stes. Over the years more complex probablstc phylogenetc models have been proposed, whch relax the ste ndependence evoluton assumpton. These complex models that are more bologcally realstc, such as the one by Sepel and Haussler (2003), often face the problem of nfeasble lkelhood computatons, for example for the task of selectng sutable parameters for the model. To overcome ths problem Jojc et al. (2004) suggested to use varatonal approxmatons that lower bound the lkelhood of data, and showed that such bounds tend to be close to the true lkelhood. In ths paper, we develop tght upper bounds on the lkelhood of a gven data, that are close to lower bounds so that good estmates of the lkelhood become avalable. Correspondng author. T. Speed and H. Huang (Eds.): RECOMB 2007, LNBI 4453, pp , c Sprnger-Verlag Berln Hedelberg 2007

2 Varatonal Upper Bounds for Probablstc Phylogenetc Models 227 Our new approxmaton method s applcable for a wde range of probablstc models, ncludng the dscussed phylogenetc models. The method assumes a smple dstrbuton Q whch approxmates the target dstrbuton P of the model, and usng Jensen s nequalty t upper bounds the lkelhood of data wth a functon of Q and P.Thesmplcty of Q yelds a bound that can be computed effcently. Our method s complementary to known varatonal methods that lower bound the lkelhood (e.g. Jordan et al., 1999), and can use an approxmatng dstrbuton Q suggested by these methods to bound the lkelhood also from above. We appled our method to algned DNA sequences of varous lengths from human n the regon of the CFTR gene and homologous from eght mammals, and found the upper bounds to be apprecably close to the true lkelhood whenever t could be computed. When computng the exact lkelhood was not feasble, we demonstrated the proxmty of the upper and lower varatonal bounds, mplyng a tght approxmaton of the lkelhood. The rest of the paper s organzed as follows: Secton 2 brefly descrbes phylogenetc HMM models n terms of Bayesan networks or DAG models, and provdes a quck overvew regardng varatonal technques that lower bound the lkelhood of data. Secton 3 develops our man contrbuton whch are varatonal upper bounds for probablstc models such as Bayesan networks. The expermental results are descrbed n Secton 4. Fnally, we dscuss the lmtatons of varatonal methods. 2 Prelmnares We provde background nformaton regardng phylogenetc HMM trees, to whch the varatonal upper bounds suggested heren are appled (Secton 2.1), and outlne known varatonal lower bounds of the lkelhood of data, whch turn out to be close to our upper bounds (Secton 2.2). 2.1 Phylogenetc HMM Model We consder the Phylogenetc HMM model descrbed by Sepel and Haussler (2003). Snce the model s gven n terms of condtonal probabltes, t s convenent to descrbe t as a DAG model, as done by Jojc et al. (2004). We repeat the descrpton of the model from there wth mnor changes. Gven a doman of nterest havng a set of fnte varables s =(s 1,...,s n ) wth a postve jont dstrbuton p(s), a DAG model for s s a par (G, P ) where G s a drected acyclc graph and P s a set of condtonal probablty dstrbutons. A DAG model s also often called a Bayesan network (e.g. Pearl 1988, Jensen 2001). Each node s n G corresponds to a varable n s, and to a dstrbuton p(s pa(s )), called a local probablty dstrbuton, where pa(s ) are the parents of s n the graph. The jont dstrbuton s gven by p(s) = n =1 p(s pa(s )). Consequently, the assumed ndependence relatonshps between random varables are represented through absence of edges n the model. A DAG model structure that assumes that evoluton takes place ndependently at each nucleotde ste s llustrated n Fgure 1a for a smple tree wth fve speces. The

3 228 Y. Wexler and D. Geger (a) (b) Fg. 1. Probablstc phylogenetc trees expressed as DAG models. (a) The Neyman-Felsensten tree model that assumes ndependent evoluton n stes. (b) The dnucleotde phylogenetc HMM model suggested by Sepel and Haussler (2003). unknown nucleotde n an ancestor speces at ste j s denoted as h j, and the observed nucleotde of an exstng speces at ste j s denoted as yj. Ths s the usual model for whch Felsensten s algorthm for computng lkelhood of data s readly applcable. The model of Sepel and Haussler (2003) does not assume that stes are ndependent, and therefore, edges that connect varables of adjacent stes are added (Fgure 1b). Ths fgure llustrates the phylogenetc HMM model of Sepel and Haussler (2003). In ths model, a nucleotde of speces at ste j depends on the nucleotde of that speces at ste j 1, and ts ancestor s nucleotdes at stes j 1 and j. Ths model s also called the dnucleotde HMM model, snce the two nucleotdes of speces and k at ste j,wherek s the ancestor speces of, are dependent only on the two nucleotdes of that speces at ste j 1. AddtonalmorecomplexmodelsaredscussednSepelandHaussler (2003). The local probablty dstrbutons of ths model are determned by a contnuous-tme Markov matrx Q of base substtuton rates. The matrx Q s of sze 16 16, and gven evolutonary tme t, whch s the branch length n the tree, the condtonal probabltes p(s j,s j 1 sk j,sk j 1 ) are obtaned from Q, wherek s the ancestor speces of. Ths dstrbuton then determnes the desred probabltes p(s j s j 1,sk j,sk j 1 ).LetP(t) be the matrx of substtuton probabltes for branch length t. ThenP (t) sgvenbythe soluton to the dfferental equaton d dtp (t) =P (t)q wth ntal condtons P (0) = I, whch s P (t) =e Qt. Wth Q beng dagonalzable as Q = SΛS 1, the matrx P (t) can be computed as P (t) =Se Λt S 1,wheree Λt s the dagonal matrx obtaned by exponentatng each element on the man dagonal of Λt. A standard crteron to choose between two DAG models s to prefer a model wth hgher log-lkelhood of the data. However, for the phylogenetc HMM model descrbed here, computng the log-lkelhood of data s not feasble, and therefore approxmatons are needed. In the next secton we revew known approxmatons that gve lower bounds.

4 Varatonal Upper Bounds for Probablstc Phylogenetc Models Varatonal Lower Bounds The problem of computng the lkelhood, P (Y = y) = h P (Y = y, H = h)), n DAG models s NP-hard (Cooper, 1990; Dagum & Luby, 1993), and although there are many DAG models where exact algorthms are feasble, there are others n whch the tme and space complexty makes the use of such algorthms nfeasble. In these cases fast yet accurate approxmatons are desred. Heren, we call the task of computng the lkelhood by the term nference. Varatonal technques such as the ones suggested by Jordan et al. (1999) are a powerful tool for effcent approxmate nference that offers guarantees n the form of lower bounds. In partcular, let P (X) be a jont dstrbuton over a set of dscrete varables X wth the goal to compute the margnal probablty P (Y = y), wherey X. Further assume that ths exact computaton s not feasble. The dea s to replace P wth a dstrbuton Q for whch exact nference s feasble, and compute a lower bound for P (Y = y) by usng Jensen s nequalty: log P (y)= log P (y, h) Q(h) P (y, h) Q(h)log Q(h) Q(h) = D(Q(H) P (Y =y, H)) h h where H = X \ Y and D( ) denotes the KL dvergence between two probablty dstrbutons. To obtan tght lower bounds several varatonal algorthms were devsed that try to fnd an approxmatng dstrbuton Q whch mnmzes the KL dvergence between Q and the target dstrbuton P ( [15,8,17,1,7]). Varatonal approaches such as the mean feld, generalzed mean feld, and structured mean feld dffer only wth respect to the famly of approxmatng dstrbutons that can be used. Such varatonal technques were appled by Jojc et al. (2004) to fnd lower bounds for the phylogenetc HMM models. The lower bounds computed n the results secton heren use a newer algorthm for fndng tghter lower bounds suggested by Geger et al. (2006). 3 Varatonal Upper Bounds We denote dstrbutons by P (x) and Q(x), whereq s not necessarly a normalzed dstrbuton. Let X be a set of varables and x be an nstantaton of these varables. Let P (x) = n =1 Ψ (d ) and Q(x) = n =1 Φ (d ) where d s the projecton of the nstantaton x to the varables n D X, the subsets {D } n =1 can overlap, and n s the number of sets D. Consder the margnal probablty P (Y = y) = h P (y, h) = h Ψ (d ) where X = Y H. We assume throughout that Q(x) s tractable n the sense that the margnal probablty Q(Y = y) s feasble to compute, whle P (Y = y) s not feasble to compute. We now develop an upper bound for P (Y = y) as summarzed n Theorems 1 & 2. Accordng Jensen s nequalty, f f s a concave functon and Z = {z 1,...,z n } s a set of real numbers then f( n =1 w z ) n =1 w f(z ), where each w 0 and n =1 w =1. By usng the concavty of the log functon and Jensen s nequalty for concave functons, we get the followng upper bound: P (Y = y) = h e log Ψ(d) = h e w(h)logψ(d)(1/w (h)) (1)

5 230 Y. Wexler and D. Geger h e log w(h)ψ(d)(1/w (h)) = h w (h)ψ (d ) (1/w(h)) where w (h) =1for every nstantaton h. Note that ths bound can be obtaned also by usng the weghted power means nequalty 1. Eq. 1 holds wth equalty regardless of the values of potentals Ψ f and only f w (h) = log Ψ (d ) log P (h, y). (2) Gven a tractable dstrbuton Q(x) = n =1 Φ (d ) we set w (h) = approxmates the optmal but ntractable choce gven by Eq. 2. Wth these values for w (h), and usng the dentty x logy z wrtten as: P (Y = y) h log Φ (d ) k log Φ k(d k ) m log Φ(d) log Q(h,y),whch = y logx z, Eq. 1 can be Φ m (d m ) log Ψ (d ) log Φ (d ) (3) The upper bound n Eq. 3 holds wth equalty f Q equals P, because by replacng all occurrences of Φ (d ) wth Ψ (d ) we get P (Y = y) h log Ψ (d ) k log Ψ k(d k ) Ψ m (d m )= m h Ψ m (d m )=P(Y = y) m Eq. 3 remans hard to compute untl the sum over h s dvded nto smaller sums. To obtan a tractable bound we use the arthmetc-geometrc means nequalty, k log Φ k(d k ) k log Φ k(d k ) 1/n,wherelog Φ k (d k ) > 0. To use ths nequalty 1 n we set all potentals Φ (d ) to be greater than 1. The resultng tractable upper bound stemmng from Eq. 3 s the followng: P (Y = y) 1 n h =1 n log Φ (d ) m Φ m (d m ) log Ψ (d ) log Φ (d ) log Φ m (d m ) 1/n (4) Consequently, the followng theorem holds. 1 The weghted power mean Mw(Z) r of a seres of real numbers Z = {z 1,...,z n} s defned for every real r R as Mw(z r 1,...,z n)= [ n ] 1/r =1 wzr f r 0 n =1 zw f r =0 where w 1,...,w n are postve real numbers such that n =1 w =1. Note that M w(z) r r 0 Mw(Z). 0 The power mean nequalty states that for two real numbers s, t, the relaton s<tmples Mw s < Mw, t and the upper bounds are obtaned by settng s = 0, t = 1,and z = Ψ (d ) (1/w).

6 Varatonal Upper Bounds for Probablstc Phylogenetc Models 231 Theorem 1 (upper bound). Let H and Y be two dsjont sets of varables such that H Y = X, and let P (x) and Q(x) be dstrbutons that factor accordng to P (x) = n =1 Ψ (d ) and Q(x) = n =1 Φ (d ) where d s the projecton of the nstantaton x to the varables n D X. Then the followng s an upper bound on P (Y = y), P (Y = y) 1 log Φ (d ) n D h\d m Φ m (d m ) log Ψ (d ) log Φ (d ) log Φ m (d m ) 1/n (5) Proof: The proof s mmedate from Eq. 4 where we replace the sums over and h,and dvde the sum over h such that frst we sum over varables n D and then over the rest of the varables n H. Assumng that M =max { D } s at most a gven constant, the tme needed to compute the bound gven n Eq. 5 s lnear n the number of varables n the model and proportonal to the tme needed to compute Q(y). Therefore, the tractablty of ths bound s a drect consequence of the assumpton of tractable nference on dstrbuton Q. Snce the maxmal sze M of the sets n the model can sometme be large enough to sgnfcantly slow computatons of the upper bound, we develop a more effcent method to compute the upper bound that does not depend on M. To do so, we use the followng lemma. Lemma 1. Gven two sets of postve real numbers X = {x 1...,x n } and Y = {y 1...,y n } and a postve real number r, the followng nequaltes hold. If 0 <r 1,then If 1 r<2,then n =1 x r y =1 ) r ) 1 r x y 1. y =1 n =1 =1 x r y =1 ) 2 r x y =1 =1 x 2 y ) r 1. For r =1equaltes hold. Proof: We use the Eucldeancase of Hölder s nequalty, statng that for two sets of postve real numbers X = {x 1...,x n } and Y = {y 1...,y n }, and for two real numbers p, q 1 such that 1 p + 1 q =1, ( n n ) 1/p 1/q n x y x p. For 0 <r 1,wegetusngHölder s nequalty, n =1 x r y = n =1 ( x y ) r y r 1 =1 ( x y y q j j=1 ) r p ) 1/p =1 y (r 1) q ) 1/q.

7 232 Y. Wexler and D. Geger n =1 Settng p = 1 r and q = 1 1 r we get n =1 x r y =1 ) r ) 1 r x y 1. y =1 Smlarly, for1 r<2,weget usnghölder s nequalty, x r y = n =1 ( x y ) ( ) x 2 r 1 2 r y =1 Settng p = 1 2 r and q = 1 r 1 we get ( x y ) ) ( (2 r) p n ( ) x 2 (r 1) q ) 1/q 1/p. =1 y n =1 x r y =1 ) 2 r x y =1 x 2 y ) r 1. Theorem 2 (Effcent upper bound). Let H and Y be two dsjont sets of varables such that H Y = X, and let P (x) and Q(x) be dstrbutons that factor accordng to P (x) = n =1 Ψ (d ) and Q(x) = n =1 Φ log Ψ (d ) where Ψ > 1, Φ > 1 and log Φ < 2 for every = 1,...,n, and where d s the projecton of the nstantaton x to the varables n D X. In addton, let U denote the set of nstantatons of D for whch Φ (d ) Ψ (d ), and let L denote the rest of nstantatons of D. Then the followng s an upper bound on P (Y = y), [ ] P (Y = y) 1 log Φ (d )Λ L + log Φ (d )Λ U (6) n d L d U where Λ L = Φ m (d m ) log Φ h\d m m (d m ) 1/n and Λ U = Φ m (d m ) log Φ h\d m m (d m ) 1/n log Ψ (d ) log Φ (d ) 2 log Ψ (d ) log Φ (d ) 1 log Φ h\d m m (d m ) 1/n Φ m (d m ) 2 log Φ h\d m m (d m ) 1/n 1 log Ψ (d ) log Φ (d ) log Ψ (d ) log Φ (d ) 1 Proof: Lemma 1 mples that when Φ (d ) Ψ (d ) > 1, we can replace every bracketed term [ ] h\d m Φ m (d m ) log Ψ (d ) log Φ (d ) / log Φ m (d m ) 1/n n Eq. 5 wth Λ L and when 1 <Φ (d ) <Ψ (d ), we can replace t wth Λ U,snce log Ψ(d) log Φ (d ) < 2. Computng each term, Λ U or Λ L, nvolves only two sums of products, where each sum factors accordng to dstrbuton Q. These computatons can be performed by usng any

8 Varatonal Upper Bounds for Probablstc Phylogenetc Models 233 algorthm such as bucket elmnaton algorthm or the sum-productalgorthm descrbed by Dechter (1999) and Kschschang, Frey & Loelger (2001). Accordng to Eq. 6 only a lnear number of calls to such procedures are needed to obtan the upper bound. log Ψ log Φ If each potental Ψ and Φ s multpled by a large factor α, all the terms approachoneas α grows. Ths reducesthe accuracygapwhenusnghölder s nequalty log Ψ n Eq. 6 wth r = log Φ. In addton, note that multplyng the potentals Φ by α also serves the tghtness of the arthmetc-geometrc nequalty used to obtan Eq. 5, snce log Φj for each par of potentals Φ j and Φ k, the rato log Φ k approaches one as α grows. A log Ψ large enough α guarantees that log Φ < 2 for all sets D and thus the applcablty of Theorem 2. In our experments we use ln α = Approxmatons for Phylogenetc HMM Models The dnucleotde phylogenetc HMM model of Sepel and Haussler (2003), descrbed n Secton 2.1, lead to mprovements over prevous models n several bologcal tasks such as gene fndng. But, despte ts enhanced power, t also requres evaluatng an ntractable lkelhood for the purpose of fndng optmal parameters for the model. Jojc et al. (2004) used varatonal technques, smlar to the ones descrbed n Secton 2.2 to lower bound the lkelhood of data, and showed that when the exact lkelhood can be computed (although wth much effort), the approxmatons were tght. We use the upper bounds suggested n Secton 3 to compute the lkelhood of phylogenetc trees wth a small error, by boundng t tghtly from above and below. Frst, we show the upper bounds are close to the true lkelhood when ths can be computed. Then, for larger phylogenetc trees, where computng the exact lkelhood s nfeasble, we show the proxmty of the lower and upper bounds. To set a tractable approxmatng dstrbuton Q, we use a parameter k whch determnes ts topology: sets that contan varables from stes ck and ck +1,forc =1, 2, 3,..., are splt nto two dsjont subsets, D 1 and D 2,whereD 1 contans only varables n D from ste ck and D 2 contans the rest of the varables n D. Ther respectve potentals Φ (d ) therefore factor accordng to Φ (d )=Φ 1 (d 1 )Φ 2 (d 2 ). In our experments we used k =10when computng the exact lkelhood was feasble and k =5when the lkelhood computaton was nfeasble. The lower bounds were obtaned by usng a recent varatonal algorthm called VIP* (Geger et al., 2006). We repeat each upper bound computaton twce, wth the dfference of the way potentals Φ are chosen. The frst choce s what we call non-nformatve (NI), where each potental Φ (d ) = m j=1 Φ j(d j ) s a product of m sub-potentals of sets D j D. A sub-potental Φ j (d j ) s set to be the 1/m power of the average value of Ψ (d ) of all nstantatons d consstent wth d j. More formally, Φ j (d j ) ( 1 1/m = C dj d C dj Ψ(d )) where Cdj s the set of nstantatons d consstent wth d j. The second choce of potentals, called varatonal-based (VB), s based on varatonal algorthms,such as VIP*, that optmze the approxmatng dstrbuton Q n order

9 234 Y. Wexler and D. Geger to set tght lower bounds on the lkelhood. If the topology of Q gven for these algorthms follows the factorzaton suggested n Secton 3 (.e. every potental Ψ n P has ts correspondng potental Φ n Q), the potentals found by these optmzaton algorthms to lower bound the lkelhood can also serve to upper bound t usng the method proposed heren. We ran the tests on data used by Sepel and Haussler (2003) that contans sequences from human n the regon of the CFTR gene and homologous from eght mammals: chmp, baboon, cow, pg, cat, dog, mouse and rat. The sequences are algned, and we used portons of ths algnment to obtan our results. The substtuton probabltes n all models were computed from the dnucleotde substtuton matrx obtaned by Jojc et al. (2004), and the branch lengths n each tree were randomly chosen, normally dstrbuted around predetermned means. The frst tests used two data sets, smlar to those used by Jojc et al. (2004), where each set conssted of three sequences. The sequences n set A were taken from the cow, mouse and human genomes and were of length 30Knc, and the sequences n set B were taken from the cow, pg and dog genomes and were of length 20Knc. Fgure 2a and 2b plot the upper bounds versus the exact log-lkelhoods of trees wth dfferent branch lengths. Lower bounds are also shown n the fgure to demonstrate the tghtness level of these bounds. The average dfferences for the trees n set A between the upper bounds and the exact lkelhoods were 1% for the NI method Fg. 2. Upper and lower bounds on the lkelhood of data of phylogenetc HMM models for sets A, B and C wth dfferent branch lengths. (a) & (b) Bounds versus the exact lkelhood for models of sets A and B. (c) Bounds for models of set C, for whch computng the exact lkelhood s nfeasble.

10 Varatonal Upper Bounds for Probablstc Phylogenetc Models 235 (a) (b) Fg. 3. Accuracy and run-tme as a functon of parameter k of decomposng the model. (a) Accuracy as a functon of k. (b) Run-tme as a functon of k. Fg. 4. The dfference n accuracy between upper bounds computed va Eq. 5 and bounds computed va Eq. 6 and 0.95% for the VB method, and for trees n set B the average dfferences were 0.97% (NI) and 0.9% (VB). The upper and lower bounds for an addtonal set of algned sequences that contaned sequences of length 30Knc from all nne organsms (Set C) are llustrated n Fgure 2c. For ths set t s nfeasble to compute the exact lkelhood, but the proxmty of the upper and lower bounds allows us to predct the lkelhood wth a small error. The NI method yelded an average of 1.64% dfference from the lower bounds and the VB method yelded an average of 1.52% from the lower bounds for the models n ths set. As shown n Fgure 2, both choces of potentals (NI and VB) performed smlarly, wth a small advantage of the VB method over NI n most experments. In other experments we performed, we found that arbtrary choce of potentals often lead to sgnfcant decrease n the tghtness of the bounds (up to 45%), and therefore an algorthm s desred to fnd potentals that lead to tght bounds. The parameter k used for decomposng the tree model nto parts of k stes s a tradeoff between run-tme and accuracy: the larger k s the more tme consumng t s to compute the upper bounds, however, the bounds computed are also more accurate. The

11 236 Y. Wexler and D. Geger default value of k was set to 10 for trees n Set A. Fgure 3 shows the results for these trees as a functon of k n terms of accuracy and n terms of run-tme. Fnally, we tested the dfference n accuracy between upper bounds computed va Eq. 5 and those computed va Eq. 6. The expected run-tme rato between these two methods s the average probablty table sze n the model. Snce no preprocessng such as summng over some varables was executed, the expected rato was Asshown n Fgure 4, the dfferences n accuracy of the upper bounds were neglgble, less than 0.05% of ther log value, when appled to phylogenetc trees n data set A. Ths mples that when the sze of the probablty tables s large, Eq. 6 s an attractve and effcent alternatve to Eq Dscusson Computng the lkelhood of many probablstc models s nfeasble and calls for effcent approxmatons. Our results on phylogenetc models show that the suggested upper bounds are apprecably tght and together wth other varatonal methods allow to compute the lkelhood almost exactly n feasble tme. We have also started usng the upper bounds to approxmate other probablstc models and beleve that they can be appled to a wde range of models and for varous tasks. One addtonal task we explore s boundng the MAP assgnment probablty n order to set optmal parameters for models where fndng the exact MAP assgnment s nfeasble. The goodness of the bounds heavly depends on the choce of an approxmatng dstrbuton Q, andmore work on choosng useful Q functons s desred, as ndcated by Xng et al. (2004). As wth varatonal methods that offer lower bounds on the lkelhood, f the dependence of varables under Q largely dffers from ther dependence under the target dstrbuton P, these methods yeld loose bounds. When explorng probablstc models to genetc lnkage analyss, as used by Fshlson and Geger (2002), we found that the varatonal methods we used dd not offer suffcently good approxmatng dstrbutons for these models, and therefore dd not gve tght enough bounds. Geger et al. (2006) provded results of varatonal technques on genetc lnkage analyss problems and showed that although the lower bounds followed the shape of the lkelhood functon, the dfference from the true log-lkelhood reached 20%. The dffculty n fndng good approxmatons to ths model may le n the level of determnsm of the model: relaxng determnstc dependence relatonshps between varables reduced accuracy far more than when relaxng mld dependence relatonshps. When computng the upper bounds suggested heren for genetc lnkage analyss, the results were wthn 10% from the true log-lkelhood. Ackowledgements The research s supported by the Israel Scence Foundaton and the Israel Scence Mnstry.

12 Varatonal Upper Bounds for Probablstc Phylogenetc Models 237 References 1. C. Bshop and J. Wnn. Structured varatonal dstrbutons n VIBES. In Artfcal Intellgence and Statstcs, Key West, Florda, USA, Socety for Artfcal Intellgence and Statstcs. 2. G. Cooper. Probablstc nference usng belef networks s NP-hard. Artfcal Intellgence, 42: , P. Dagum and M. Luby. Approxmatng probablstc nference n Bayesan belef networks s NP-hard. Artfcal Intellgence, 60(1): , R. Dechter. Bucket elmnaton: A unfyng framework for reasonng. Artfcal Intellgence, 113(1-2):41 85, J. Felsensten. Evolutonary trees from DNA sequences: a maxmum lkelhood approach. Journal of Molecular Evoluton, 17: , M. Fshelson and D. Geger. Exact genetc lnkage computatons for general pedgrees. Bonformatcs, 18:S189 S198, D. Geger, C. Meek, and Y. Wexler. A varatonal nference procedure allowng nternal structure for overlappng clusters and determnstc constrants. Journal of Artfcal Intellgence Research (JAIR), 27:1 23, Z. Ghahraman and M. I. Jordan. Factoral hdden Markov models. Machne Learnng, 29: , F. V. Jensen. Bayesan Networks and Decson Graphs. Sprnger-Verlag New York, Inc., V. Jojc, N. Jojc, C. Meek, D. Geger, A. Sepel, D. Haussler, and D. Heckerman. Effcent approxmatons for learnng phylogenetc HMM models from data. Bonformatcs, 20: , M. I. Jordan, Z. Ghahraman, T. S. Jaakkola, and S. L. K. An ntroducton to varatonal methods for graphcal models. Learnng Graphcal Models. MIT Press, F. R. Kschschang, B. J. Frey, and H. A. Loelger. Factor graphs and the sum-product algorthm. IEEE Transactons on nformaton theory, 47(2): , February J. Neyman. Molecular studes of evoluton: a source of novel statstcal problems. In S. S. Gupta and J. Yackel (eds), Statstcal descon theory and related topcs, pages Academc Press, New York, J. Pearl. Probablstc reasonng n ntellgent systems: networks of plausble nference. Morgan Kaufmann, San Mateo, CA, L. K. Saul and M. I. Jordan. Explotng tractable substructures n ntractable networks. In D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo, edtors, Advances n Neural Informaton Processng Systems (NIPS), volume 8, pages The MIT Press, A. Sepel and D. Haussler. Combnng phylogenetc and hdden markov models n bosequence analyss. In RECOMB 03: Proceedngs of the seventh annual nternatonal conference on Research n computatonal molecular bology, pages , New York, NY, USA, ACM Press. 17. W. Wegernck. Varatonal approxmatons between mean feld theory and the juncton tree algorthm. In UAI 00: Proceedngs of the 16th Conference on Uncertanty n Artfcal Intellgence, pages , San Francsco, CA, USA, Morgan Kaufmann Publshers Inc. 18. E. P. Xng, M. I. Jordan, and S. Russell. Graph partton strateges for generalzed mean feld nference. In AUAI 04: Proceedngs of the 20th conference on Uncertanty n artfcal ntellgence, pages , Arlngton, Vrgna, Unted States, AUAI Press.