Chapter 5: Least-Square Methods for System Identification

Transkript

1 Chapter 5: Least-Square Methods for Syste Identfcaton Syste Identfcaton: an Introducton (5.) Least-Squares Estators (5.3) Statstcal Propertes & the Maxu Lkelhood Estator (5.7) LSE for Nonlnear Models (5.8) Jyh-Shng Roger Jang et al., Neuro-Fuzzy and Soft Coputng: A Coputatonal Approach to Learnng and Machne Intellgence, Frst Edton, Prentce Hall, 997 Syste Identfcaton: Introducton (5.) Goal Deterne a atheatcal odel for an unknown syste (or target syste) by observng ts nput-output data pars Purposes o predct a syste s behavor, as n te seres predcton & weather forecastng o explan the nteractons & relatonshps between nputs & outputs of a syste Ch. 5: Least-squares estators

2 Syste Identfcaton: Introducton (5.) (cont.) Purposes (cont.) o desgn a controller based on the odel of a syste, as an arcraft or shp control Sulate the syste under control once the odel s known Syste Identfcaton: Introducton (5.) (cont.) here are an steps that are nvolved Structure dentfcaton Paraeter dentfcaton Ch. 5: Least-squares estators

3 Syste Identfcaton: Introducton (5.) (cont.) Structure dentfcaton Apply a-pror knowledge about the target syste to deterne a class of odels wthn whch the search for the ost sutable odel s to be conducted; ths class of odel s denoted by a functon y = f(u,θ) where: y s the odel output u s the nput vector Is the paraeter vector f depends on the proble at hand & on the desgner s experence & the laws of nature governng the target syste Syste Identfcaton: Introducton (5.) (cont.) Paraeter dentfcaton he structure of the odel s known, however we need to apply optzaton technques n order to deterne the paraeter vector θ = θˆ such that the resultng odel ŷ = f(u, θˆ ) descrbes the syste approprately: y ŷ 0 wth y assgned to u Ch. 5: Least-squares estators 3

4 Syste Identfcaton: Introducton (5.) (cont.) Block dagra for paraeter dentfcaton Syste Identfcaton: Introducton (5.) (cont.) he data set coposed of desred nput-output pars (u;y) ( =,,) s called the tranng data Syste dentfcaton needs to do both structure & paraeter dentfcaton repeatedly untl satsfactory odel s found: t does ths as follows: Specfy & paraeterze a class of atheatcal odels representng the syste to be dentfed Perfor paraeter dentfcaton to choose the paraeters that best ft the tranng data set Conduct valdaton set to see f the odel dentfed responds correctly to an unseen data set ernate the procedure once the results of the valdaton test are satsfactory. Otherwse, another class of odel s selected & repeat step to 4 Ch. 5: Least-squares estators 4

5 Least-Squares Estators (5.3) General for: y = θ f (u) + θ f (u) + + θ n f n (u) (*) where: u = (u,, u p ) s the odel nput vector f,, f n are known functons of u θ,, θ n are unknown paraeters to be estated Least-Squares Estators (5.3) (cont.) he task of fttng data usng a lnear odel s referred to as lnear regresson We collect a tranng data set {(u;y), =,, } Equaton (*) becoes: f(u ) θ + f(u) θ fn (u) θn = y f(u ) θ + f(u ) θ fn (u ) θn = y M f(u ) θ + f(u ) θ fn (u ) θn = y Whch s equvalent to: A θ = y Ch. 5: Least-squares estators 5

6 Least-Squares Estators (5.3) (cont.) Where: A s an *n atrx whch s: θ s n* unknown paraeter vector: and y s an * output vector: y y = M y and a A θ = y θ= A - y (soluton) = f(u) Lfn (u) A = M f(u ) Lfn (u ) θ θ = M θn [ f (u ),...,f (u )] n Least-Squares Estators (5.3) (cont.) We have outputs & n fttng paraeters to fnd (or equatons & n unknown varables) Usually s greater than n, snce the odel s just an approxaton of the target syste & the data observed ght be corrupted, therefore an exact soluton s not always possble! o overcoe ths nherent conceptual proble, an error vector e s added to copensate A θ + e = y Ch. 5: Least-squares estators 6

7 Least-Squares Estators (5.3) (cont.) Our goal conssts now of fndng reduces the errors between y & n ze θ y a θ θˆ (not dervable!) that ŷ = a θ rather n ze θ ( y a θ) = Least-Squares Estators (5.3) (cont.) If e = y - Aθ then: E( θ) = = = (y a θ) = e e = (y Aθ) (y Aθ) We need to copute: n(y Aθ) θ (y Aθ) Ch. 5: Least-squares estators 7

8 Least-Squares Estators (5.3) (cont.) heore [least-squares estator] he squared error s nzed when θ = (called the least-squares estators, LSE) satsfes the noral equaton A A θˆ = A y, f A A s nonsngular, θˆ s unque & s gven by = (A A) - A y θˆ θˆ Least-Squares Estators (5.3) (cont.) Exaple he relatonshp s between the sprng length & the force appled L = k f + k 0 (lnear odel) θˆ Goal: fnd = (k o, k ) that best fts the data for a gven force fo, we need to deterne the correspondng sprng length L0 Soluton : provde pars (L0,f0) and (L,f) and solve a lnear syste of equatons and varables k and k0 kˆ. However, because of nosy data, ths soluton s not & kˆ 0 relable! Soluton : use a larger tranng set (L,f) Ch. 5: Least-squares estators 8

9 Least-Squares Estators (5.3) (cont.) ranng data for the sprng exaple Least-Squares Estators (5.3) (cont.) snce y = e + Aθ, we can wrte: therefore the LSE of [k0,k] whch nzes e e = 7 = e s equal to : kˆ kˆ 0 = (A..5 e.9. e 3..5 k 0 e k + = { e θ M e { e 3 A) A A.0 y = 0.44 y Ch. 5: Least-squares estators 9

10 Least-Squares Estators (5.3) (cont.) We rely on ths estaton because we have ore data If we are not happy wth the LSE estators then we can ncrease the odel s degree of freedo such that: L = k 0 + k f + k f + + k n f n (least square polynoal!) Least-Squares Estators (5.3) (cont.) Hgher order odels ft better the data but they do not always reflect the nner law that governs the syste For exaple, when f s ncreasng toward 0N, the length s decreasng! Ch. 5: Least-squares estators 0

11 Statstcal Propertes & the Maxu Lkelhood Estator (5.7) Statstcal qualtes of LSE Defnton [unbased estator] An estator θˆ of the paraeter θ s unbased f E( ) = θ, where E[.] s the statstcal expectaton θˆ Defnton [nal varance] An estator θˆ s a nu varance estator f for any other estator θ*: cov( θˆ ) cov(θ*), where cov(θ) s the covarance atrx of the rando vector θ. Ch. 5: Least-squares estators

12 Statstcal Propertes & the Maxu Lkelhood Estator (5.7) (cont.) Statstcal qualtes of LSE (cont.) Gauss-Markov condtons: he error vector e s a vector of uncorrelated rando varables, each wth zero ean & the sae varance σ. hs eans that: E[e] = 0 & E[ee] = σi e E e ( e,e ) E(e = ) E(ee ) E(e e ) = σ I E(e ) Statstcal Propertes & the Maxu Lkelhood Estator (5.7) (cont.) heore [Gauss-Markov]: LSE s unbased & has nu varance. Proof: E[ θˆ ] = E[(A A) - A y] = (A A) - A E[y] = (A A) - A Aθ = θ snce E[y] = E[Aθ] + E(e) = Aθ and (A A) - = A - (A ) - Ch. 5: Least-squares estators

13 Statstcal Propertes & the Maxu Lkelhood Estator (5.7) (cont.) Maxu lkelhood (ML) estator ML s one of the ost wdely used technque for paraeter estaton of a statstcal dstrbuton ML defnton: For a saple of n observatons (of a probablty densty functon ) x, x,, x n, the lkelhood functon L s defned by: L = f(x ; θ) f(x ; θ) f(x n ; θ) Statstcal Propertes & the Maxu Lkelhood Estator (5.7) (cont.) Maxu lkelhood (ML) estator (cont.) he crteron for choosng θ s: pck a value of θ that provdes a hgh probablty of obtanng the actual observed data x, x,, x n θˆ herefore, ML estator s defned as the value of θ whch axzes L: L (ˆ θ) = 0 θ or equvalently: ln L (ˆ θ) = 0 θ Ch. 5: Least-squares estators 3

14 Statstcal Propertes & the Maxu Lkelhood Estator (5.7) (cont.) Maxu lkelhood (ML) estator (cont.) Exaple : ML estaton for exponental dstrbuton f(x; θ) = θ - e -x /θ L = ( θ = θ - e x / θ exp ) ( θ ( θ x ) s the lkelhood functon for observatons x, x,, x. ln L = ln θ θ herefore : θ ˆ = = x x e x / θ )...( θ e x / θ ln L = + θ θ θ ) x = 0 Statstcal Propertes & the Maxu Lkelhood Estator (5.7) (cont.) Maxu lkelhood (ML) estator (cont.) Exaple : ML estaton for noral dstrbuton f(x; µ, σ) = x µ exp πσ σ where µ and σ are respectvely the ean & the varance unknown paraeters. For observatons x, x,, x, we have: Ch. 5: Least-squares estators 4

15 Statstcal Propertes & the Maxu Lkelhood Estator (5.7) (cont.) - Exaple (cont.) L = πσ exp σ ( x µ ) ( πσ ) ( x µ ) and lnl = - * ln σ herefore : lnl = 0 µ σ lnl = 0 σ σ σˆ = ( x µ ˆ ) ( x µ ) 3 ( σ ) ( x µ ) = 0 µ ˆ = x = 0 Statstcal Propertes & the Maxu Lkelhood Estator (5.7) (cont.) Maxu lkelhood (ML) estator (cont.) Equvalence between LSE & MLE heore: Under the Gauss condtons & f each coponent of the vector e follows a noral dstrbuton then the LSE of θ = MLE of θ Ch. 5: Least-squares estators 5

16 LSE for Nonlnear Models (5.8) Nonlnear odels are dvded nto fales Intrnscally lnear Intrnscally nonlnear hrough approprate transforatons of the nput-output varables & fttng paraeters, an ntrnscally lnear odel can becoe a lnear odel By ths transforaton nto lnear odels, LSE can be used to optze the unknown paraeters LSE for Nonlnear Models (5.8) (cont.) Exaple: Decay of the radoactvty of checals y = aebt (a, b < 0 are to be deterned) (t, y) are the nput-output data gven a tranng data {(t, y ), =,, } nzng the error yelds: E(a,b) = bt ( y ae ) we obtan nonlnear equatons wth respect to a & b! herefore, we are better off to lnearze the odel before. = E = a E = b = = ( y ae bt )( e bt ) = 0 ( y ae bt )( at e bt ) = 0 Ch. 5: Least-squares estators 6

17 LSE for Nonlnear Models (5.8) (cont.) lny = lna + bt wth lna = a, lny = y and b = b y = a + b t whch s an ntrnscally lnear odel. he lnear & nonlnear odels are close but can soetes dffer sgnfcantly! he followng table shows a faly of ntrnscally lnear odels LSE for Nonlnear Models (5.8) (cont.) Ch. 5: Least-squares estators 7

18 LSE for Nonlnear Models (5.8) (cont.) Suppose that the underlyng odel s: y = ae bt LSE for Nonlnear Models (5.8) (cont.) Ch. 5: Least-squares estators 8