Dealing with Missing Data

Transkript

1 Dealing with Missing Data Challenges and Solutions Nicole Erler Department of Biostatistics, Erasmus Medical Center N_Erler wwwnerlercom NErler 13 January 2020

2 Handling Missing Values is Easy! Functions automatically exclude missing values: ## [] ## Residual standard error: 2305 on 69 degrees of freedom ## (25 observations deleted due to missingness) ## Multiple R-squared: , Adjusted R-squared: ## F-statistic: 1407 on 5 and 69 DF, p-value:

3 Handling Missing Values is Easy! Functions automatically exclude missing values: ## [] ## Residual standard error: 2305 on 69 degrees of freedom ## (25 observations deleted due to missingness) ## Multiple R-squared: , Adjusted R-squared: ## F-statistic: 1407 on 5 and 69 DF, p-value: Imputation is super easy: library("mice") imp <- mice(mydata) However 1

4 Handling Missing Values Correctly is Not So Easy! Complete case analysis is usually biased 2

5 Handling Missing Values Correctly is Not So Easy! Complete case analysis is usually biased (Imputation) methods make certain assumptions, eg: missingness is M(C)AR 2

6 Handling Missing Values Correctly is Not So Easy! Complete case analysis is usually biased (Imputation) methods make certain assumptions, eg: missingness is M(C)AR the incomplete variable has a certain conditional distribution (eg normal) 2

7 Handling Missing Values Correctly is Not So Easy! Complete case analysis is usually biased (Imputation) methods make certain assumptions, eg: missingness is M(C)AR the incomplete variable has a certain conditional distribution (eg normal) all associations are linear 2

8 Handling Missing Values Correctly is Not So Easy! Complete case analysis is usually biased (Imputation) methods make certain assumptions, eg: missingness is M(C)AR the incomplete variable has a certain conditional distribution (eg normal) all associations are linear compatibility and congeniality 2

9 Handling Missing Values Correctly is Not So Easy! Complete case analysis is usually biased (Imputation) methods make certain assumptions, eg: missingness is M(C)AR the incomplete variable has a certain conditional distribution (eg normal) all associations are linear compatibility and congeniality violation bias 2

10 Imputation??? Remind me, how did that imputation thing work again??? 3

11 Imputation Imputation filling in missing values with (good) "guesses" 4

12 Imputation Imputation filling in missing values with (good) "guesses" Important: Missing values uncertainty This needs to be taken into account!!! 4

13 Imputation Imputation filling in missing values with (good) "guesses" Important: Missing values uncertainty This needs to be taken into account!!! Donald Rubin (in the 1970s): Represent each missing value with multiple imputed values Multiple Imputation Note: Imputation is not the only approach to handle missing values (Also: maximum likelihood, inverse probability weighting, ) 4

14 Multiple Imputation incomplete data multiple imputed datasets analysis results pooled results 1 Imputation: impute multiple times multiple completed datasets 2 Analysis: analyse each of the datasets 3 Pooling: combine results, taking into account additional uncertainty 5

15 Imputation Step Two main approaches Joint Model Multiple Imputation the "original" approach often using a multivariate normal distribution 6

16 Imputation Step Two main approaches Joint Model Multiple Imputation the "original" approach often using a multivariate normal distribution Multiple Imputation with Chained Equations (MICE) also: Fully Conditional Specification (FCS) now often considered the gold standard 6

17 Multiple Imputation with Chained Equations (MICE) For each incomplete variable, specify a model using all other variables: }{{} full conditionals x 1 x 2 x 3 x 4 NA NA NA NA NA NA 7

18 Multiple Imputation with Chained Equations (MICE) For each incomplete variable, specify a model using all other variables: }{{} full conditionals x 1 x 2 x 3 x 4 NA NA NA NA NA NA x 1 x 2 + x 3 + x 4 + x 2 x 1 + x 3 + x 4 + x 3 x 1 + x 2 + x 4 + x 4 x 1 + x 2 + x 3 + 7

19 Multiple Imputation with Chained Equations (MICE) For each incomplete variable, specify a model using all other variables: }{{} full conditionals For example: x 1 x 2 x 3 x 4 NA NA NA NA NA NA x 1 x 2 + x 3 + x 4 + x 2 x 1 + x 3 + x 4 + x 3 x 1 + x 2 + x 4 + x 4 x 1 + x 2 + x 3 + linear regression logistic regression 7

20 Multiple Imputation with Chained Equations (MICE) MICE is an iterative algorithm: start with initial guess x 1 x 2 x 3 x 4 NA NA NA NA NA NA 8

21 Multiple Imputation with Chained Equations (MICE) MICE is an iterative algorithm: start with initial guess update x 1 based on initial values of x 2, x 3, x 4, x 1 x 2 x 3 x 4 NA NA NA NA NA NA 8

22 Multiple Imputation with Chained Equations (MICE) MICE is an iterative algorithm: start with initial guess update x 1 based on initial values of x 2, x 3, x 4, update x 2 based on new x 1 and initial values of x 3, x 4, x 1 x 2 x 3 x 4 NA NA NA NA NA NA 8

23 Multiple Imputation with Chained Equations (MICE) MICE is an iterative algorithm: start with initial guess update x 1 based on initial values of x 2, x 3, x 4, update x 2 based on new x 1 and initial values of x 3, x 4, update x 1 again, based on updated x 2, x 3, x 4, x 1 x 2 x 3 x 4 NA NA NA NA NA NA 8

24 Multiple Imputation with Chained Equations (MICE) MICE is an iterative algorithm: start with initial guess update x 1 based on initial values of x 2, x 3, x 4, update x 2 based on new x 1 and initial values of x 3, x 4, update x 1 again, based on updated x 2, x 3, x 4, until convergence x 1 x 2 x 3 x 4 NA NA NA NA NA NA 8

25 Multiple Imputation with Chained Equations (MICE) MICE is an iterative algorithm: start with initial guess update x 1 based on initial values of x 2, x 3, x 4, update x 2 based on new x 1 and initial values of x 3, x 4, update x 1 again, based on updated x 2, x 3, x 4, until convergence x 1 x 2 x 3 x 4 NA NA NA NA NA NA Values from last iteration one imputed dataset 8

26 MICE Makes Assumptions (Imputation) methods make certain assumptions, eg: missingness is M(C)AR the incomplete variable has a certain conditional distribution (eg normal) all associations are linear compatibility and congeniality 9

27 Missing Data Mechanisms Missing Completely At Random (MCAR) Missing At Random (MAR) Missing Not At Random (MNAR) 10

28 Missing Data Mechanisms Missing Completely At Random (MCAR) p(r X obs, X mis ) = p(r) Missingness is independent of all data questionnaire got lost in mail Missing At Random (MAR) Missing Not At Random (MNAR) 10

29 Missing Data Mechanisms Missing Completely At Random (MCAR) p(r X obs, X mis ) = p(r) Missingness is independent of all data Missing At Random (MAR) p(r X obs, X mis ) = p(r X obs ) Missingness depends only on observed data questionnaire got lost in mail overweight participants are less likely to report their chocolate consumption (and we know their weight) Missing Not At Random (MNAR) 10

30 Missing Data Mechanisms Missing Completely At Random (MCAR) p(r X obs, X mis ) = p(r) Missingness is independent of all data Missing At Random (MAR) p(r X obs, X mis ) = p(r X obs ) Missingness depends only on observed data questionnaire got lost in mail overweight participants are less likely to report their chocolate consumption (and we know their weight) Missing Not At Random (MNAR) p(r X obs, X mis ) p(r X obs ) Missingness depends (also) on unobserved data overweight participants are less likely to report their weight 10

31 MICE Makes Assumptions (Imputation) methods make certain assumptions, eg: missingness is M(C)AR the incomplete variable has a certain conditional distribution (eg normal) all associations are linear compatibility and congeniality In case of MNAR: MICE bias 11

32 MICE Makes Assumptions (Imputation) methods make certain assumptions, eg: missingness is M(C)AR the incomplete variable has a certain conditional distribution (eg normal) all associations are linear compatibility and congeniality 12

33 Imputation Model Misspecification x 1 x 2 + x 3 + x 4 + x 2 x 1 + x 3 + x 4 + x 3 x 1 + x 2 + x 4 + x 4 x 1 + x 2 + x 3 + For example: linear regression logistic regression 13

34 Imputation Model Misspecification x 1 x 2 + x 3 + x 4 + x 2 x 1 + x 3 + x 4 + x 3 x 1 + x 2 + x 4 + x 4 x 1 + x 2 + x 3 + For example: linear regression logistic regression count count incomplete covariate incomplete covariate incomplete covariate covariate 13

35 Imputation Model Misspecification count count incomplete covariate incomplete covariate incomplete covariate covariate misspecification of the residual distribution misspecification of the association structure 14

36 Imputation Model Misspecification count count incomplete covariate incomplete covariate incomplete covariate covariate misspecification of the residual distribution misspecification of the association structure Partial solutions: Predictive Mean Matching Passive imputation 14

37 Imputation Model Misspecification count count incomplete covariate incomplete covariate incomplete covariate covariate misspecification of the residual distribution misspecification of the association structure Partial solutions: Predictive Mean Matching Passive imputation But can get tedious requires knowledge (about data & methods) users often inexperienced and/or lazy 14

38 MICE Makes Assumptions (Imputation) methods make certain assumptions, eg: missingness is M(C)AR the incomplete variable has a certain conditional distribution (eg normal) all associations are linear compatibility and congeniality Model misspecification bias 15

39 MICE Makes Assumptions (Imputation) methods make certain assumptions, eg: missingness is M(C)AR the incomplete variable has a certain conditional distribution (eg normal) all associations are linear compatibility and congeniality 16

40 Compatibility & Congeniality Compatibility A joint distribution exists, that has the full conditionals (imputation models) as its conditional distributions 17

41 Compatibility & Congeniality Compatibility A joint distribution exists, that has the full conditionals (imputation models) as its conditional distributions Congeniality The imputation model is compatible with the analysis model 17

42 Compatibility & Congeniality in MICE MICE is based on the idea of Gibbs sampling Exploits the fact that a joint distribution is fully determined by its full conditional distributions 18

43 Compatibility & Congeniality in MICE MICE is based on the idea of Gibbs sampling Exploits the fact that a joint distribution is fully determined by its full conditional distributions But: In MICE Imputation models are specified directly no guarantee that a corresponding joint distribution exists 18

44 Compatibility & Congeniality in MICE joint distribution Gibbs MICE full conditionals 19

45 Compatibility & Congeniality in MICE joint distribution Gibbs MICE full conditionals Is this a problem? often not but it can be when imputation/analysis models contradict each other different assumptions are made during analysis and imputation the outcome cannot easily be included in the imputation models 19

46 Example 1: Contradicting Models Analysis model with a quadratic association: y = β 0 + β 1 x + β 2 x 2 + x y observed missing 20

47 Example 1: Contradicting Models Imputation model for x (when using MICE naively): x = θ 10 + θ 11 y +, ie, a linear relation between x and y is assumed x y observed missing imputed 21

48 Example 1: Contradicting Models Imputation model for x (when using MICE naively): x = θ 10 + θ 11 y +, ie, a linear relation between x and y is assumed x y fit on complete fit on imputed observed missing imputed 21

49 Example 2: Contradicting Models Analysis model with interaction term: y = β 0 + β x x + β z z + β xz xz +, ie, y again has a non-linear relationship with x x y missing (z = 0) missing (z = 1) observed (z = 0) observed (z = 1) 22

50 Example 2: Contradicting Models Imputation model for x (when using MICE naively): x = θ 10 + θ 11 y + θ 12 z +, x y missing (z = 0) missing (z = 1) observed (z = 0) observed (z = 1) imputed (z = 0) imputed (z = 1) 23

51 Example 2: Contradicting Models Imputation model for x (when using MICE naively): x = θ 10 + θ 11 y + θ 12 z +, y missing (z = 0) missing (z = 1) observed (z = 0) observed (z = 1) imputed (z = 0) imputed (z = 1) true imputed x 23

52 Example 3: Longitudinal / Multi-level Data id time y x NA NA NA NA 24

53 Example 3: Longitudinal / Multi-level Data Imputation in long format rows are treated as independent imputations in baseline covariates will vary over time bias Can we use data in wide format (one row per subject)? can be very inefficient not always possible id time y x NA NA NA NA 25

54 Example 3: Longitudinal / Multi-level Data y y time time y y time time 26

55 Compatibility & Congeniality in MICE Lack of compatibility / congeniality can become a problem for MICE in settings with Non-linear associations non-linear effects interaction terms complex outcomes multi-level settings time-to-event outcomes What can we do in these settings? 27

56 Imputation in Complex Settings Remember, the problem is joint distribution Gibbs MICE full conditionals 28

57 Imputation in Complex Settings Remember, the problem is joint distribution Gibbs MICE full conditionals Solution: Start with the joint distribution! 28

58 Imputation in Complex Settings Remember, the problem is joint distribution Gibbs MICE full conditionals Solution: Start with the joint distribution! New problem: What is the multivariate distribution of multiple variables of different types? 28

59 Imputation in Complex Settings Remember, the problem is joint distribution Gibbs MICE full conditionals Solution: Start with the joint distribution! New problem: What is the multivariate distribution of multiple variables of different types? Usually, the joint distribution is not of any known form 28

60 Joint Model Imputation Multivariate Normal Model Approximate the joint distribution by a known multivariate (usually normal) distribution this is Joint Model Multiple Imputation assures compatibility & congeniality can t handle non-linear associations 29

61 Joint Model Imputation Multivariate Normal Model Approximate the joint distribution by a known multivariate (usually normal) distribution this is Joint Model Multiple Imputation assures compatibility & congeniality can t handle non-linear associations Sequential Factorization Factorize the joint distribution into (a sequence of) conditional distributions assures compatibility & congeniality can handle non-linear associations 29

62 Sequential Factorization A joint distribution p(y, x) can be written as the product of conditional distributions: p(y, x) = p(y x) p(x) (or alternatively p(y, x) = p(x y) p(y)) 30

63 Sequential Factorization A joint distribution p(y, x) can be written as the product of conditional distributions: p(y, x) = p(y x) p(x) (or alternatively p(y, x) = p(x y) p(y)) This can be extended for more variables: p(y, x 1,, x p ) = p(y x 1,, x p ) p(x 1 x 2,, x p ) p(x 2 x 3,, x p ) p(x p ) 30

64 Sequential Factorization in the Bayesian Framework Joint Distribution p(y, X, θ) = p(y X, θ) }{{} analysis model p(x θ) }{{} imputation part θ contains regr coefficients, variance parameters, p(θ) }{{} priors 31

65 Sequential Factorization in the Bayesian Framework Joint Distribution p(y, X, θ) = p(y X, θ) }{{} analysis model p(x θ) }{{} imputation part θ contains regr coefficients, variance parameters, Imputation part p(θ) }{{} priors X {}}{ p( x 1,, x p, X compl θ) = p(x 1 X compl, θ) p(x 2 X compl, x 1, θ) p(x 3 X compl, x 1, x 2, θ) 31

66 Sequential Factorization in the Bayesian Framework Extension for a Multi-level Setting p(y X, b, θ) p(x θ) }{{}}{{} analysis imputation model part p(b θ) }{{} random effects p(θ) }{{} priors 32

67 Sequential Factorization in the Bayesian Framework Extension for a Multi-level Setting p(y X, b, θ) p(x θ) }{{}}{{} analysis imputation model part p(b θ) }{{} random effects p(θ) }{{} priors Extension for a Time-to-Event Outcome p(t, D X, θ) }{{} analysis model p(x θ) }{{} imputation part p(θ) }{{} priors 32

68 Sequential Factorization in the Bayesian Framework Extension for a Multi-level Setting p(y X, b, θ) p(x θ) }{{}}{{} analysis imputation model part p(b θ) }{{} random effects p(θ) }{{} priors Extension for a Time-to-Event Outcome p(t, D X, θ) }{{} analysis model Extension for a Multivariate Outcome p(y 1, y 2 X, θ) } {{ } analysis model p(x θ) }{{} imputation part p(x θ) }{{} imputation part p(θ) }{{} priors p(θ) }{{} priors 32

69 MICE vs Sequential Factorization Imputation in MICE p(x 1 y, X compl, x 2, x 3, x 4,, θ) p(x 2 y, X compl, x 1, x 3, x 4,, θ) p(x 3 y, X compl, x 1, x 2, x 4,, θ) Sequential Factorization p(y X compl, x 1, x 2, x 3,, θ) p(x 1 X compl, θ) p(x 2 X compl, x 1, θ) p(x 3 X compl, x 1, x 2, θ) 33

70 MICE vs Sequential Factorization Imputation in MICE p(x 1 y, X compl, x 2, x 3, x 4,, θ) p(x 2 y, X compl, x 1, x 3, x 4,, θ) p(x 3 y, X compl, x 1, x 2, x 4,, θ) Sequential Factorization p(y X compl, x 1, x 2, x 3,, θ) p(x 1 X compl, θ) p(x 2 X compl, x 1, θ) p(x 3 X compl, x 1, x 2, θ) No issues with complex outcomes, eg: multi-level survival non-linear effects congeniality compatibility 33

71 MICE vs Sequential Factorization Imputation in MICE p(x 1 y, X compl, x 2, x 3, x 4,, θ) p(x 2 y, X compl, x 1, x 3, x 4,, θ) p(x 3 y, X compl, x 1, x 2, x 4,, θ) Sequential Factorization p(y X compl, x 1, x 2, x 3,, θ) p(x 1 X compl, θ) p(x 2 X compl, x 1, θ) p(x 3 X compl, x 1, x 2, θ) Analysis model part of specification parameters of interest directly available no need for pooling simultaneous analysis and imputation 34

72 Joint Analysis and Imputation in Sequential Factorization is implemented in the package JointAI 35

73 Joint Analysis and Imputation in Sequential Factorization is implemented in the package JointAI Bayesian analysis of incomplete data using (generalized) linear regression (generalized) linear mixed models ordinal (mixed) models parametric (Weibull) time-to-event models Cox proportional hazards models 35

74 Joint Analysis and Imputation in Sequential Factorization is implemented in the package JointAI Bayesian analysis of incomplete data using (generalized) linear regression (generalized) linear mixed models ordinal (mixed) models parametric (Weibull) time-to-event models Cox proportional hazards models on CRAN: GitHub: website: 35

75 Joint Analysis and Imputation in standard regression mixed model type outcome covariate outcome covariate normal lognormal (soon) (soon) Gamma beta (soon) (soon) (soon) binomial poisson (soon) ordinal multinomial (soon) (soon) (soon) Available soon: Joint models (of longitudinal & time-to-event data) Multivariate models 36

76 JointAI: How does it work? Requirements: ( JAGS (Just Another Gibbs Sampler; 37

77 JointAI: How does it work? Requirements: ( JAGS (Just Another Gibbs Sampler; Installation: installpackages("jointai") 37

78 JointAI: How does it work? Requirements: ( JAGS (Just Another Gibbs Sampler; Installation: installpackages("jointai") Usage: library("jointai") res <- lm_imp(sbp ~ age + gender + smoke + occup, data = NHANES, niter = 300) 37

79 JointAI: How does it work? traceplot(res) 38

80 JointAI: How does it work? summary(res) ## ## Linear model fitted with JointAI ## ## Call: ## lm_imp(formula = SBP ~ age + gender + smoke + occup, data = NHANES, ## niter = 300) ## ## Posterior summary: ## Mean SD 25% 975% tail-prob GR-crit ## (Intercept) ## age ## genderfemale ## smokeformer ## smokecurrent ## occuplooking for work ## occupnot working ## ## Posterior summary of residual std deviation: ## Mean SD 25% 975% GR-crit ## sigma_sbp ## ## ## MCMC settings ## [] 39

81 What is left to do? (Imputation) methods make certain assumptions, eg: missingness is M(C)AR the incomplete variable has a certain conditional distribution all associations are linear compatibility and congeniality 40

82 What is left to do? (Imputation) methods make certain assumptions, eg: missingness is M(C)AR the incomplete variable has a certain conditional distribution all associations are linear compatibility and congeniality extension to MNAR using pattern mixture model 40

83 What is left to do? (Imputation) methods make certain assumptions, eg: missingness is M(C)AR the incomplete variable has a certain conditional distribution all associations are linear compatibility and congeniality extension to MNAR using pattern mixture model non-parametric Bayesian methods 40

84 What is left to do? (Imputation) methods make certain assumptions, eg: missingness is M(C)AR the incomplete variable has a certain conditional distribution all associations are linear compatibility and congeniality extension to MNAR using pattern mixture model non-parametric Bayesian methods semi-parametric methods 40

85 Take-Home Message handling missing values correctly: not that easy all methods have assumptions violation bias 41

86 Take-Home Message handling missing values correctly: not that easy all methods have assumptions violation bias good use of (imputation) methods requires knowledge of the data knowledge of the methods knowledge of the software time & patience! 41

87 Take-Home Message handling missing values correctly: not that easy all methods have assumptions violation bias good use of (imputation) methods requires knowledge of the data knowledge of the methods knowledge of the software time & patience! JointAI aims to facilitate correct handling of missing values by assuring compatibility & congeniality simultaneous analysis & imputation especially for complex settings 41

88 Take-Home Message handling missing values correctly: not that easy all methods have assumptions violation bias good use of (imputation) methods requires knowledge of the data knowledge of the methods knowledge of the software time & patience! JointAI aims to facilitate correct handling of missing values by assuring compatibility & congeniality simultaneous analysis & imputation especially for complex settings There is no magical solution that will always work in all settings 41

89 Thank you for your attention N_Erler NErler wwwnerlercom