performance-measures.RmdRecall that the analysis models of interest were the cause-specific Cox proportional hazards models for relapse (REL) and non-relapse mortality (NRM), \(h_k(t \vert X, Z) = h_{k0}(t)\exp(\beta_k X + \gamma_k Z)\) for \(k = \{1,2\}\). We then had two main sets of estimands of interest:
Define \(j = 1,2,...,n_{\text{sim}}\) simulation replications, which begin by simulating an independent dataset for each \(j\) according to some parametrisation, defined by a scenario. In our notation, we suppress \(l = 1,2,...,L\) corresponding to the simulation scenarios.
Let \(\theta\) represent an element of \(\theta_{\text{regr}}\). At each simulation replication, \(M\) imputed datasets are created for each of the four imputation-based methods described in section 5.3 of the paper. In each of these \(M\) datasets, both cause-specific Cox models are fit. The regression coefficients and their standard errors are then pooled according to Rubin’s rules - yielding a vector \(\boldsymbol{\hat{\theta}_{j}} = [\hat{\theta}_{j}, \ \widehat{\text{SE}}(\hat{\theta}_{j})]\). For the complete case analysis, \(\boldsymbol{\hat{\theta}_{j}}\) simply contains the estimated coefficient and standard error from the models fit on the complete-cases (no pooling involved). We then define the performance measures as follows:
Mean: \[ \hat{\theta} = \frac{1}{n_{\text{sim}}} \sum_{j = 1}^{n_{\text{sim}}} \hat{\theta}_j \]
Standard error: \[ \widehat{\text{SE}}(\hat{\theta}) = \frac{1}{n_{\text{sim}}} \sum_{j = 1}^{n_{\text{sim}}} \widehat{\text{SE}}(\hat{\theta}_j) \]
Empirical standard error:
\[
\widehat{\text{EmpSE}}(\hat{\theta}) = \sqrt{\frac{1}{n_{\text{sim}} - 1} \sum_{j = 1}^{n_{\text{sim}}} (\hat{\theta}_j - \hat{\theta})^2}
\]
Bias: \[ \widehat{\text{Bias}}(\hat{\theta}) = \frac{1}{n_{\text{sim}}} \sum_{j = 1}^{n_{\text{sim}}} \hat{\theta}_j - \theta \]
Coverage: \[ \widehat{\text{Cov}}(\hat{\theta}) = \frac{1}{n_{\text{sim}}} \sum_{j = 1}^{n_{\text{sim}}} \mathbf{1}\{ \hat{\theta}_{\text{low},j} < \theta < \hat{\theta}_{\text{upp},j} \} \] where the bounds of the 95% confidence interval \(\hat{\theta}_{\text{low},j}\) and \(\hat{\theta}_{\text{upp},j}\) are computed as \(\hat{\theta}_{j} \pm z_{\alpha/2} \times \widehat{\text{SE}}(\hat{\theta}_{j})\) for the complete-case analysis, whereas for the imputation methods they are based on the \(t\) distribution - see confint.mipo.
Root mean square error: \[ \widehat{\text{RMSE}}(\hat{\theta}) = \sqrt{\frac{1}{n_{\text{sim}} - 1} \sum_{j = 1}^{n_{\text{sim}}} (\hat{\theta}_j - \theta)^2} \]
Monte Carlo standard errors for all measures except RMSE were computed as per the formulas in the tutorial by Morris, White, and Crowther (2019). The Monte Carlo standard error for the RMSE was computed by using the approximate jackknife estimator implemented in the simhelpers package - see the relevant vignette.
To obtain the predicted probabilities when using the imputation methods, the cause-specific models fitted in each imputed dataset are used to create predictions, which are then pooled using Rubin’s rules. For computational reasons, standard errors were not recorded, and so the pooling simply involved averaging the probabilities across imputed datasets. Letting \(\theta\) instead represent an element of \(\theta_{\text{pred}}\), the pooled probability at replication \(j\) of a simulation scenario is defined as
\[ \hat{\theta}_j = \frac{1}{M} \sum_{m=1}^M \hat{\theta}_m, \] where \(\hat{\theta}_m\) is the predicted probability obtained in the \(m^{\text{th}}\) imputed dataset.
The performance measures for the predicted probabilities are the same as those outlined in the previous section, with the exceptions of Standard error and Coverage (since they were not recorded).