Function acceptance_prop Overfits to LAS Policy

[nwood717]

I want to start by saying that everything below is in the spirit of collaboration. I'm glad there are others out there delving deeply into the space of organ allocation simulation, and I hope we can mutually benefit from one another. Ok, now on to the issue at hand. . .

If two simulations use the same acceptance module (what we at the SRTR think of more generally as a placement mechanism) but different allocation policies, we are implicitly assuming that the accept/decline behavior of candidates is invariant across policies (i.e., the probability of acceptance under different allocation policies is reasonably approximated by the same model(s)). This assumption is likely false, however the degree to which it is violated will depend both on the degree of policy change under consideration and the variables which are allowed to inform the acceptance models.

The acceptance_prob function in COMET's accept_organ.R file includes splines on offer_rank. Consequently, using that model across e.g. the LAS and CAS policies assumes that the relationship between a candidate's position on the match run and the odds of acceptance is the same for both policies (and in particular, is as it was under LAS, since data from that policy were used to train the model). This is somewhat borne out by running COMET for LAS and CAS and then seeing that the distributions of offer rank at acceptance are essentially the same.

library(COMET)
library(dplyr)
library(ggplot2)

las <- run_simulation(days = 400, can_start = 1000,
                     match_alg = match_las, wl_model = "LAS15", post_tx_model = "LAS15",
                     wl_weight = 2, post_tx_weight = 1, wl_cap = 365, post_tx_cap = 365, seed = 26638)


cas <- run_simulation(days = 400, can_start = 1000,
                     match_alg = match_cas, wl_model = "CAS23", post_tx_model = "CAS23",
                     wl_weight = 0.25, post_tx_weight = 0.25, wl_cap = 365, post_tx_cap = 1826,
                     bio_weight = .15, pld_weight = 0.05, peds_weight = 0.2, efficiency_weight = 0.1, seed = 26638)


las$recipient_database$name <- 'LAS'
cas$recipient_database$name <- 'CAS'

df <- rbind(
  las$recipient_database %>% select(offer_rank, name), 
  cas$recipient_database %>% select(offer_rank, name))

df %>%                               
  group_by(name) %>% 
  summarize(min = min(offer_rank),
            q1 = quantile(offer_rank, 0.25),
            median = median(offer_rank),
            mean = mean(offer_rank),
            q3 = quantile(offer_rank, 0.75),
            max = max(offer_rank))

#  A tibble: 2 x 7
#   name    min    q1 median  mean    q3   max
#  <chr> <dbl> <dbl>  <dbl> <dbl> <dbl> <dbl>
# 1 CAS       1     2      7  22.8    23   553
# 2 LAS       1     2      6  22.9    23   402

df %>% ggplot(aes(offer_rank, fill = name)) + geom_density(alpha = 0.2)

This is also seen in Table 4 of A modular simulation framework for organ allocation, which shows that the distribution for offer rank at acceptance is essentially the same for all four policies. By contrast, the median offer rank (what I call offer number) at acceptance has doubled (~6 to ~12) since lung allocation transitioned from LAS to CAS .

https://srtr.org/tools/donation-and-transplant-system-explorer/

Based on this I have the following suggestions for COMET.

1. Rename the acceptance_prob function to indicate the policy from which data were used to train it (I assume LAS, so maybe las_accept_prob), and add some documentation with specifics on how that model was trained, the data used, etc.)
2. Consider creating another acceptance module which is trained using CAS data, which will be more applicable to future simulations that wish to compare the current version of CAS to proposed versions of CAS. Add similar details about model training in documentation.
3. Consider how to mitigate the problem of overfitting to whatever policy was used to create any acceptance module.
4. Let the acceptance module be a parameter of run_simulation so that the user can select which model to use, as well as create and use his own function for determining the probability that each candidate accepts without having to futz with the source code.

-Nick