The gold standard concists of 52 concept set targets. For each concept set target, A combination of vocabulary lookup
and OHDSI's Pheobe 2.0 tool were used to find possibly relevant concepts, of which 25 were randomly sampled. In
addition, LLMs were used to generate clinical definitions of each concept set target.
The 1,300 target-definition-concept triplets were manually reviewed to determine whether the concept should be part of the concept set according to the definition.
PPV, sensitivity, and specificity are computed overall, and stratified by concept set target domain.
evaluateConceptAdjudication(concepts)The data frame returned by getConceptsForAdjudication() with an extra 'adjudication' column, having
value 'YES' if the concept should be included in a concept set for the target, or 'NO' otherwise.
A data frame with performance statistics.
concepts <- getConceptsForAdjudication()
# Using random adjudication for this example:
concepts$adjudication <- sample(c("YES", "NO"), nrow(concepts), replace = TRUE)
evaluateConceptAdjudication(concepts)
#> # A tibble: 4 × 8
#> targetDomain tp fp tn fn ppv sensitivity specificity
#> <chr> <int> <int> <int> <int> <dbl> <dbl> <dbl>
#> 1 CONDITION 61 229 263 47 0.210 0.565 0.535
#> 2 MEASUREMENT 35 119 114 32 0.227 0.522 0.489
#> 3 PROCEDURE 46 150 165 39 0.235 0.541 0.524
#> 4 All 142 498 542 118 0.222 0.546 0.521