Using Keeper with Large Language Models
Martijn Schuemie
2024-06-18
Source:vignettes/UsingKeeperWithLlms.Rmd
UsingKeeperWithLlms.RmdIntroduction
This vignette describes how one could use Keeper to generate patient summaries, and have them be reviewed by a large-language model (LLM).
Running Keeper
As an example, we’ll run Keeper on Eunomia. Eunomia is an OHDSI package that contains a tiny simulated dataset in the Common Data Model (CDM). It is mostly focused on NSAIDs and gastrointestinal (GI) bleeding, so we’ll use GI bleed as our example outcome.
Setting up Eunomia
First we must download and install Eunomia:
install.packages("Eunomia")Next, we can obtain connection details to the Eunomia database. (Note: this will download the database from the internet):
library(Eunomia)
connectionDetails <- getEunomiaConnectionDetails()We can have the default set of cohorts generated in Eunomia:
createCohorts(connectionDetails)## Cohorts created in table main.cohort## cohortId name
## 1 1 Celecoxib
## 2 2 Diclofenac
## 3 3 GiBleed
## 4 4 NSAIDs
## description
## 1 A simplified cohort definition for new users of celecoxib, designed specifically for Eunomia.
## 2 A simplified cohort definition for new users ofdiclofenac, designed specifically for Eunomia.
## 3 A simplified cohort definition for gastrointestinal bleeding, designed specifically for Eunomia.
## 4 A simplified cohort definition for new users of NSAIDs, designed specifically for Eunomia.
## count
## 1 1844
## 2 850
## 3 479
## 4 2694Run Keeper
Next, we run Keeper. We meticulously select concepts for each Keeper category:
keeper <- createKeeper(
connectionDetails = connectionDetails,
databaseId = "Synpuf",
cdmDatabaseSchema = "main",
cohortDatabaseSchema = "main",
cohortTable = "cohort",
cohortDefinitionId = 3,
cohortName = "GI Bleed",
sampleSize = 100,
assignNewId = TRUE,
useAncestor = TRUE,
doi = c(4202064, 192671, 2108878, 2108900, 2002608),
symptoms = c(4103703, 443530, 4245614, 28779),
comorbidities = c(81893, 201606, 313217, 318800, 432585, 4027663, 4180790, 4212540,
40481531, 42535737, 46271022),
drugs = c(904453, 906780, 923645, 929887, 948078, 953076, 961047, 985247, 992956,
997276, 1102917, 1113648, 1115008, 1118045, 1118084, 1124300, 1126128,
1136980, 1146810, 1150345, 1153928, 1177480, 1178663, 1185922, 1195492,
1236607, 1303425, 1313200, 1353766, 1507835, 1522957, 1721543, 1746940,
1777806, 19044727, 19119253, 36863425),
diagnosticProcedures = c(4087381, 4143985, 4294382, 42872565, 45888171, 46257627),
measurements = c(3000905, 3000963, 3003458, 3012471, 3016251, 3018677, 3020416,
3022217, 3023314, 3024929, 3034426),
alternativeDiagnosis = c(24966, 76725, 195562, 316457, 318800, 4096682),
treatmentProcedures = c(0),
complications = c(132797, 196152, 439777, 4192647)
)## | | | 0% | |======================= | 33% | |=============================================== | 67% | |======================================================================| 100%
## | | | 0% | |======================= | 33% | |=============================================== | 67% | |======================================================================| 100%
## Getting cohort table.
## | | | 0% | |======================= | 33% | |=============================================== | 67% | |======================================================================| 100%
## Getting patient data for keeperOutput.
## | | | 0% | |== | 3% | |==== | 6% | |====== | 9% | |======== | 12% | |=========== | 15% | |============= | 18% | |=============== | 21% | |================= | 24% | |=================== | 27% | |===================== | 30% | |======================= | 33% | |========================= | 36% | |============================ | 39% | |============================== | 42% | |================================ | 45% | |================================== | 48% | |==================================== | 52% | |====================================== | 55% | |======================================== | 58% | |========================================== | 61% | |============================================= | 64% | |=============================================== | 67% | |================================================= | 70% | |=================================================== | 73% | |===================================================== | 76% | |======================================================= | 79% | |========================================================= | 82% | |=========================================================== | 85% | |============================================================== | 88% | |================================================================ | 91% | |================================================================== | 94% | |==================================================================== | 97% | |======================================================================| 100%The output is a table with one row per person:
keeper## # A tibble: 100 × 18
## personId age gender observationPeriod visitContext presentation
## <dbl> <dbl> <chr> <chr> <chr> <chr>
## 1 65 37 Female -13639.0 days - 7823.0 days "Inpatient V… Gastrointes…
## 2 88 37 Female -13752.0 days - 4861.0 days "Inpatient V… Gastrointes…
## 3 38 43 Female -15759.0 days - 3750.0 days "" Gastrointes…
## 4 64 40 Male -14445.0 days - 12337.0 days "" Gastrointes…
## 5 28 42 Male -15336.0 days - 7715.0 days "Inpatient V… Gastrointes…
## 6 93 39 Female -14487.0 days - 8564.0 days "Inpatient V… Gastrointes…
## 7 30 40 Female -14875.0 days - 7770.0 days "Inpatient V… Gastrointes…
## 8 6 36 Female -13187.0 days - 6189.0 days "Inpatient V… Gastrointes…
## 9 35 36 Female -13369.0 days - 16395.0 days "Inpatient V… Gastrointes…
## 10 69 39 Male -14315.0 days - 14063.0 days "" Gastrointes…
## # ℹ 90 more rows
## # ℹ 12 more variables: comorbidities <chr>, symptoms <chr>, priorDisease <chr>,
## # priorDrugs <chr>, priorTreatmentProcedures <chr>,
## # diagnosticProcedures <chr>, measurements <chr>, alternativeDiagnosis <chr>,
## # afterDisease <chr>, afterTreatmentProcedures <chr>, afterDrugs <chr>,
## # death <chr>LLM review
We can convert the Keeper output to prompts for a LLM, and parse the output of the LLM to get a classification of whether the patient truly had GI bleeding.
Generating prompts
We need two prompts: the system prompt is a general description of how the LLM should behave. The (main) prompt contains the patient-specific information. First we create settings, then we generate the system prompt:
# Use the default settings:
settings <- createPromptSettings()
systempPrompt <- createSystemPrompt(setting = settings,
diseaseName = "Gastrointestinal bleeding")
writeLines(systempPrompt)## Act as a medical doctor reviewing a patient's healthcare data captured during routine clinical care, such as electronic health records and insurance claims.
## Write a medical narrative that fits the recorded health data followed by a determination of whether the patient had Gastrointestinal bleeding.
##
## Remember that recording a diagnosis for a disease could occur either because the patient had the disease or as justification for performing a diagnostic procedure to determine whether the patient has the disease. A diagnosis by itself or accompanied with only diagnostic procedures may therefore be insufficient evidence, even if recorded more than once. Lack of additional evidence of Gastrointestinal bleeding other than the diagnosis and diagnostic procedures probably means that the patient was only being tested, and does not actually have Gastrointestinal bleeding. However, it unlikely that a patient will be tested many times over, so an abundance of diagnoses will mean the patient has the disease.
##
## In your final summary, indicate "yes" if the most probable scenario is that the patient had Gastrointestinal bleeding.
## Indicate "no" if it is not the most probable scenario, for example when it is more likely that the patient was tested for the disease but the diagnosis was not confirmed. Also indicate "no" when there is insufficient information to say anything about the relative probability of scenarios.
##
## Use the following format:
##
## Clinical narrative:
##
## Evidence in favor of Gastrointestinal bleeding:
##
## Evidence against Gastrointestinal bleeding:
##
## Summary: (Only "yes" or "no")Then, for each patient we can generate the main prompt:
row <- keeper[1, ]
prompt <- createPrompt(setting = settings,
diseaseName = "Gastrointestinal bleeding",
keeperRow = row)
writeLines(prompt)## Demographics and details about the visit: Female, thirty-seven yo; Visit: Inpatient Visit (1.0 days)
##
## Diagnoses recorded on the day of the visit: Gastrointestinal hemorrhage;
##
## Diagnoses recorded prior to the visit: Peptic ulcer (day -2689); Ulcerative colitis (day -264)
##
## Treatments recorded prior to the visit: celecoxib (day -84.0, for 0.0 days);
##
## Diagnostic procedures recorded proximal to the visit: None
##
## Laboratory tests recorded proximal to the visit: None
##
## Alternative diagnoses recorded proximal to the visit: None
##
## Diagnoses recorded after the visit: None
##
## Treatments recorded during or after the visit: NoneGenerate a response
Next, we can use the system prompt and prompt together to query the LLM. Many LLMs are available, including open source ones. Setting up and interacting with the LLM is beyond the scope of this vignette. Here we assume a LLM was used to generate this response:
response <- "Summary: yes"Parse the response
LLMs can be quite verbose, and often we just want a yes or no answer.
For this we can use the parseLlmResponse() function that
uses a set of patterns to decide between ‘yes’, ‘no’, or ‘I don’t
know’:
parseLlmResponse(response)## [1] "yes"