Adding Custom Patient-Level Prediction Algorithms
Jenna Reps, Martijn J. Schuemie, Patrick B. Ryan, Peter R. Rijnbeek
2026-11-13
Source:vignettes/AddingCustomModels.Rmd
AddingCustomModels.RmdIntroduction
This vignette describes how you can add your own custom algorithms in
the Observational Health Data Sciences and Informatics (OHDSI) PatientLevelPrediction
package. This allows you to fully leverage the OHDSI
PatientLevelPrediction framework for model development and validation.
This vignette assumes you have read and are comfortable with building
single patient level prediction models as described in the
vignette('BuildingPredictiveModels').
We invite you to share your new algorithms with the OHDSI community through our GitHub repository.
Algorithm Code Structure
A model is described by a set<Name> function that returns its
modelSettings. PatientLevelPrediction does the rest:
fitClassifier() prepares the data, runs the hyperparameter
search with cross-validation, fits the final model and builds the
plpModel, and predictPlp(),
savePlpModel() and loadPlpModel() use the same
settings later. The model itself supplies a few functions, the hooks,
which are named in its settings. The xgboost model in
GradientBoostingMachine.R is a complete example.
Set
The set<Name> function takes the hyperparameter values to
search and returns a list of class modelSettings with:
- param - a named list of hyperparameter values;
createHyperparameterSettings()decides how candidates are drawn from them (a grid by default) - settings - the model’s settings and its hooks:
- modelName, modelType (
"binary") and seed - prepareData - turns the training data into what train expects,
usually
"toSparseM"(a sparse matrix with one row per patient). If it has asettingsargument, it also gets these settings. - train - fits the model for one set of hyperparameters
- predict - predicts risks with a fitted model
- variableImportance - gives the importance of each covariate
- saveType - how the fitted model is saved: the name of a save handler
(see below),
"RtoJson"for a model that is an R list, or"file"for a model that is a folder of files
- modelName, modelType (
Each hook can be a function, the name of a function, or
"pkg::function". A name is looked up in
PatientLevelPrediction and then on the search path, so plain names are
meant for the package’s own functions; "pkg::function" is
an exported function of an installed package, which is loaded but does
not need to be attached. Use the "pkg::function" form for
hooks in your own package, so that saved models still work in a new R
session. All hooks are looked up before fitting starts, so a misspelled
one fails at once.
For example, a model called madeUp with two hyperparameters:
setMadeUp <- function(a = c(1, 4, 10), b = 2, seed = NULL) {
# add input checks here...
if (is.null(seed)) {
seed <- sample(100000, 1)
}
param <- list(
a = a,
b = b
)
settings <- list(
modelName = "Made Up",
modelType = "binary",
seed = seed,
prepareData = "toSparseM",
train = "myPackage::fitMadeUp",
predict = "myPackage::predictMadeUp",
variableImportance = "myPackage::varImpMadeUp",
saveType = "myPackage::saveLoadMadeUp"
)
result <- list(
param = param,
settings = settings
)
class(result) <- "modelSettings"
return(result)
}Train
The train hook fits the model for one candidate. It gets:
- dataMatrix - the prepared data, a sparse matrix with rows as
patients and columns as features when prepareData is
"toSparseM" - labels - a data.frame where labels$outcomeCount is the class label (0 or 1) per patient
- hyperParameters - a named list with one value per hyperparameter
- settings - the settings from the set function, such as the seed
and returns the fitted model.
fitMadeUp <- function(dataMatrix, labels, hyperParameters, settings) {
# set the seed for reproducibility
set.seed(settings$seed)
model <- madeUpModel(
X = dataMatrix,
Y = labels$outcomeCount,
a = hyperParameters$a,
b = hyperParameters$b
)
return(model)
}An error in train or predict stops the search, so that problems in
the model’s code show up at once. When the model knows that a candidate
simply cannot be fitted, for example because it runs out of memory, it
can call stopCandidate() instead. The search then warns,
records the candidate with a missing performance and the message, and
goes on with the next one; it stops only if every candidate is
skipped.
fitMadeUp <- function(dataMatrix, labels, hyperParameters, settings) {
model <- tryCatch(
madeUpModel(X = dataMatrix, Y = labels$outcomeCount, a = hyperParameters$a),
madeUpOutOfMemory = function(e) stopCandidate("out of memory")
)
return(model)
}Predict
The predict hook gets:
- plpModel - during the search the model returned by train, and after
fitting the
plpModel - data - during the search a part of the prepared data, and when
predicting on new data a
plpDataobject - cohort - the patients to predict for
It returns the cohort with a column value holding the
predicted risks. The rows must keep the original rowIds:
toSparseM() numbers the matrix rows and keeps the original
in originalRowId.
predictMadeUp <- function(plpModel, data, cohort) {
if (inherits(data, "plpData")) {
matrixObjects <- toSparseM(
plpData = data,
cohort = cohort,
map = plpModel$covariateImportance %>%
dplyr::select("columnId", "covariateId")
)
newData <- matrixObjects$dataMatrix
cohort <- matrixObjects$labels
} else {
newData <- data
}
if (inherits(plpModel, "plpModel")) {
model <- plpModel$model
} else {
model <- plpModel
}
prediction <- cohort
prediction$value <- stats::predict(model, newData)
# give back the original rowIds
prediction <- prediction %>%
dplyr::select(-"rowId") %>%
dplyr::rename(rowId = "originalRowId")
attr(prediction, "metaData") <- list(modelType = attr(plpModel, "modelType"))
return(prediction)
}As the madeUp model uses the standard R prediction, it could reuse
xgboost’s prediction function instead:
predict = "PatientLevelPrediction::predictXgboost".
Variable importance
The variableImportance hook gets the fitted model and a data.frame called covariateMap with the covariateId and columnId of each column of the feature matrix. It returns a data.frame with the columns covariateId, covariateValue (the importance) and included (1 if the model uses the covariate, otherwise 0). If it fails, the model is still fitted, without importances.
varImpMadeUp <- function(model, covariateMap) {
covariateMap$covariateValue <- madeUpImportance(model)[covariateMap$columnId]
covariateMap$included <- as.numeric(covariateMap$covariateValue != 0)
return(covariateMap[, c("covariateId", "covariateValue", "included")])
}Saving and loading
A save handler is a function that returns a list with:
- save - a function of the fitted model and a file path that writes the model
- load - a function of a file path that reads it back
- fileName - optionally, the name of the file (or folder) to write in
the model folder, a plain name without a path; the default is
"model.json"
saveLoadMadeUp <- function() {
list(
fileName = "model.rds",
save = function(model, file) saveRDS(model, file),
load = function(file) readRDS(file)
)
}savePlpModel() calls save with
file.path(modelFolder, fileName) and
loadPlpModel() calls load with the same path.
Acknowledgments
Considerable work has been dedicated to provide the
PatientLevelPrediction package.
citation("PatientLevelPrediction")## To cite PatientLevelPrediction in publications use:
##
## Reps JM, Schuemie MJ, Suchard MA, Ryan PB, Rijnbeek P (2018). "Design
## and implementation of a standardized framework to generate and
## evaluate patient-level prediction models using observational
## healthcare data." _Journal of the American Medical Informatics
## Association_, *25*(8), 969-975.
## <https://doi.org/10.1093/jamia/ocy032>.
##
## A BibTeX entry for LaTeX users is
##
## @Article{,
## author = {J. M. Reps and M. J. Schuemie and M. A. Suchard and P. B. Ryan and P. Rijnbeek},
## title = {Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data},
## journal = {Journal of the American Medical Informatics Association},
## volume = {25},
## number = {8},
## pages = {969-975},
## year = {2018},
## url = {https://doi.org/10.1093/jamia/ocy032},
## }
Please reference this paper if you use the PLP Package in your work:
This work is supported in part through the National Science Foundation grant IIS 1251151.