fixed broken migrations using “ADD COLUMN”
Fixed broken links in manaual
Fixed issue with Andromeda calls in characterization on R 4.3.x
Fix for Vroom issue on mac
Added support for newer API to OhdsiShinyModules (while maintaining support for old version)
Resolved issue with incidence rates sometimes exporting as null/infinite by setting to 0 in all error cases
Added requirement for Andromeda 0.6.0 (which is implicitly required by DatabaseConnector)
Fixed issue saving cohort subsets to csv file for viewing and analysis in shiny app
Do not run orphan concepts checks for any subset cohorts
Remove use of lookback period for IR calculations - this is now a setting of the call to the package
Added data migration to support subsets in database schema (allow future functionality to take care of them)
Added functionality to
launchDiagnosticsExplorer to make publishing to poist connect/shinyapps.io more straightforward (still requires removal of ggiraph)
Moved most shiny code to
Removed CohortExplorer app as it’s now part of a new package.
Added support for custom FeatureExtraction features
Fixed error when checking for cdm_source table
.data$ usage across package to fix tidyselect warning
Major refactoring of shiny app to use modular code for ease of maintenance
Some tests for shiny modules in
Added support for table prefixes in Diagnostics Explorer databases (e.g. cg_cohort_definition)
Enabled annotation on local instances of DiagnosticsExplorer
Added yaml configuration for diagnostics explorer app, including docs on usage
Storage of version number in database results file to allow future migrations
Slight optimization of cohort characterization queries in diagnostics explorer
Support for database migrations by adding
migrateDataModel functionality. Versions of data generated with CohortDiagnostics 3.0.0 are intended to be future compatible. i.e. if you have an sqlite results file or postgres database generated with version 3.0.0 of cohort diagnostics new shiny app functionality will be supported if you run
migrateDataModel on it.
Changed type to dataType and fieldName to columnName to align with strategus modules
Refactored export of characterization results to use common export
Ensure that tests cases always use a continuous covariate
Batch operations for executing cohort relationship, time series, and feature extraction based diagnostics.
New parameter minCharacterizationMean. This introduces a cut off for the output of FeatureExtraction. In the absence of the parameter the output would have atleast one row for every covariateId in the datasource - most having very low count to be useful for diagnostics.
Fixed issue uploading results to postgres db caused by null values in primary key field. Removed constraint to fix issue.
index_event_breakdown having duplicate entries where concept is observed in the same domain multiple times
Many other issues resolved in shiny codebase
Changed default batch size for characterization feature extraction from 100 to 5 as it was causing performance issues on redshift clusters.
Allow setting of batch size for feature extraction with
options("CohortDiagnostics-FE-batch-size" = batchSize)
Fixed issue with writing csvs caused by update to SqlRender camelCaseToSnakeCase function check that caused execution to crash if parameters were null.
Fixed issue with observation period overflowing sql integer on BigQuery causing execution to crash
Updated old/incorrect documentation on package usage
Fixed bug with new versions of CohortGenerator v0.5.0 causing cohort definition sets in package to not load
Fixed bug in shiny app where multiple runs on the same database would cause the app to crash when selecting database
Time series diagnostics removed
Removed runCohortDiagnostics function - this has now been completely replaced with executeDiagnostics
loadCohortsFromPackage function as this is now replaced with
Removed instantiate cohort functionality,
instantiateCohortSet should now be used with the
inclusionStatisticsFolder parameter, this is now all exported directly from
CohortGenerator without the need to generate this first.
Removed usage of Rdata files in DiagnosticsExplorer shiny app and function to create them
createMergedResultsFile which outputs a shiny app
Added support for any
SqlRender/DatabaseConnector compatible database (note, this is experimental. Postgres and sqlite are the only backends recommended for use in production environments)
Improved metadata collection and storage from runs of cohort diagnostics.
Removed phenotype_id field from data ddl
Additional checks to the output of cohort diagnostics to ensure it conforms to its own results data model. The new function (internal) is makeDataExportable. Results data model csv file has been enhanced with new fields, including a field to specify if the value is to be subjected to privacy protection (i.e. min cell count, eg. person count). Note a bug was discovered in the orphan concepts and included source concepts that was leading to duplication of row records by primary key. This bug has been fixed by calculating its max value grouped by primary keys. It will be fixed in another commit.
New optional diagnostics computes temporal relationship between any two cohorts. The settings for the temporal relationship between cohorts defaults to be the same as temporalCovariateSettings. This diagnostics will be integrated into the characterization output of diagnostics explorer, where cohorts will be covariates.
New optional diagnostics called time series diagnostics. Time series diagnostics takes as input a calendar period range, and in that calendar period range for calendar units (year, quarter, month) computes the approximate new occurrence (approximates incidence) and observations (approximates prevalence) of the cohort start and cohort end dates during the calendar period.
Added new lines in NEWS.md to fix package site.
Updated vignette to fix minor issues
Cohorts with zero counts are now stored in results and display in shiny app
Changes: 1. Added
executeDiagnostics function which aims to replace
runCohortDiagnosticsas the main interface to the package.
Updated vignette on “Running Cohort Diagnostics” to give clearer instructions
Removed vignettes on usage that are no longer required
Improved testing across database platforms
Moved some not particularly useful warnings to
Bug Fixes: 1. User code removed from CohortExplorer to fix issue #618
Fixed bug with
runBreakdownIndexEvents = TRUE failed for drug_era table with SQL error - Issue #695
Fixed error when computing incidence rates on BigQuery.
Fixed error when
cdm_source table is empty (warning remains).
Fixed error when instantiating cohorts on BigQuery.
Minor cosmetic changes to diagnostics explorer shiny app. Typo fix
Fix for warning from type-convert https://github.com/OHDSI/CohortDiagnostics/issues/661
Use Sex instead of Gender in Diagnostics Explorer https://github.com/OHDSI/CohortDiagnostics/issues/676
Improved tool tip
Various improvements to plots for consistent color, axis labels and labels
Visit context table addition
Diagnostic explorer is now a distinct shiny application from phenotype library. PhenotypeExplorer is a stand alone shiny app in package PhenotypeLibrarian.
Lot of UX changes. Reactivity deferred on drop down menus.
Changes to improve app stability.
Index event breakdown now has subject count
Index event breakdown calculates _source_concept_id from source fields in CDM tables.
Vocabulary database schema is now supported.
Metadata (vocabulary version information from data source) is now collected.
OracleTempSchema use deprecated in favor of tempEmulationSchema.
Run against external concept count has been removed, as concept counts data is not available. Function ‘runCohortDiagnosticsUsingExternalCounts’ is removed.
Removed code related to referentConceptId = phenotypeId/1000 as it does not always hold true.
Create cohort table function is now private. Please use instantiate cohort.
checkInputFileEncoding is not exported as a public function (as not the scope of CohortDiagnostics).
Updated results data model to include new tables (resolved_concepts).
Cohort Diagnostics results data model now compliant with standard characterization output.
Support for cohort_censor_stats table in webapi 2.8.1 #387
Add time series diagnostics computation. Output is not in Diagnostics explorer in this version.
Any improvements to help with usability and stability. Informative messages to help with debugbing as needed.
phenotypeDescription is no longer supported as input for cohort diagnostics.
databaseName and databaseDescription should be non NULL
Fixed computation of standard deviation and standard difference of mean for binary covariates.
Many improvements in performance when computing diagnostics. Now allows computation of diagnostics for many (>100) cohorts at once.
The Diagnostics Explorer Shiny app can now also run against results data in a database. Added functions for uploading the diagnostics data to a database. This becomes necessary when dealing with very large data (e.g. more than 100 cohort definitions).
Added ability to group cohort definitions by phenotypes, and add descriptions of phenotypes. Currently the only consequence of grouping by phenotype is that cohort overlap is only computed within a phenotype. Grouping by phenotypes is done by adding a
phenotypeId field to the
cohortsToCreate file, and by adding a
phenotypeDescription file. See the
examplePhenotypeLibraryPackage in the repo for an example.
The cohort overlap plot now uses stacked bar charts instead of a Venn diagram to allow showing multiple comparisons across multiple databases at once.
The cohort characteristics comparison plot now colors by domain, and allows filtering by domain.
Switching from ‘plotly’ to ‘ggiraph’ for interactive plotting.
Added a Visit Context diagnostic.
Fixed error when many concept sets have to be instantiated.
Removed ohdsi/SqlRender from Remotes https://github.com/OHDSI/CohortDiagnostics/issues/189
Fixed Digit precision for RJSONIO::toJson and fromJSON https://github.com/OHDSI/CohortDiagnostics/issues/161 This is an important fix. If digit precision is not explicitly specified in RJSONIO, then scientific notation is used. This issue seems to only happen when an integer id (conceptId, conceptSetId, cohortId etc) >= 10,000,000 (which is rare). Please use this update if you have id’s > 10,000,000.
New features: 1. Minor UI changes to Diagnostics explorer. Added missing sort.
Added better labels for plots.
Bug fixes: 1. Changes dependency to ROhdsiWebApi (>= 1.1.0)
New features: 1. All objects in DiagnosticsExplorer are sorted by default #173
Bug fixes: 1. Ensure concept sets across cohort definitions are unique #174 (changes dependency to ROhdsiWebApi (>= 1.1.0))
Changes: 1. New function to retrieve concept set json from cohort json
New function to retrieve concept set sql from cohort sql
DiagnosticsExplorer shiny app - DataTable now rendered using server side processing. Bug fixes and UI improvements.
DiagnosticsExplorer shiny app - Phenotype library mode (released)
DiagnosticsExplorer shiny app - Combine included source concepts and orphan concepts into one submenu https://github.com/OHDSI/CohortDiagnostics/issues/129
Bug fixes: 1. https://github.com/OHDSI/CohortDiagnostics/issues/167
Changes: 1. Shiny app UI improvements
Link out to Atlas and Athena from cohortId. Supports baseUrl.
(beta - unreleased) support for Phenotype library. Shiny app will look for two additional csv files phenotypeDescription and cohortDescription that put the DiagnosticExplorer in Phenotype Library Mode. Plan to release in future version >= 1.2
Changed default selections for temporal characterization
Added minimum threshold value to covariate_value and temporal_covariate_value with default value = 0 (future release, we plan to make this 0.005 i.e. 0.5%)
Changes: 1. Added temporal characterization
Bug fixes: 1. Circe-be update introduced bug in parsing concept sets in cohort definition.
Error handling: Use checkmate R-package to provide more informative error messages.
Refactor runCohortDiagnostics: added new function by refactoring existing private functions. This new function get the JSON and parameterized OHDSI SQL for the cohorts for which diagnostics has been requested
Note: The code has been partially refactored to depend on tidyverse.
Adapting to new ROhdsiWebApi (>= 1.0.0) interface.
Added log info that provides addition run time details #72
Fixing broken getTimeDistributions function.
Fixing broken instantiateCohort function.
Added database information tab to Diagnostics Explorer Shiny app.
Using Andromeda instead of ff to store large data objects (used for characterization).
Fixed error when cohort definition has no concept sets.
Fixed error in index event breakdown when entry event contained multiple criteria for the same domain.
Fixed error related to tempEmulationSchema argument (needed on Oracle and BigQuery).
Fixed use of deprecated function, causing warnings and causing older versions of ParallelLogger to error.
Fixed error in orphan code SQL (hard-coded database schema).
Fixed bug when concept counts become too big for a regular INT.
Fixed counts of standard concepts shown when using external concept count table.
Orphan code check now resolves concept sets instead of using verbatim concepts. Should include specificity.
Added option to use the same y-axis for all databases in incidence rate plot.
Added runCohortDiagnosticsUsingExternalCounts function.
Added option to use cohort definitions in WebAPI instead of those stored in study package.
Fixed bug causing weird false positives in orphan codes (caused by transforming concept IDs to scientific notation).
Better handling in viewer when some analyses are not executed.
Now also showing cohorts that had zero entries.
Fixed error when there was only 1 cohort.