NEWS.md
This release includes:
4 new data quality check types have been added in this release:
plausibleStartBeforeEnd
: The number and percent of records with a value in the cdmFieldName field of the cdmTableName that occurs after the date in the plausibleStartBeforeEndFieldName.plausibleAfterBirth
: The number and percent of records with a date value in the cdmFieldName field of the cdmTableName table that occurs prior to birth.plausibleBeforeDeath
: The number and percent of records with a date value in the cdmFieldName field of the cdmTableName table that occurs after death.plausibleGenderUseDescendants
: For descendants of CONCEPT_ID conceptId (conceptName), the number and percent of records associated with patients with an implausible gender (correct gender = plausibleGenderUseDescendants).The 3 temporal plausibilty checks are intended to replace plausibleTemporalAfter
and plausibleDuringLife
, for a more comprehensive and clear approach to various temporality scenarios. plausibleGenderUseDescendants
is intended to replace plausibleGender
, to enhance readability of the DQD results and improve performance. The replaced checks are still available and enabled by default in DQD; however, in a future major release, these checks will be deprecated. Please plan accordingly.
For more information on the new checks, please check the Check Type Definitions documentation page. If you’d like to disable the deprecated checks, please see the suggested check exclusion workflow in our Getting Started code here.
plausibleUnitConceptIds
has been reduced, and the lists of plausible units for those measurements have been re-reviewed and updated for accuracy. This change is intended to improve performance and reliablility of this check. Please file an issue if you would like to contribute additional measurements + plausible units to be checked in the futureplausibleValueLow
thresholds have been corrected to prevent false positive failures from occurringWe have begun an initiative to add more comprehensive user documentation at the data quality check level. A dedicated documentation page is being created for each check type. Each check’s page will include detailed information about how its result is generated and what to do if it fails. Guidance is provided for both ETL developers and data users.
9 pages have been added so far, and the rest will come in a future release. Check them out here and please reach out with feedback as we continue improving our documentation!
This release includes:
A new function writeDBResultsToJson
which can be used to write DQD results previously written to a database table (by setting writeToTable
= TRUE in executeDqChecks
or by using the writeJsonResultsToTable
function) into a JSON file in the standard DQD JSON format.
vocabDatabaseSchema
where appropriateThis release includes:
vroom
This release includes:
The following changes involve updates to the default data quality check threshold files. If you are currently using an older version of DQD and update to v2.4.0, you may see changes in your DQD results. The failure threshold changes are fixes to incorrect thresholds in the v5.4 files and thus should result in more accurate, easier to interpret results. The unit concept ID changes ensure that long-invalid concepts will no longer be accepted as plausible measurement units.
measurePersonCompleteness
and measureValueCompleteness
were fixed in the v5.4 table & field level threshold files. This issue has existed since v5.4 support was initially added in March 2022
measurePersonCompleteness
checks had a threshold of 0 when it should have been 95 or 100measureValueCompleteness
checks had a threshold of 100 when it should have been 0, and many had no threshold (defaulting to 0) when it should have been 100measurePersonCompleteness
for the DEATH table has been toggled to Yes
, with a threshold of 100plausibleUnitConceptIds
have been updated to 720870. Concept 9117 became non-standard and was replaced with concept 720870, on 28-Mar-2022plausibleUnitConceptIds
have been removed. These concepts were deprecated on 05-May-2022convertJsonResultsFileCase
in Shiny app was appended with DataQualityDashboard::
. This prevents potential issues related to package loading and function naming conflictsSome minor refactoring of testthat files and package build configuration and some minor documentation updates were also added in this release.
This release includes:
sqlOnly
and sqlOnlyIncrementalInsert
to TRUE in executeDqChecks
will return (but not run) a set of SQL queries that, when executed, will calculate the results of the DQ checks and insert them into a database table. Additionally, sqlOnlyUnionCount
can be used to specify a number of SQL queries to union for each check type, allowing for parallel execution of these queries and potentially large performance gains. See the SqlOnly vignette for detailsconvertJsonResultsFileCase
can be used to convert the keys in a DQD results JSON file between snakecase and camelcase. This allows reading of v2.1.0+ JSON files in older DQD versions, and other conversions which may be necessary for secondary use of the DQD results file. See function documentation for detailsviewDqDashboard
will now automatically convert the case of pre-v2.1.0 results files to camelcase so that older results files may be viewed in v2.3.0+This release includes:
cohortTableName
parameter added to executeDqChecks
. Allows user to specify the name of the cohort table when running DQD on a cohort. Defaults to "cohort"
YYYYMMDD
to conform to SqlRender standardvocabDatabaseSchema
and cohortDatabaseSchema
where appropriateoutputFile
parameter from DQD setup vignette (variable not set in script)And some minor documentation updates for clarity/accuracy.
This release includes:
offset
column name in v5.4 thresholds file so that this column is skipped by DQD in all cases (use of reserved word causes failures in some SQL dialects)This release includes:
outputFolder
parameter for the executeDqChecks
function is now REQUIRED and no longer has a default value. This may be a breaking change for users who have not specified this parameter in their script to run DQD.
No material changes from v1.4, this adds a correct DESCRIPTION
file with the correct DQD version
This release provides support for CDM v5.4
and incorporates minor bug fixes related to incorrectly assigned checks in the control files.
This fixes a small bug and removes a duplicate record in the concept level checks that was throwing an error.
This release includes additional concept level checks to support the OHDSI Symposium 2020 study-a-thon and bug fixes to the writeJSONToTable
function. This is the release that study-a-thon data partners should use.
This is a bug fix release that updates how notes are viewed in the UI and adds CDM table, field, and check name to the final table.