Skip to content

parent_selector

Stated parent selection for new SNOMED CT clinical concepts.

The RF2 exporter writes new concepts as sufficiently defined, so the classifier can infer fully defined supertypes from the attributes but never primitive ones. The stated parent therefore has to be a primitive concept, ideally the proximal primitive parent.

Allowed parents are the primitive SNOMED descendants of root_concept_id (Clinical finding); the root itself is the fallback.

Algorithm, per source term

  1. Neighbours: the num_neighbours most similar standard SNOMED concepts (embedding search) with similarity >= min_similarity. With exclude_source_descendants the source concept and its descendants are dropped first, because they would not exist for a genuinely new concept.
  2. Candidates: the primitive ancestors of each neighbour (the neighbour itself excluded).
  3. Support: the similarity-weighted share of neighbours below each candidate.
  4. Selection: candidates with support >= min_support, minus any candidate that is an ancestor of another qualifying candidate, deepest first (min_levels_of_separation from the root), capped at max_parents. When neighbours agree this is their proximal primitive parent; when they disagree it is an intermediate primitive.
  5. Fallback: the root when no candidate qualifies.

Usage::

with psycopg.connect(conn_str) as conn:
    stated_parents, details = build_stated_parents_map(
        raw_results, conn, schema, config.hierarchy.parent_selection
    )
# stated_parents: {source_concept_id (or source_key): [parent_sctid, ...]}

PrimitiveAncestry dataclass

Strict primitive ancestors per concept, restricted to the allowed parent set.

Source code in src/ariadne/hierarchy/parent_selector.py
76
77
78
79
80
81
82
83
84
@dataclass
class PrimitiveAncestry:
    """Strict primitive ancestors per concept, restricted to the allowed parent set."""

    concepts: dict[int, PrimitiveConcept] = field(default_factory=dict)
    ancestors: dict[int, set[int]] = field(default_factory=dict)

    def is_ancestor(self, ancestor_id: int, concept_id: int) -> bool:
        return ancestor_id in self.ancestors.get(concept_id, ())

build_stated_parents_map(raw_results, conn, schema, settings, *, neighbours=None)

Select stated parents for every entry of the hierarchy pipeline results.

Parameters:

Name Type Description Default
raw_results list[dict]

Result dicts from process_hierarchy (hierarchy_results_raw.json). Uses source_concept_id (or source_key) and source_term.

required
conn Connection

Open psycopg connection to the vocabulary database.

required
schema str

OHDSI Vocabularies schema name.

required
settings ParentSelectionConfig

config.hierarchy.parent_selection.

required
neighbours dict[str, list[Neighbour]] | None

Pre-computed {source_term: [Neighbour, ...]}; searched (and cached) when omitted.

None

Returns:

Type Description
dict[int | str, list[str]]

(stated_parents, details). stated_parents maps each source key to parent SCTIDs, as

DataFrame

expected by export_to_rf2(stated_parent=...). details has one row per selected parent

tuple[dict[int | str, list[str]], DataFrame]

with its support, depth and whether it is the fallback.

Source code in src/ariadne/hierarchy/parent_selector.py
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
def build_stated_parents_map(
    raw_results: list[dict],
    conn: psycopg.Connection,
    schema: str,
    settings: ParentSelectionConfig,
    *,
    neighbours: dict[str, list[Neighbour]] | None = None,
) -> tuple[dict[int | str, list[str]], pd.DataFrame]:
    """Select stated parents for every entry of the hierarchy pipeline results.

    Args:
        raw_results: Result dicts from ``process_hierarchy`` (``hierarchy_results_raw.json``). Uses
            ``source_concept_id`` (or ``source_key``) and ``source_term``.
        conn: Open psycopg connection to the vocabulary database.
        schema: OHDSI Vocabularies schema name.
        settings: ``config.hierarchy.parent_selection``.
        neighbours: Pre-computed ``{source_term: [Neighbour, ...]}``; searched (and cached) when omitted.

    Returns:
        ``(stated_parents, details)``. *stated_parents* maps each source key to parent SCTIDs, as
        expected by ``export_to_rf2(stated_parent=...)``. *details* has one row per selected parent
        with its support, depth and whether it is the fallback.
    """
    entries = [(key, _source_term(e)) for e in raw_results if (key := _source_key(e)) is not None]
    if neighbours is None:
        neighbours = retrieve_neighbours([term for _, term in entries], settings)

    excluded: dict[int, set[int]] = defaultdict(set)
    if settings.exclude_source_descendants:
        source_ids = [key for key, _ in entries if isinstance(key, int)]
        pool_ids = {n.concept_id for hits in neighbours.values() for n in hits}
        excluded = fetch_descendants(source_ids, pool_ids, conn, schema)
        for sid in source_ids:
            excluded.setdefault(sid, set()).add(sid)

    usable: dict[int | str, list[Neighbour]] = {}
    for key, term in entries:
        skip = excluded.get(key, set()) if isinstance(key, int) else set()
        usable[key] = [
            n for n in neighbours.get(term, []) if n.concept_id not in skip and n.similarity >= settings.min_similarity
        ][: settings.num_neighbours]

    ancestry = fetch_primitive_ancestry(
        {n.concept_id for hits in usable.values() for n in hits}, conn, schema, settings
    )
    root = fetch_root_concept(conn, schema, settings)

    stated_parents: dict[int | str, list[str]] = {}
    rows = []
    for key, term in entries:
        parents = select_primitive_parents(
            usable[key], ancestry, min_support=settings.min_support, max_parents=settings.max_parents
        )
        if not parents:
            parents = [ParentCandidate(root, 0.0)]
        is_fallback = all(p.concept.concept_id == root.concept_id for p in parents)
        stated_parents[key] = [p.concept.concept_code for p in parents]
        for rank, parent in enumerate(parents, start=1):
            rows.append(
                {
                    "source_concept_id": key,
                    "source_term": term,
                    "parent_rank": rank,
                    "parent_concept_id": parent.concept.concept_id,
                    "parent_concept_code": parent.concept.concept_code,
                    "parent_concept_name": parent.concept.concept_name,
                    "parent_depth": parent.concept.depth,
                    "support": round(parent.support, 4),
                    "num_neighbours": len(usable[key]),
                    "is_fallback": is_fallback,
                }
            )

    details = pd.DataFrame(rows)
    fallback_count = int(details.drop_duplicates("source_concept_id")["is_fallback"].sum()) if rows else 0
    logger.info(
        "Stated parents: %d terms with a primitive parent below the root, %d fell back to %s.",
        len(stated_parents) - fallback_count,
        fallback_count,
        root.concept_name,
    )
    return stated_parents, details

fetch_descendants(ancestor_ids, candidate_ids, conn, schema)

Return {ancestor_id: {candidate_id, ...}} for candidates that descend from (or equal) each ancestor.

Source code in src/ariadne/hierarchy/parent_selector.py
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
def fetch_descendants(
    ancestor_ids: Iterable[int],
    candidate_ids: Iterable[int],
    conn: psycopg.Connection,
    schema: str,
) -> dict[int, set[int]]:
    """Return ``{ancestor_id: {candidate_id, ...}}`` for candidates that descend from (or equal) each ancestor."""
    ancestor_ids = sorted({int(a) for a in ancestor_ids})
    candidate_ids = sorted({int(c) for c in candidate_ids})
    result: dict[int, set[int]] = defaultdict(set)
    if not ancestor_ids or not candidate_ids:
        return result
    query = sql.SQL(
        """
        SELECT ancestor_concept_id, descendant_concept_id
        FROM {schema}.concept_ancestor
        WHERE ancestor_concept_id = ANY(%s)
          AND descendant_concept_id = ANY(%s)
        """
    ).format(schema=sql.Identifier(schema))
    with conn.cursor() as cur:
        cur.execute(query, (ancestor_ids, candidate_ids))
        for ancestor_id, descendant_id in cur.fetchall():
            result[ancestor_id].add(descendant_id)
    return result

fetch_primitive_ancestry(concept_ids, conn, schema, settings)

Fetch the strict primitive ancestors of concept_ids and, recursively, of those ancestors.

The ancestors of the ancestors are needed to tell which candidate parents subsume each other.

Source code in src/ariadne/hierarchy/parent_selector.py
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
def fetch_primitive_ancestry(
    concept_ids: Iterable[int],
    conn: psycopg.Connection,
    schema: str,
    settings: ParentSelectionConfig,
) -> PrimitiveAncestry:
    """Fetch the strict primitive ancestors of *concept_ids* and, recursively, of those ancestors.

    The ancestors of the ancestors are needed to tell which candidate parents subsume each other.
    """
    query = sql.SQL(
        """
        WITH primitive_concept AS (
            SELECT c.concept_id,
                   c.concept_code,
                   c.concept_name,
                   root.min_levels_of_separation AS depth
            FROM {schema}.concept c
            JOIN {schema}.concept_ancestor root
              ON root.descendant_concept_id = c.concept_id
             AND root.ancestor_concept_id = %s
            JOIN {schema}.concept_relationship cr
              ON cr.concept_id_1 = c.concept_id
             AND cr.concept_id_2 = %s
             AND cr.relationship_id = 'Has status'
             AND cr.invalid_reason IS NULL
            WHERE c.vocabulary_id = 'SNOMED'
        )
        SELECT ca.descendant_concept_id,
               p.concept_id,
               p.concept_code,
               p.concept_name,
               p.depth
        FROM {schema}.concept_ancestor ca
        JOIN primitive_concept p
          ON p.concept_id = ca.ancestor_concept_id
        WHERE ca.descendant_concept_id = ANY(%s)
          AND ca.min_levels_of_separation >= 1
        """
    ).format(schema=sql.Identifier(schema))

    ancestry = PrimitiveAncestry()
    pending = {int(cid) for cid in concept_ids}
    with conn.cursor() as cur:
        while pending:
            cur.execute(
                query,
                (settings.root_concept_id, settings.primitive_status_concept_id, sorted(pending)),
            )
            for cid in pending:
                ancestry.ancestors.setdefault(cid, set())
            for descendant_id, parent_id, code, name, depth in cur.fetchall():
                ancestry.ancestors[descendant_id].add(parent_id)
                ancestry.concepts[parent_id] = PrimitiveConcept(parent_id, str(code), name, int(depth))
            pending = set(ancestry.concepts) - set(ancestry.ancestors)
    return ancestry

most_specific(concept_ids, ancestry)

Drop every concept that is an ancestor of another concept in the set.

Source code in src/ariadne/hierarchy/parent_selector.py
92
93
94
95
def most_specific(concept_ids: Iterable[int], ancestry: PrimitiveAncestry) -> set[int]:
    """Drop every concept that is an ancestor of another concept in the set."""
    ids = set(concept_ids)
    return {a for a in ids if not any(ancestry.is_ancestor(a, b) for b in ids if b != a)}

proximal_primitive_parents(concept_id, ancestry)

Proximal primitive parents of an existing concept, deepest first.

Source code in src/ariadne/hierarchy/parent_selector.py
133
134
135
136
def proximal_primitive_parents(concept_id: int, ancestry: PrimitiveAncestry) -> list[PrimitiveConcept]:
    """Proximal primitive parents of an existing concept, deepest first."""
    parent_ids = most_specific(ancestry.ancestors.get(concept_id, ()), ancestry)
    return sorted((ancestry.concepts[pid] for pid in parent_ids), key=lambda c: (-c.depth, c.concept_id))

retrieve_neighbours(terms, settings)

Return the candidate_pool_size most similar standard SNOMED concepts per term.

Results are cached in settings.neighbour_cache_file; only uncached terms are searched.

Source code in src/ariadne/hierarchy/parent_selector.py
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
def retrieve_neighbours(terms: Sequence[str], settings: ParentSelectionConfig) -> dict[str, list[Neighbour]]:
    """Return the ``candidate_pool_size`` most similar standard SNOMED concepts per term.

    Results are cached in ``settings.neighbour_cache_file``; only uncached terms are searched.
    """
    cache = _load_neighbour_cache(settings)
    missing = [term for term in dict.fromkeys(terms) if term not in cache]
    if not missing:
        return {term: cache[term] for term in terms}

    logger.info("Searching neighbours for %d terms (%d cached)…", len(missing), len(cache))
    search_settings = PgvectorSearchSettings(
        max_candidates=settings.candidate_pool_size,
        filter=ConceptFilterSettings(standard_concept=["S"], vocabulary_ids=["SNOMED"]),
        include_synonyms=False,
        include_mapped_terms=True,
    )
    local = threading.local()
    searchers: list[PgvectorConceptSearcher] = []
    searchers_lock = threading.Lock()

    def search(term: str) -> list[Neighbour]:
        if not hasattr(local, "searcher"):
            local.searcher = PgvectorConceptSearcher(settings=search_settings)
            with searchers_lock:
                searchers.append(local.searcher)
        result_df = local.searcher.search_term(term)
        if result_df is None:
            return []
        # PgvectorConceptSearcher returns cosine distance; convert to similarity.
        return [
            Neighbour(int(row.concept_id), str(row.concept_name), 1.0 - float(row.score))
            for row in result_df.itertuples(index=False)
        ]

    try:
        for done, (idx, hits) in enumerate(run_concurrently(missing, search, settings.max_workers), start=1):
            cache[missing[idx]] = hits
            if done % 50 == 0:
                logger.info("  %d / %d terms searched", done, len(missing))
    finally:
        cost = sum(s.get_total_cost() for s in searchers)
        for searcher in searchers:
            searcher.close()
        _save_neighbour_cache(cache, settings)
        logger.info("Neighbour search embedding cost: $%.6f", cost)

    return {term: cache[term] for term in terms}

select_primitive_parents(neighbours, ancestry, *, min_support, max_parents)

Pick the deepest primitive ancestors shared by enough of the neighbours.

Parameters:

Name Type Description Default
neighbours Sequence[Neighbour]

Usable neighbours (already filtered for exclusions and similarity).

required
ancestry PrimitiveAncestry

Primitive ancestors of the neighbours and of every candidate parent.

required
min_support float

Minimum similarity-weighted share of neighbours below a candidate.

required
max_parents int

Maximum number of parents returned.

required

Returns:

Type Description
list[ParentCandidate]

Parents ordered deepest first, then by support. Empty when nothing qualifies.

Source code in src/ariadne/hierarchy/parent_selector.py
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
def select_primitive_parents(
    neighbours: Sequence[Neighbour],
    ancestry: PrimitiveAncestry,
    *,
    min_support: float,
    max_parents: int,
) -> list[ParentCandidate]:
    """Pick the deepest primitive ancestors shared by enough of the neighbours.

    Args:
        neighbours: Usable neighbours (already filtered for exclusions and similarity).
        ancestry: Primitive ancestors of the neighbours and of every candidate parent.
        min_support: Minimum similarity-weighted share of neighbours below a candidate.
        max_parents: Maximum number of parents returned.

    Returns:
        Parents ordered deepest first, then by support. Empty when nothing qualifies.
    """
    total = sum(n.similarity for n in neighbours)
    if total <= 0:
        return []

    weight: dict[int, float] = defaultdict(float)
    for neighbour in neighbours:
        for parent_id in ancestry.ancestors.get(neighbour.concept_id, ()):
            weight[parent_id] += neighbour.similarity

    qualifying = {pid for pid, w in weight.items() if w / total >= min_support - _SUPPORT_TOLERANCE}
    ranked = sorted(
        most_specific(qualifying, ancestry),
        key=lambda pid: (-ancestry.concepts[pid].depth, -weight[pid], pid),
    )
    return [ParentCandidate(ancestry.concepts[pid], weight[pid] / total) for pid in ranked[:max_parents]]