Analysis of the NKDUC to SNOMED Mapping
This script contains the code and the results of our analysis for our mapping of NKDUC items to SNOMED CT. We analysed the reliability of the mapping and the coverage rate of SNOMED CT for the NKDUC.
The information about the NKDUC and the cross map is available as csv file in the subfolder ./data.
This script includes the following sections:
- Load Data
- Compute Reliability of the mapping (Fleiss Kappa)
- Compute the Reliability of the equivalence Rating
- Compute the Coverage Rate
Importing and Preprocessing the Data
# Mapping
import pandas as pd
from statsmodels.stats.inter_rater import fleiss_kappa
import numpy as np
res = pd.read_csv("./data/mapping-results.csv")
# res.finales_konzept = res.finales_konzept.map({1: True, 1: False}).astype('bool')
# Recode Kapitel
chapters_de = {
1: "Stammdaten",
2: "Allgemeinstatus",
3: "Wundanamnese",
4: "Wundstatus",
5: "Diagnostik",
6: "Therapie",
7: "Patient Related Outcome (PRO)",
8: "Ernährung",
9: "Edukation"
}
chapters_en = {
1: "Master Patient Data",
2: "Allgemeinstatus",
3: "Wound Evaluation",
4: "Wound Status",
5: "Diagnostics",
6: "Therapy",
7: "Patient Related Outcome (PRO)",
8: "Nutritional Evaluation",
9: "Patient Education"
}
res.kapitel = res.kapitel.map(chapters_de)
res.kapitel = res.kapitel.astype('category').cat.as_ordered()
# Compute the number of items in the chapters
n_chapter = res.loc[:, ['kapitelbezeichnung', 'id']].drop_duplicates('id').groupby('kapitelbezeichnung').count()
#n_overall = res.loc[:, ['id']].drop_duplicates('id').shape[0]
#pd.DataFrame({n_overall, index=['Overall']})
n_chapter.loc['Overall'] = n_chapter.sum(axis=0)
n_chapter.columns = ['n']
# Equivalence Labels
equ_cat = res.equ_jens.drop_duplicates().sort_values()
iso_categories = {
'Equivalence of meaning; lexical, as well as conceptual': 1,
'Equivalence of meaning, but with synonymy.': 2,
'Source concept is broader and has a less specific meaning than the target concept': 3,
'Source concept is narrower and has a more specific meaning than the target concept': 4,
'No map is possible': 5,
}
Reliability of the Mapping
Proportional Agreement between Mappers
mapping = res.pivot(index = "id", columns= "id_mapper", values= "snomed_code")
mapping.columns = ["map1", "map2", "map3"]
mapping['agree2'] = mapping.iloc[:, 1] == mapping.iloc[:, 2]
mapping['agree3'] = mapping.iloc[:, 0] == mapping.iloc[:, 2]
mapping['agree1'] = mapping.iloc[:, 0] == mapping.iloc[:, 1]
mapping.loc[:, 'agreement_perc'] = mapping.loc[:, ['agree1', 'agree2', 'agree3']].astype('int').sum(axis=1) / 3
mapping = res.loc[:, ['id', 'kapitelbezeichnung']].merge(right=mapping.reset_index(), on='id')
agree = mapping.loc[:, ['kapitelbezeichnung', 'agreement_perc']].groupby('kapitelbezeichnung').apply(np.mean)
agree.loc['Overall'] = mapping.agreement_perc.mean()
Fleiss-Kappa
def reshape_for_fleiss_kappa(data, index='id', columns='snomed_code', n_rater=3):
"""
Restructures the dataframe to compute the Fleiss Kappa Value
Parameters
----------
data : pandas.DataFrame
A dataframe with columns snomed_code, id (the item id of the nkduc items)
Returns
-------
numpy.array
A numpy array, which columns represent the snomed_ct code and rows an item of the NKDUC
The values in the cells represent the number of raters, who chose the code for the NKDUC items
"""
# Reshape data
# add a column with only ones (to count the number of agreements)
data.loc[:, 'cnt'] = np.array([1 for i in range(data.shape[0])])
# To compute fleiss kappa, the data has to be reshaped into a structure
# in which nkduc items are in the rows and each snomed concept is in a col
agreement = data.pivot_table(values='cnt', index=index, columns=columns, aggfunc='sum', fill_value = 0) # reshape data
# print(np.array(agreement).sum(axis=1))
assert all(np.array(agreement).sum(axis=1) == n_rater) # row sums should all be three (three ratings for each item)
return agreement
d = {}
for chapter in res.kapitelbezeichnung.drop_duplicates().values:
df = res.loc[res.kapitelbezeichnung == chapter, :]
fk_result = fleiss_kappa(reshape_for_fleiss_kappa(df))
d[chapter] = fk_result
d['Overall'] = fleiss_kappa(reshape_for_fleiss_kappa(res))
kappa = pd.DataFrame(d, index=[0]).transpose()
kappa.columns = ['kappa']
/Users/jens/anaconda3/lib/python3.7/site-packages/pandas/core/indexing.py:362: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
self.obj[key] = _infer_fill_value(value)
/Users/jens/anaconda3/lib/python3.7/site-packages/pandas/core/indexing.py:543: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
self.obj[item] = s
mapping = agree.merge(kappa, left_index = True, right_index=True).merge(n_chapter, left_index=True, right_index=True)
# Display Mapping Reliability Results
mapping.style.format({'agreement_perc': "{:.2%}",
'kappa': "{:.3f}"})
| agreement_perc | kappa | n | |
|---|---|---|---|
| kapitelbezeichnung | |||
| 01 Stammdaten | 60.78% | 0.575 | 34 |
| 02 Allgemeinstatus | 75.76% | 0.754 | 66 |
| 03 Wundanamnese | 58.33% | 0.568 | 24 |
| 04 Wundstatus | 38.60% | 0.366 | 57 |
| 05 Diagnostik | 33.33% | 0.280 | 14 |
| 06 Therapie | 39.73% | 0.367 | 73 |
| Overall | 52.36% | 0.512 | 268 |
Table 1: Results of the Mapping, the table presents the proportional agreement for three mapper in the first column and the Fleiss-Kappa Reliability statistic in the second. The third column presents the total number of items in that section of the NKDUC. The last row represents the overall agreement, kappa and total number of items that were mapped in this project.
Analysis of the Equivalence Rating
Fleiss-Kappa Statistic
# Overall Kappa
df1 = res.loc[:, ['equ_jens', 'equ_mareike']].reset_index()
df2 = df1.melt(id_vars='index')
df3 = df2.pivot_table(values='variable', columns='value', aggfunc='count', index='index', fill_value=0)
kappa = {'Overall': fleiss_kappa(df3)}
# Kappa by NKDUC Chapter
for chapter in res.kapitelbezeichnung.drop_duplicates().values:
df1 = res.loc[res.kapitelbezeichnung == chapter, ['equ_jens', 'equ_mareike']].reset_index()
df2 = df1.melt(id_vars='index')
df3 = df2.pivot_table(values='variable', columns='value', aggfunc='count', index='index', fill_value=0)
kappa[chapter] = fleiss_kappa(df3)
kappa = pd.DataFrame(kappa, index=[0]).transpose()
kappa.columns = ['kappa']
Agreement
agreement_chapter = res.loc[:, ['kapitelbezeichnung', 'agreement']].groupby('kapitelbezeichnung').apply(np.mean)
agreement_overall = res.loc[:, ['agreement']].apply(np.mean)
agreement_overall = pd.DataFrame({'Overall': agreement_overall}).transpose()
agreement = agreement_overall.append(agreement_chapter)
# Merge kappa and agreement values
equivalence = pd.merge(kappa, agreement, left_index=True, right_index=True).merge(n_chapter, left_index=True, right_index=True)
equivalence = equivalence.sort_index()
equivalence.loc[:, 'n'] = equivalence.n * 3
# Display Equivalence Rating Results
equivalence.loc[:, ['agreement', 'kappa', 'n']].style.format({'agreement': "{:.2%}", 'kappa': "{:.3f}"})
| agreement | kappa | n | |
|---|---|---|---|
| 01 Stammdaten | 83.33% | 0.772 | 102 |
| 02 Allgemeinstatus | 89.90% | 0.835 | 198 |
| 03 Wundanamnese | 70.83% | 0.583 | 72 |
| 04 Wundstatus | 74.27% | 0.641 | 171 |
| 05 Diagnostik | 59.52% | 0.408 | 42 |
| 06 Therapie | 75.34% | 0.667 | 219 |
| Overall | 78.48% | 0.702 | 804 |
Table 2: Reliability of the Equivalence Rating between two Rater. To evaluate the coverage rate and the quality of the Mapping, two persons conducted an equivalence rating as it is described in the ISO Technical Report 12300. During this task, the two persons independently rated the semantic equivalence of the source NKDUC items and the target SNOMED CT concepts.
Coverage Rate
all(res.loc[:, ['id','finales_konzept']].groupby('id').count().finales_konzept == 3)
False
concept_ids = res.drop_duplicates(['id']).loc[:, ['id']]
df=res.loc[res.finales_konzept==1, ['id','snomed_code', 'finaler_beschluss', 'descriptor','finales_konzept']]
df.head()
df.groupby('id').apply(np.mean).shape[0]
res.loc[:, ['id', 'map']].drop_duplicates().shape
(268, 2)
# Create a dataset that contain the final concepts
final_map=res.loc[res.finales_konzept==1, ['id','snomed_code', 'descriptor', 'equi_final']].drop_duplicates()
final_map=res.drop_duplicates('id').loc[:, ['id', 'kapitelbezeichnung', 'finaler_beschluss']].merge(final_map, on='id', how='left')
final_map.loc[:, 'descriptor'] = final_map['descriptor'].fillna('no map')
final_map.loc[:, 'snomed_code'] = final_map['snomed_code'].fillna('no map')
final_map.loc[:, 'equi_final'] = final_map['equi_final'].fillna('No map is possible')
final_map['snomed_code'] = final_map['snomed_code'].apply(lambda x: False if x == 'no map' else True)
final_map.pivot_table(index='kapitelbezeichnung', values='snomed_code', aggfunc=('count', 'mean')).style.format({'mean': "{:.2%}"})
| count | mean | |
|---|---|---|
| kapitelbezeichnung | ||
| 01 Stammdaten | 34 | 64.71% |
| 02 Allgemeinstatus | 66 | 89.39% |
| 03 Wundanamnese | 24 | 83.33% |
| 04 Wundstatus | 57 | 78.95% |
| 05 Diagnostik | 14 | 78.57% |
| 06 Therapie | 73 | 78.08% |
The table above is not part of the publication, it is a quick overview of the coverage rate, i.e., how many concepts could be matches, regardless of the equality category of the match itself.
cont_table = final_map.pivot_table(index='equi_final', values='snomed_code', columns='kapitelbezeichnung', aggfunc='count')
assert np.array(cont_table.fillna(0)).sum()==268 # Check that 268 items are included in the coverage rate analysis
# Count number of items (equality categories) in each chapter and each categories
df1 = final_map.pivot_table(index=['kapitelbezeichnung', 'equi_final'], values='snomed_code', aggfunc='count').reset_index()
# Add the count of items in each chapter so that the proportion of matches can be computed for
# each chapter and each matching category
# e.g., 12 items in 01-Allgemeinstatus had a match of category 5: no map is possible
df2=df1.merge(n_chapter, right_index=True, left_on='kapitelbezeichnung', how='left')
# This line actually computes the percentage/ coverage rate dependent of the chapter + category (see above)
df2.loc[:, 'coverage_perc']=df2.snomed_code / df2.n
# Pretty print
# Add number of total items and the relative items
l = list()
for i in range(df2.shape[0]):
l.append("{:.1%} (n={})".format(df2.coverage_perc[i], df2.snomed_code[i]))
# Append the computes values as new column to pd.dataframe
df2.loc[:, 'coverage'] = pd.Series(l)
assert all(df2.groupby('kapitelbezeichnung').coverage_perc.sum().values == 1) # check if perc adds up to 1 (100%)
# Actually format for display the data (pivot_wide: Chapters in cols and equi categories in the rows)
df3=df2.pivot(index='equi_final', columns='kapitelbezeichnung', values='coverage').fillna("-").reset_index()
df3.loc[:, 'equi_final'] = df3.equi_final.astype('category')
df3.loc[:, 'equi_final'] = df3['equi_final'].cat.reorder_categories(list(iso_categories.keys()))
df3 = df3.sort_values(by='equi_final')
df3.set_index('equi_final', inplace=True)
# Display Detailled Coverage Rate
display(df3)
| kapitelbezeichnung | 01 Stammdaten | 02 Allgemeinstatus | 03 Wundanamnese | 04 Wundstatus | 05 Diagnostik | 06 Therapie |
|---|---|---|---|---|---|---|
| equi_final | ||||||
| Equivalence of meaning; lexical, as well as conceptual | 23.5% (n=8) | 59.1% (n=39) | 50.0% (n=12) | 43.9% (n=25) | 35.7% (n=5) | 38.4% (n=28) |
| Equivalence of meaning, but with synonymy. | 26.5% (n=9) | 24.2% (n=16) | 25.0% (n=6) | 21.1% (n=12) | 21.4% (n=3) | 23.3% (n=17) |
| Source concept is broader and has a less specific meaning than the target concept | 2.9% (n=1) | 3.0% (n=2) | 4.2% (n=1) | 1.8% (n=1) | - | 1.4% (n=1) |
| Source concept is narrower and has a more specific meaning than the target concept | 11.8% (n=4) | 3.0% (n=2) | - | 10.5% (n=6) | 21.4% (n=3) | 15.1% (n=11) |
| No map is possible | 35.3% (n=12) | 10.6% (n=7) | 20.8% (n=5) | 22.8% (n=13) | 21.4% (n=3) | 21.9% (n=16) |
Appendix 1: The table shows the coverage rates for each category for each section of the NKDUC. In the publication we included the table with aggregated categories, to demonstrate the number of matches (Cat 1 + 2), matches with asymmetry (Cat 3 + 4), and no matches (Cat 5).
This aggregated table is shown below
#df2.loc[:, 'equi_final'] = df2.equi_final.astype('category', categories=list(d.keys()), ordered=True)
iso_cats_agg = {
'Equivalence of meaning; lexical, as well as conceptual': 'Semantic Match present (Degree 1 and 2)',
'Equivalence of meaning, but with synonymy.': 'Semantic Match present (Degree 1 and 2)',
'Source concept is broader and has a less specific meaning than the target concept': 'Semantic Asymmetry present (Degree 3 and 4)',
'Source concept is narrower and has a more specific meaning than the target concept': 'Semantic Asymmetry present (Degree 3 and 4)',
'No map is possible': 'Semantic Match absent (Degree 5)',
}
df3.index = df3.index.map(iso_cats_agg)
# Define the category order for the categorical variable 'Final equivalence rating (equi_final)'
cats = ['Semantic Match present (Degree 1 and 2)', 'Semantic Asymmetry present (Degree 3 and 4)', 'Semantic Match absent (Degree 5)']
coverage = df2.copy(deep=True)
coverage.loc[:, 'equi_final'] = coverage.equi_final.map(iso_cats_agg)
coverage.drop('n', axis=1, inplace=True)
coverage.rename({'snomed_code': 'n'}, axis=1, inplace=True)
coverage = coverage.groupby(['kapitelbezeichnung', 'equi_final']).agg({'n': np.sum, 'coverage_perc': np.sum})
def format_aggregates(data):
l = list()
for i in range(data.shape[0]):
l.append("{:.1%} (n={})".format(data.coverage_perc[i], data.n[i]))
data.loc[:, 'formatted'] = l
return(data)
format_aggregates(coverage).reset_index(inplace=True)
coverage.loc[:, 'equi_final'] = coverage.equi_final.astype('category')
coverage.loc[:, 'equi_final'] = coverage.equi_final.cat.reorder_categories(new_categories=cats, ordered=True)
coverage_chapters = coverage.pivot(index='equi_final', columns='kapitelbezeichnung', values='formatted')
coverage_overall = coverage.set_index('equi_final').groupby(level=0).agg({'n': np.sum})
coverage_overall = coverage_overall.assign(coverage_perc=lambda x: x.n/268)
coverage_overall = coverage_overall.assign(formatted=lambda x: format_aggregates(x).formatted)
coverage_overall.rename({'formatted': 'Overall'}, axis=1, inplace=True)
coverage_overall.loc[:,['Overall']].merge(coverage_chapters, left_index=True, right_index=True, how='left')
| Overall | 01 Stammdaten | 02 Allgemeinstatus | 03 Wundanamnese | 04 Wundstatus | 05 Diagnostik | 06 Therapie | |
|---|---|---|---|---|---|---|---|
| equi_final | |||||||
| Semantic Match present (Degree 1 and 2) | 67.2% (n=180) | 50.0% (n=17) | 83.3% (n=55) | 75.0% (n=18) | 64.9% (n=37) | 57.1% (n=8) | 61.6% (n=45) |
| Semantic Asymmetry present (Degree 3 and 4) | 11.9% (n=32) | 14.7% (n=5) | 6.1% (n=4) | 4.2% (n=1) | 12.3% (n=7) | 21.4% (n=3) | 16.4% (n=12) |
| Semantic Match absent (Degree 5) | 20.9% (n=56) | 35.3% (n=12) | 10.6% (n=7) | 20.8% (n=5) | 22.8% (n=13) | 21.4% (n=3) | 21.9% (n=16) |
Table 3: Coverage rate of the mapping presented using ISO TR 12300 equivalence categories for the complete NKDUC and each of its sections.