Research

JAMA Internal Medicine | Original Investigation

Development and Validation of a Tool to Identify Patients

With Type 2 Diabetes at High Risk of Hypoglycemia-Related
Emergency Department or Hospital Use
Andrew J. Karter, PhD; E. Margaret Warton, MPH; Kasia J. Lipska, MD, MHS; James D. Ralston, MD, MPH;
Howard H. Moffet, MPH; Geoffrey G. Jackson, MHA; Elbert S. Huang, MD, MPH; Donald R. Miller, ScD

Supplemental content
IMPORTANCE Hypoglycemia-related emergency department (ED) or hospital use among
patients with type 2 diabetes (T2D) is clinically significant and possibly preventable.

OBJECTIVE To develop and validate a tool to categorize risk of hypoglycemic-related

utilization in patients with T2D.

DESIGN, SETTING, AND PARTICIPANTS Using recursive partitioning with a split-sample design,
we created a classification tree based on potential predictors of hypoglycemia-related ED or
hospital use. The resulting model was transcribed into a tool for practical application and
tested in 1 internal and 2 fully independent, external samples. Development and internal
testing was conducted in a split sample of 206 435 patients with T2D from Kaiser
Permanente Northern California (KPNC), an integrated health care system. The tool was
externally tested in 1 335 966 Veterans Health Administration and 14 972 Group Health
Cooperative patients with T2D.

EXPOSURES Based on a literature review, we identified 156 candidate predictor variables

(prebaseline exposures) using data collected from electronic medical records.

MAIN OUTCOMES AND MEASURES Hypoglycemia-related ED or hospital use during 12 months

of follow-up.

RESULTS The derivation sample (n = 165 148) had a mean (SD) age of 63.9 (13.0) years and
included 78 576 (47.6%) women. The crude annual rate of at least 1 hypoglycemia-related ED
or hospital encounter in the KPNC derivation sample was 0.49%. The resulting hypoglycemia
risk stratification tool required 6 patient-specific inputs: number of prior episodes of
hypoglycemia-related utilization, insulin use, sulfonylurea use, prior year ED use, chronic
kidney disease stage, and age. We categorized the predicted 12-month risk of any
hypoglycemia-related utilization as high (>5%), intermediate (1%-5%), or low (<1%). In the
internal validation sample, 2.0%, 10.7%, and 87.3% were categorized as high, intermediate,
and low risk, respectively, with observed 12-month hypoglycemia-related utilization rates of
6.7%, 1.4%, and 0.2%, respectively. There was good discrimination in the internal validation
KPNC sample (C statistic = 0.83) and both external validation samples (Veterans Health
Administration: C statistic = 0.81; Group Health Cooperative: C statistic = 0.79).

CONCLUSIONS AND RELEVANCE This hypoglycemia risk stratification tool categorizes the
12-month risk of hypoglycemia-related utilization in patients with T2D using only 6 inputs.
This tool could facilitate targeted population management interventions, potentially reducing
hypoglycemia risk and improving patient safety and quality of life.

Author Affiliations: Author

affiliations are listed at the end of this
article.
Corresponding Author: Andrew J.
Karter, PhD, Division of Research,
Kaiser Permanente Northern
JAMA Intern Med. doi:10.1001/jamainternmed.2017.3844 California, 2000 Broadway, Oakland,
Published online August 21, 2017. CA 94612 (andy.j.karter@kp.org).

(Reprinted) E1
2017 American Medical Association. All rights reserved.

Downloaded From: http://archinte.jamanetwork.com/ by a Columbia University User on 08/21/2017

 Research Original Investigation
A
dvances in diabetes clinical care and medical treat-
ment have reduced the risk of long-term complica-
tions and mortality for the more than 25 million Ameri-
cans who have diabetes.1 However, iatrogenic hypoglycemia
associated with glucose-lowering medication use has become
a critical public health and drug safety concern.2-4 Severe hy-
poglycemia is defined as an event necessitating assistance from
another person to actively administer carbohydrates, gluca-
gon, or other resuscitative actions.5 Such assistance is often ren-
dered professionally in emergency department (ED) or hospi-
tal encounters and is captured as hypoglycemia-related
utilization.6
Whereas the risk of severe hypoglycemia is known to be
elevated in patients with type 1 diabetes, the risk has been his-
torically underappreciated among patients with type 2 diabe-
tes (T2D), which make up most of the diabetes population. Hy-
poglycemia is now one of the most frequent adverse events in
patients with T2D and is more common than acute hypergly-
cemic emergencies (eg, hyperosmolar hyperglycemic state),7
particularly among older patients and those with a longer
history of diabetes.8 One in 4 emergency hospitalizations for
adverse drug events is related to hypoglycemia, and these rates
are higher in older patients.9 Severe hypoglycemia has been
associated with falls and automobile accidents,10 cardiovas-
cular autonomic dysfunction and ventricular arrhythmia,11
dementia,12 and death.13,14 Patients report that fear of hypo-
glycemia can dissuade them from initiating newly prescribed
insulin.15 Hypoglycemia is also strongly predictive of poorer
health-related quality of life16 and more diabetes distress.17 Hy-
poglycemia-related utilization is costly; total annual direct
medical costs were estimated at approximately $1.8 billion in
2009 in the United States.18
The risk of hypoglycemia varies widely in patients with
T2D. 19 Whereas interventions to prevent hypoglycemia
exist,20-24 there are no validated methods to target these
interventions efficiently. Accordingly, we developed and
validated a hypoglycemia risk stratification tool to catego-
rize 12-month risk of hypoglycemia-related emergency
department (ED) or hospital use among patients with T2D.
This study was approved by the institutional review boards
of Kaiser Permanente, the Bedford Veterans Health Admin-
istration, and Group Health Cooperative; the requirement
that informed consent be obtained from study participants
was waived.
Methods
Study Design
We used a prospective cohort study design to develop a risk
tool to categorize the 12-month risk of hypoglycemia-related
ED or hospital use. Selection of prebaseline candidate predic-
tors was based on a literature review of clinical risk factors as-
sociated with hypoglycemia and limited to data typically avail-
able in electronic medical records (EMRs). We derived and
internally validated this tool in a split sample (4:1) of 206 435
adult patients with T2D in an integrated health care delivery
system (Kaiser Permanente Northern California [KPNC]) using
E2 JAMA Internal Medicine Published online August 21, 2017 (Reprinted)
2017 American Medical Association. All rights reserved.
Downloaded From: http://archinte.jamanetwork.com/ by a Columbia University User on 08/21/2017
Identifying Patients at Risk of Hypoglycemia-Related Use
Key Points
Question Can electronic medical records be used to reliably
categorize risk of future hypoglycemia-related emergency
department or hospital use in patients with type 2 diabetes?
Findings We developed and validated a risk stratification tool that
categorized patients 12-month risk of hypoglycemia-related
utilization using only 6 electronic medical recordbased inputs
(patient history of hypoglycemia-related utilization, insulin use,
sulfonylurea use, emergency department use, chronic kidney
disease, and age). Tool performance was validated in 2 fully
independent populations.
Meaning This hypoglycemia risk stratification tool could facilitate
efficient targeting of population management interventions to
reduce hypoglycemia risk and improve patient safety.
clinical and demographic data from EMRs. We used recursive
partitioning in the derivation sample to create a risk classifi-
cation tree. The classification tree leaf nodes were further cat-
egorized into high-, intermediate-, or low-risk groups on the
basis of predicted risk and then transcribed into the hypogly-
cemia risk stratification tool. After testing the tool in the in-
ternal sample, we conducted external validation in 2 com-
pletely independent samples of patients with T2D from the
Veterans Administration Diabetes Epidemiology Cohort
(DEpiC) (VA sample, n = 1 335 966)25 and from Group Health
Cooperative (GH sample, n = 14 972).
Study Population
Using EMR data from KPNC, we identified 233 330 adults (21
years as of the baseline date of January 1, 2014) with diabetes
with continuous health plan membership for 24 months pre-
baseline and pharmacy benefits for 12 months prebaseline. We
excluded 24 719 patients with unknown diabetes type and 3615
with probable type 1 diabetes according to an algorithm (based
on age of onset <30 years and use of insulin alone26). The re-
maining 206 435 eligible patients with T2D were randomly split
into an 80% derivation sample (n = 165 148) for tool develop-
ment and a 20% internal validation sample (n = 41 287). Simi-
lar eligibility criteria were applied in the creation of the 2 ex-
ternal validation samples.
Outcome
Our outcome was the occurrence of any hypoglycemia-
related ED or hospital use during 12 months postbaseline. This
was defined by having any ED visit with a primary diagnosis
of hypoglycemia or a hospitalization with a principal diagno-
sis of hypoglycemia. Hypoglycemia cases were ascertained ac-
cording to a validated definition27 (any of the following Inter-
national Classification of Diseases, Ninth Revision (ICD-9), codes:
251.0, 251.1, 251.2, 962.3, or 250.8, without concurrent 259.8,
272.7, 681.XX, 682.XX, 686.9X, 707.1-707.9, 709.3, 730.0-
730.2, or 731.8 codes). Secondary discharge diagnoses for hy-
poglycemia were not used because they are often attribut-
able to events that occurred during the ED or hospital encounter
(eg, inpatient insulin management, sepsis, acute renal
failure28).
jamainternalmedicine.com
 Identifying Patients at Risk of Hypoglycemia-Related Use
Exposures
On the basis of a literature review of clinical risk factors asso-
ciated with hypoglycemia in T2D, we selected 156 (122 cat-
egorical and 34 continuous) candidate clinical, demographic,
and behavioral predictor variables for model development
(eTable 1 in the Supplement). To increase the utility of our pre-
diction model in other health care settings (transportability,
usability29), we excluded variables that were expensive or
impractical to collect, had ambiguous meanings, or were not
typically available in an EMR. Medication exposures were based
on pharmacy dispensings during 6 months prebaseline; labo-
ratory values were based on the last test results within 2 years
prebaseline; and prior medical events (eg, history of hypogly-
cemia-related utilization using the same outcome definition)
were based on all available prebaseline records (maximum, 18
years of medical history; mean [SD], 16.5 [3.4] years).
Statistical Analysis
We used standard methodology for model development in-
cluding a split sample and internal and external validation.30-33
We first regressed the outcome (any hypoglycemia-related uti-
lization) on each of the 156 candidate predictors using uni-
variate logistic regression models to generate odds ratios. We
selected candidate variables that had a resulting P < .10. We
then used recursive partitioning (using SAS JMP, version 1234)
on the selected candidates to construct a binary classifica-
tion tree to predict the occurrence of at least 1 hypoglycemia-
related utilization episode 12 months postbaseline. Recur-
sive partitioning is widely used to generate clinical decision
support tools.32,35,36 This method uses a machine-learning,
nonlinear, and nonparametric approach to split (partition)
events into pairs of subgroups based on continuous or cat-
egorical predictors, and has the unique advantage that it iden-
tifies complex nested interactions, unlike linear modeling
methods. Recursive partitioning also optimizes cut points
rather than relying on prespecification. Thus, the resultant clas-
sification tree identifies predictors that may be important for
1 segment of the population but not others, as well as identi-
fying critical thresholds in continuous or ordinal predictors.
We pruned branches from the classification tree in an at-
tempt to optimize predictive accuracy (performance), model
simplicity, practicality of implementation, and intuitive clini-
cal interpretation.31,33 Overly complex models (overfitting),
while potentially offering somewhat greater precision, may be
less practical, increase the decision and classification costs (ex-
pense and time of compiling the predictors), and introduce
propagated error associated with predictors measured with un-
certainty.
Validation Studies
Model accuracy was assessed in the internal validation sample
using standard metrics.30-32(pp255-310),37-39 Discrimination, the
ability of a model to accurately distinguish between subjects
who do vs do not develop the outcome, is based on the area
under the receiver-operator curve (C statistic), with greater than
0.7 classified as good discrimination. We also visually as-
sessed calibration (the extent to which the predicted risks over-
or underestimate the observed risks) using calibration plots.
jamainternalmedicine.com
2017 American Medical Association. All rights reserved.
Downloaded From: http://archinte.jamanetwork.com/ by a Columbia University User on 08/21/2017
Original Investigation Research
Given that our goal was not to quantify the numeric probabil-
ity of a hypoglycemia episode for a given patient but rather to
stratify our population into categories of risk, we focused on
model discrimination over calibration.33,40
We further evaluated components of generalizability33,39
(ie, reproducibility in patients not used for the derivation of
the model) and transportability (practical application in dif-
ferent settings) in the 2 external primary care T2D popula-
tions (VA and GH samples).32,33,41 Because these samples in-
cluded a different disease severity and case mix of patients with
T2D from distinct geographical locations with different meth-
ods for identifying patients with diabetes, the validation ex-
ercises also tested the spectrum, geographic, and method-
ological transportability of the model.32,33
Sensitivity Analyses
We developed this risk stratification tool using all of the pa-
tients available medical histories at KPNC (up to 18 years); how-
ever, long enrollment may be uncommon in other health care
settings. Thus, we conducted further analyses to evaluate
whether our tool was sensitive to restrictions in available length
of enrollment (prebaseline period transportability33). We also
evaluated temporal sensitivity (historical transportability33) of
the tool by applying it to KPNC data in the subsequent year
(using a baseline date of January 1, 2015). This tool was opti-
mized for patients with T2D, and because not all health care
settings can reliably determine diabetes type from their EMR,
we also evaluated tool performance when patients with type
1 diabetes were included (ie, sensitivity to misclassification).
Finally, as a measure of ecological validity of this tool, we evalu-
ated the association between the predicted level of risk of
hypoglycemia-related utilization and actual, self-reported
severe hypoglycemia events based on a 2005 survey42 of 15 231
patients with T2D.
Results
Model Selection
The final classification tree was based on 6 patient-specific vari-
ables: total number of prior episodes of hypoglycemia-
related ED or hospital utilization (0, 1-2, 3 times), number of
ED encounters for any reason in the prior 12 months (<2, 2
times), insulin use (yes/no), sulfonylurea use (yes/no), pres-
ence of severe or end-stage kidney disease (dialysis or chronic
kidney disease stage 4 or 5 determined by estimated glomer-
ular filtration rate of 29 mL/min/1.73 m2 calculated by the
Chronic Kidney Disease Epidemiology Collaboration creati-
nine equation)43 (yes/no), and age younger than 77 years (yes/
no) (Figure 1). This classification tree resulted in 10 mutually
exclusive leaf nodes, each yielding an estimated annual risk
of hypoglycemia-related utilization, which were categorized
as high (>5%), intermediate (1%-5%), or low (<1%). In the KPNC
internal validation sample, 2.0% were categorized as high risk,
10.7% as intermediate risk, and 87.3% as low risk.
We then transcribed the classification model into a simple,
checklist style, hypoglycemia risk stratification tool by map-
ping the combinations of risk factors to high, intermediate, or
(Reprinted) JAMA Internal Medicine Published online August 21, 2017 E3
 Research Original Investigation
Figure 1. Classification Tree for Hypoglycemia-Related Emergency Department (ED) or Hospital Use
165 148 KPNC Derivation Sample
496 3 Prior hypoglycemia-related 4948 1-2 Prior hypoglycemia-related
ED or hospital encounters ED or hospital encounters
Observed rate = 14.9%
2753 Insulin-Yes 2195 Insulin-No
Observed rate = 5.1% Observed rate = 2.0%
42 539 Sulfonylurea-Yes 86 720 Sulfonylurea-No
Observed rate = 0.1%
34 710 Age <77 y 7829 Age 77 y
Observed rate = 1.1%
34 174 Stage 4 or 5 CKD-No 536 Stage 4 or 5 CKD-Yes
Observed rate = 0.3% Observed rate = 2.8%
Hypoglycemic-related utilization was defined by having any ED visit with a
primary diagnosis of hypoglycemia or a hospitalization with a principal diagnosis
of hypoglycemia. Hypoglycemia cases were ascertained with any of the
following International Classification of Diseases, Ninth Revision, codes: 251.0,
251.1, 251.2, 962.3, or 250.8, without concurrent 259.8, 272.7, 681.XX, 682.XX,
686.9X, 707.1-707.9, 709.3, 730.0-730.2, or 731.8 codes.27 The classification
tree was developed using the 808 out of 165 148 T2D adults (derivation
sample) from Kaiser Permanente who had such utilization (4.9 events per 1000
low risk of having any hypoglycemia-related utilization in the
following 12 months (Figure 2). This tool instructs the user to
identify only 1 of 6 mutually exclusive options, where the first
5 are each defined by a unique combination of predictor vari-
ables, and the sixth option is indicated only after ruling out
all other options (eTable 2 in the Supplement provides the
source code).
Patient Characteristics
We compared the distribution of the 6 predictor variables in
our derivation and validation samples (Table 1). There were no
significant differences in the distribution of the 6 predictors
between the KPNC derivation vs validation samples, but there
were significant differences across external validation samples.
The proportion of men and women was similar in the KPNC
and GH samples, while the VA sample was predominantly men.
The mean age was similar across sites, although the propor-
tion older than 77 years was greater in the VA (24.6%), fol-
lowed by KPNC (17.8%), and GH samples (13.2%). The VA
sample had the highest proportion of patients with severe or
end-stage kidney disease (3.7%), and 3 or more prior hypogly-
cemic events (0.4%) vs 2.0% and 0.1%, respectively, in GH. In-
sulin use was lower in the KPNC samples (20.3%) compared
with the GH (30.9%) and VA samples (30.2%). Sulfonylurea use
was higher in the KPNC samples (34.9%) compared with the
VA (25.0%) and GH (22.5%) samples. The observed annual rate
E4 JAMA Internal Medicine Published online August 21, 2017 (Reprinted)
2017 American Medical Association. All rights reserved.
Downloaded From: http://archinte.jamanetwork.com/ by a Columbia University User on 08/21/2017
Identifying Patients at Risk of Hypoglycemia-Related Use
>5% (High risk) 1%-5% (Intermediate risk) <1% (Low risk)
159 704 No prior hypoglycemia-related
ED or hospital encounters
129 259 Insulin-No 30 445 Insulin-Yes
4230 2 ED visits prior year 26 215 <2 ED visits prior year
Observed rate = 2.1%
23 018 Age <77 y 3197 Age 77 y
Observed rate = 0.7% Observed rate = 1.7%
person-years) in 2014. The classification is based on 6 predictor variables from
the electronic medical record and resulted in 10 mutually exclusive leaf nodes.
The criterion for each node is displayed with the corresponding number of
individuals (n) who met that criterion. The 12-month observed rate of any
hypoglycemia-related ED or hospital use is displayed in each leaf node and
categorized as high (>5% risk), intermediate (1%-5% risk), or low risk (<1% risk).
CKD indicates chronic kidney disease.
of hypoglycemia-related utilization was lower in GH patients
(0.30%) compared with the KPNC derivation sample (0.49%)
and the VA (0.51%).
Model Validation
Internal validation of the classification tree model indicated
high discrimination (C statistic = 0.83) and good calibration (no
significant differences between predicted and observed risk:
Pearson 2 goodness-of-fit P = .31) (Table 2). The odds ratios
of hypoglycemia-related utilization among those categorized
as high relative to low risk were large in each sample: KPNC
internal validation sample (34.6; 95% CI, 24.2-49.3), VA (23.3;
95% CI, 21.9-24.7), and GH (20.7; 95% CI, 8.6-45.0; P < .001).
The tool also performed well in terms of discrimination in
the external validation samples (VA C statistic = 0.81; GH C sta-
tistic = 0.79). Visual inspection of the calibration plots showed
a reasonable match between the predicted and observed risk
of hypoglycemia-related utilization within the 10 leaf nodes
(Figure 3). However, the tool somewhat overestimated risk
among the leaf nodes in the intermediate- and higher-risk cat-
egories in the external validation samples.
Sensitivity Analyses
In the sensitivity analyses for length of available medical his-
tory, good discrimination was confirmed despite shorter medi-
cal history (C statistic = 0.82, 0.83, 0.84 for 2, 5, and 10
jamainternalmedicine.com
 Identifying Patients at Risk of Hypoglycemia-Related Use
Figure 2. Hypoglycemia Risk Stratification Tool
Tool Inputs
How many times has the patient ever had hypoglycemia-related utilization in an ED (primary diagnosis of hypoglycemiaa) or
hospital (principal diagnosis of hypoglycemiaa) (0, 1-2, 3 times)?
How many times has the patient gone to an ED for any reason in the prior 12 months (<2, 2 times)?
Does the patient use insulin (yes/no)?
Does the patient use sulfonylurea (yes/no)?
Does the patient have severe or end-stage kidney disease (CKD stage 4 or 5) (yes/no)?
Is the patient <77 years old (yes/no)?
Instructions: The 6 inputs above are used to identify one of the mutually exclusive exposure groups and the corresponding risk
category (high, low, or intermediate) for hypoglycemia-related ED or hospital utilizationb in the following 12 months. The first 5
options are defined by unique combinations of predictor variables, while the sixth option is indicated only after ruling out the first
5 options.
3 Prior hypoglycemia-related ED or hospital utilization
1-2 Prior hypoglycemia-related ED or hospital utilization AND
Insulin user
No prior hypoglycemia-related ED or hospital utilization AND
No insulin AND
No sulfonylurea use
No prior hypoglycemia-related ED or hospital utilization AND
No insulin AND
Uses sulfonylurea AND
Age <77 years AND
Does not have severe or end-stage kidney disease
No prior hypoglycemia-related ED or hospital utilization AND
Uses insulin AND
Age <77 years AND
<2 ED visits in prior year
All other risk factor combinations
years, respectively). We then evaluated temporal sensitivity
(historical transportability33). There was good discrimination
(C statistic = 0.83) even after KPNC experienced a significant
21% increase in the rate of hypoglycemia-related utilization
(0.48% vs 0.59% for 2014 and 2015, respectively). We also
evaluated sensitivity to misclassification of diabetes type by
including all diabetes patients in our sample and found good
discrimination (C statistic = 0.84).
In the assessment of ecological validity, there was a strong
association between predicted risk of hypoglycemia-related uti-
lization and self-reported severe hypoglycemia (ie, hypogly-
cemia necessitating assistance in the past 12 months). Pa-
tients categorized as high risk by the tool were 5 times more
likely (49.7% vs 9.2%; P < .001) to self-report a severe hypo-
glycemic episode relative to those categorized as low risk.
Discussion
Health care systems currently lack an evidence-based method
for efficiently and systematically identifying patients with T2D
at risk of hypoglycemia-related ED or hospital use. We devel-
oped and validated a pragmatic hypoglycemia risk stratifica-
tion tool that uses 6 factors to categorize the 12-month risk
of hypoglycemia-related utilization. This tool uses EMR
data only and requires no patient contact; it offers an effi-
cient, low-cost approach for identifying patients for targeted
interventions to reduce their risk of hypoglycemia.21,44-48 Be-
cause of the harms and costs associated with hypoglycemia,
jamainternalmedicine.com
2017 American Medical Association. All rights reserved.
Downloaded From: http://archinte.jamanetwork.com/ by a Columbia University User on 08/21/2017
Original Investigation Research
High risk (>5%)
a
Hypoglycemia cases were
ascertained with any of the
following International Classification
of Diseases, Ninth Revision, codes:
251.0, 251.1, 251.2, 962.3, or 250.8,
without concurrent 259.8, 272.7,
Low risk (<1%) 681.XX, 682.XX, 686.9X,
707.1-707.9, 709.3, 730.0-730.2, or
731.8 codes.27
b
Hypoglycemic-related utilization
was defined by having any
emergency department (ED) visit
with a primary diagnosis of
hypoglycemia or a hospitalization
Intermediate risk (1%-5%)
with a principal diagnosis of
hypoglycemia.
high-risk patients are candidates for an elevated level of scru-
tiny. Identifying medication overtreatment49-51 and includ-
ing hypoglycemia rates as a health planlevel quality mea-
sure have been recommended to drive accountability and
quality improvement.52-54
This tool is intended to offer a practical method to risk
stratify patients for population management. For example, in-
tensive interventions aimed at reducing hypoglycemia risk
could be targeted at the minority of patients with T2D in the
high-risk category (2% of patients with diabetes at KPNC).
These interventions could include deintensifying or simpli-
fying medication regimens, addressing impaired hypoglyce-
mic awareness, prescribing glucagon kits or continuous
glucose monitors, making referrals to clinical pharmacists or
nurse care managers, providing additional diabetes educa-
tion, and regularly asking about hypoglycemia events occur-
ring outside the medical setting. Clinician discussions could
address potential contributors to hypoglycemia, including
behavioral (eg, meal skipping 55 ), psychosocial (eg, food
insufficiency56,57), or socioeconomic (eg, deprivation58) fac-
tors. Similarly, a lower-cost, less intensive intervention could
be designed for patients in the intermediate-risk category (11%
at KPNC). The intervention could include system-level struc-
tural modifications such as risk-based glycemic targets, auto-
mated clinical alert flags in the EMR, and automated messag-
ing to patients with elevated risk. Moreover, the tool could be
modified to identify specific subsets of risk groups such those
with 3 or more hypoglycemia-related ED or hospital encoun-
ters whom our model identifies as having the highest risk.
(Reprinted) JAMA Internal Medicine Published online August 21, 2017 E5
 Research Original Investigation Identifying Patients at Risk of Hypoglycemia-Related Use

Table 1. Demographic Characteristics, Baseline Predictor Variables, and Outcome Rates in the Internal Derivation and Validation Samples (Kaiser
Permanente Northern California [KPNC]) and the 2 External Validation Samples (Group Health Cooperative [GH] and Veterans Administration [VA])

Derivation Sample Validation Samples Contrasts, P Value

KPNC
Derivation vs
KPNC KPNC vs GH KPNC vs VA
KPNC KPNC GH VA Validation Validation Validation
Parameter (n = 165 148) (n = 41 287) (n = 14 972) (n = 1 335 966) Sample Samples Samples
Demographic Characteristics
Sex, No. (%)
Male 86 572 (52.4) 21 705 (52.6) 7671 (51.2) 1 288 576 (96.5)
.59 .005 <.001
Female 78 576 (47.6) 19 582 (47.4) 7301 (48.8) 47 390 (3.5)
Age, mean (SD), y 63.9 (13.0) 64.0 (13.0) 63.4 (12.2) 68.8 (11.1) .38 <.001 <.001
Race/ethnicity, No. (%)
White 73 796 (44.7) 18 531 (44.9) 10 276 (68.6) 927 577 (69.4)
Black 17 039 (10.3) 4253 (10.3) 986 (6.6) 231 210 (17.3)
Hispanic 26 924 (16.3) 6676 (16.2) 818 (5.5) 70 429 (5.3) .94 <.001 <.001
Other 45 066 (27.3) 11 238 (27.2) 2344 (15.7) 42 170 (3.2)
Unknown 2323 (1.4) 589 (1.4) 548 (3.7) 64 580 (4.8)
Model Input Variables
Prior
hypoglycemic-related
utilization,a No. (%)
None 159 704 (96.7) 39 897 (96.6) 14 672 (98.0) 1 282 378 (96.0)
1-2 times 4948 (3.0) 1268 (3.1) 284 (1.9) 48 194 (3.6) .72 <.001 <.001
3 times 496 (0.3) 122 (0.3) 16 (0.1) 5394 (0.4)
Current diabetes
medications, No. (%)
Sulfonylurea 57 701 (34.9) 14 405 (34.9) 3362 (22.5) 333 756 (25.0) .85 <.001 <.001
Insulin 33 578 (20.3) 8391 (20.3) 4629 (30.9) 403 538 (30.2) .97 <.001 <.001
Emergency
department visits in
prior year, No. (%)
0-1 148 662 (90.0) 37 104 (89.9) 13 532 (90.4) 1 165 838 (87.3)
.37 .07 <.001
2 16 486 (10.0) 4183 (10.1) 1440 (9.6) 170 128 (12.7)
Chronic kidney disease
stage,b No. (%)
1-3 160 122 (97.0) 40 021 (96.9) 14 670 (98.0) 1 286 719 (96.3)
.81 <.001 <.001
4-5 5026 (3.0) 1266 (3.1) 302 (2.0) 49 247 (3.7)
Age 77 y, No. (%) 29 410 (17.8) 7344 (17.8) 1980 (13.2) 329 055 (24.6) .92 <.001 <.001
Outcome Rates
Hypoglycemia-related 808 (0.49) 192 (0.47) 45 (0.3) 6731 (0.51) .53 .008 .27
utilization during
follow-up (2014),a
No. (%)
a b
Hypoglycemic-related utilization was defined by having any emergency Chronic kidney disease stage 1 to 3 determined by estimated glomerular
department visit with a primary diagnosis of hypoglycemia or a hospitalization filtration rate43 greater than 29 mL/min/1.73 m2 calculated by the Chronic
with a principal diagnosis of hypoglycemia. Hypoglycemia cases were Kidney Disease Epidemiology Collaboration creatinine equation43; stage 4 or 5
ascertained with any of the following International Classification of Diseases, determined by estimated glomerular filtration rate less than or equal to 29
Ninth Revision, codes: 251.0, 251.1, 251.2, 962.3, or 250.8, without concurrent mL/min/1.73 m2 or requirement of dialysis.
259.8, 272.7, 681.XX, 682.XX, 686.9X, 707.1-707.9, 709.3, 730.0-730.2, or
731.8 codes.27

While it is unknown to what extent clinicians are aware
of a patients hypoglycemia risk, there is evidence that clini-
cians and patients with diabetes do not communicate about
hypoglycemia events that occur outside clinical settings.59,60
Almost all (roughly 95%) severe hypoglycemia events may go
clinically unrecognized because they did not result in ED or
hospital use.61 In an internal review of EMRs of KPNC pa-
tients with T2D, hypoglycemia was absent from the problem
lists in 85% of patients categorized by our tool as being at high
risk, underscoring the potential for this tool to increase clini-
cians awareness of the risk of hypoglycemia in their patients
with T2D.
Limitations
Some limitations should be noted. The final classification
tree was one of many possible options, chosen on the basis
of performance (eg, C statistic), parsimony, and pragmatism
(eg, we excluded predictors that are typically unavailable,

E6 JAMA Internal Medicine Published online August 21, 2017 (Reprinted) jamainternalmedicine.com

2017 American Medical Association. All rights reserved.

Downloaded From: http://archinte.jamanetwork.com/ by a Columbia University User on 08/21/2017

 Identifying Patients at Risk of Hypoglycemia-Related Use Original Investigation Research

Table 2. Calibration of Expected and Observed Rates of Hypoglycemia-Related Utilizationa Across the 3 Risk Strata in the Validation Samples

No. (% of Strata)
Risk Strata No. (% of Sample) Observed Expected P Valueb
Kaiser Permanente (Internal) Validation Sample
Low 36 041 (87.3) 75 (0.21) 76 (0.21) .75
Intermediate 4429 (10.7) 62 (1.40) 71 (1.60)
High 817 (2.0) 55 (6.73) 53 (6.49)
Total 41 287 192 (0.47) 201 (0.48)
Group Health (External) Validation Sample
Low 13 261 (88.6) 23 (0.17) 28 (0.25) .02
Intermediate 1481 (9.9) 14 (0.95) 24 (1.69)
High 230 (1.5) 8 (3.48) 15 (5.65)
Total 14 972 45 (0.30) 67 (0.47)
Veterans Administration (External) Validation Sample
Low 1 096 945 (82.1) 2691 (0.25) 2558 (0.23) <.001
Intermediate 204 913 (15.3) 2192 (1.07) 3527 (1.72)
High 34 108 (2.6) 1848 (5.42) 2255 (6.61)
Total 1 335 966 6731 (0.50) 7857 (0.62)
a b
Hypoglycemic-related utilization was defined by having any emergency P value from Pearson 2 goodness-of-fit test. For each leaf node in the
department visit with a primary diagnosis of hypoglycemia or a hospitalization classification tree, we multiplied the number of patients by the predicted rate
with a principal diagnosis of hypoglycemia. Hypoglycemia cases were of hypoglycemia events to generate the number of expected events in that
ascertained with any of the following International Classification of Diseases, node. We then summed the expected events from all nodes within each of the
Ninth Revision, codes: 251.0, 251.1, 251.2, 962.3, or 250.8, without concurrent 3 risk levels to obtain expected event counts by risk level. The expected rates
259.8, 272.7, 681.XX, 682.XX, 686.9X, 707.1-707.9, 709.3, 730.0-730.2, or for each 3-level stratum will therefore vary between samples due to case-mix
731.8 codes27; rates were calculated as the number of patients with any differences.
hypoglycemia-related utilization during the year per 1000 patients.

impractical, or costly to assess in usual care settings). In de-
veloping the model, we excluded secondary discharge diag-
noses for hypoglycemia because these events may occur
during the ED or hospital encounter (eg, inpatient insulin man-
agement, sepsis, acute renal failure28), rather than being a cause
of the encounter. On the other hand, hypoglycemia could be
a secondary diagnosis if the primary or principal diagnosis is
trauma due to an automobile accident or a serious fall caused
by hypoglycemia. Although we did not include those events
in our model development, we estimate that this would in-
clude less than 2% of ED encounters (data not shown). When
validating this model, we emphasized discrimination rather
than calibration given that our goal was to risk stratify pa-
tients into broad categories rather than predicting the con-
tinuous level of risk.40 Thus, while the tool successfully strati-
fies the population into 3 levels of risk, it should not be used
to estimate the probability of hypoglycemic-related utiliza-
tion for an individual patient. Discrimination performance of
the tool was validated internally and externally in 2 large, in-
dependent populations, suggesting generalizability. Geo-
graphic, methodologic, and spectrum transportability were also
demonstrated because these external validation populations
were from different geographical locations, used differing
methods of caring for and identifying patients with T2D, and
had different distributions of disease severity and case mix.
While the observed rates of hypoglycemia-related utilization
were lower in GH (0.3%) and somewhat higher in VA (0.51%)
compared with KPNC sample (0.49%), the tool had good dis-
crimination in all 3 samples. Some of the inconsistent find-
ings are attributable to sparse data in the GH validation sample;
only 45 events were observed during follow-up. It is also pos-
sible that the 2 external sample populations may experience
more prevention efforts than at KPNC, explaining the some-
what lower than expected risk in higher-risk patients in the
validation samples.
The tool was designed to predict hypoglycemia-related uti-
lization and thus did not take into account severe hypoglyce-
mia occurring outside the health care system (roughly 95% of
hypoglycemic events necessitating third-party assistance are
treated by persons other than medical professionals, eg, friends
or family61,62). However, we demonstrated a strong and sig-
nificant association between the tools stratification of the
risk of hypoglycemia-related utilization and a patients self-
reported severe hypoglycemia events. Therefore, we believe
that preventive interventions targeting patients identified as
high risk by this tool may reduce the rate of severe hypogly-
cemic events that do not result in utilization or clinical recog-
nition. Finally, the tool also proved robust in internal valida-
tions when patients with type 1 diabetes were included
and when the availability of longitudinal EMR data was
restricted.
This risk stratification tool was developed and validated
in 3 vertically integrated health care delivery systems (KPNC,
GH, and the VA). The tool logic may be programmed into a
systems EMR to allow for automated risk stratification based
on the EMR data. Practical barriers may complicate im-
plementing this tool for population management in horizon-
tally integrated or nonintegrated health care delivery sys-
tems, for example, where pharmacy claims may not be
readily available.

jamainternalmedicine.com (Reprinted) JAMA Internal Medicine Published online August 21, 2017 E7

2017 American Medical Association. All rights reserved.

Downloaded From: http://archinte.jamanetwork.com/ by a Columbia University User on 08/21/2017

 Research Original Investigation Identifying Patients at Risk of Hypoglycemia-Related Use

Figure 3. Calibration Plots Comparing the Expected vs Observed 12-Month Rate of Having Any Hypoglycemia-Related Utilizationa for the Interval
Derivation Sample From Kaiser Permanente Northern California (KPNC) (n = 165 148), the KPNC Internal Validation Sample (n = 41 287), the External
Validation Sample From Group Health (GH) (n = 14 972), and the External Validation Sample From the Veterans Administration (VA) (n = 1 335 966)

A KPNC derivation B KPNC internal validation

16 16

14 14

12 12
Observed Rate, %

Observed Rate, %
10 10

8 8

6 6

4 4

2 2

0 0

0 2 4 6 8 10 12 14 16 0 2 4 6 8 10 12 14 16
Predicted Rate, % Predicted Rate, %

C GH external validation D VA external validation

16 16

14 14

12 12
Observed Rate, %

Observed Rate, %

10 10

8 8

6 6

4 4

2 2

0 0

0 2 4 6 8 10 12 14 16 0 2 4 6 8 10 12 14 16
Predicted Rate, % Predicted Rate, %

a
Hypoglycemic-related utilization was defined by having any emergency
department visit with a primary diagnosis of hypoglycemia or a hospitalization
with a principal diagnosis of hypoglycemia. Hypoglycemia cases were
ascertained with any of the following International Classification of Diseases,
Ninth Revision, codes: 251.0, 251.1, 251.2, 962.3, or 250.8, without concurrent
259.8, 272.7, 681.XX, 682.XX, 686.9X, 707.1-707.9, 709.3, 730.0-730.2, or
731.8 codes.27

sive (more expensive) interventions reserved for high-risk pa-

Conclusions
This tool offers a practical, EMR-based method to stratify pa-
tients with T2D by their 12-month risk of hypoglycemia-
related ED or hospital utilization. This tool could be inte-
grated with targeted preventive interventions to facilitate
population management, which ultimately could reduce fu-
ture hypoglycemia risk and improve patient safety. The 2 cri-
teria indicating high risk are easily memorized (ie, 3 previ-
ous episodes of hypoglycemia-related utilization, or 1 or 2
episodes if treated with insulin). The criteria for intermediate
risk are more nuanced and therefore may be less likely to pro-
voke clinical action in primary care without prompting. Health
care systems could adopt a 2-level intervention, with inten-
tients and less intensive (lower cost) interventions for the in-
termediate-risk patients. Implementation of this tool could
conceivably increase clinician awareness of patients hypo-
glycemia risk. Clinical researchers may also find this tool help-
ful in identifying patients at high risk for hypoglycemic epi-
sodes for either purposeful inclusion or exclusion in clinical
trials of novel therapies and diagnostic tests. Quality improve-
ment and impact studies29 are needed to evaluate whether and
how implementation of this hypoglycemia risk stratification
tool may influence clinician behavior, patient decision mak-
ing, drug safety, and hypoglycemia incidence. Future re-
search is needed to develop patient-centered and cost-
effective interventions to reduce hypoglycemia risk in those
identified as being at high risk for hypoglycemia.

ARTICLE INFORMATION Published Online: August 21, 2017. Author Affiliations: Division of Research, Kaiser
Accepted for Publication: June 20, 2017. doi:10.1001/jamainternmed.2017.3844 Permanente Northern California, Oakland (Karter,
Warton, Moffet); Department of General Internal

E8 JAMA Internal Medicine Published online August 21, 2017 (Reprinted) jamainternalmedicine.com

2017 American Medical Association. All rights reserved.

Downloaded From: http://archinte.jamanetwork.com/ by a Columbia University User on 08/21/2017

 Identifying Patients at Risk of Hypoglycemia-Related Use
Medicine, University of California, San Francisco
(Karter); Department of Epidemiology, University
of Washington, Seattle (Karter); Department of
Health Services, University of Washington, Seattle
(Karter); Section of Endocrinology, Department of
Internal Medicine, Yale School of Medicine, New
Haven, Connecticut (Lipska); Kaiser Permanente
Washington Health Research Institute, Seattle
(Ralston, Jackson); Section of General Internal
Medicine, Department of Medicine, University of
Chicago, Chicago, Illinois (Huang); Center for
Healthcare Organization and Implementation
Research, Edith Nourse Rogers Memorial Veterans
Hospital, Bedford, Massachusetts (Miller).
Author Contributions: Dr Karter had full access to
all the data in the study and takes responsibility for
the integrity of the data and the accuracy of the
data analysis.
Concept and design: Karter, Warton, Lipska, Moffet,
Huang, Miller.
Acquisition, analysis, or interpretation of data:
Karter, Warton, Lipska, Ralston, Moffet, Jackson,
Miller.
Drafting of the manuscript: Karter, Warton, Jackson.
Critical revision of the manuscript for important
intellectual content: Karter, Warton, Lipska, Ralston,
Moffet, Huang, Miller.
Statistical analysis: Karter, Warton, Miller.
Obtained funding: Karter, Moffet.
Administrative, technical, or material support:
Karter, Ralston, Moffet, Jackson, Miller.
Supervision: Karter, Ralston, Huang.
Conflict of Interest Disclosures: The National
Institutes of Health supplied additional funding for
our hypoglycemia-related research (NIDDK
R01DK103721, R01DK081796). Drs Karter and
Huang are also supported by the NIDDK Centers for
Diabetes Translational Research (P30 DK092924
and P30 DK092949, respectively). Dr Huang was
supported by K24 DK105340. Dr Lipska receives
support from the Centers for Medicare & Medicaid
Services to develop and maintain publicly reported
quality measures, and from the Yale Claude D.
Pepper Older Americans Independence Center
(P30AG021342) and the National Institute on Aging
through the Paul Beeson Career Development
Award (K23AG048359). No other disclosures are
reported.
Funding/Support: This project was funded by the
US Food and Drug Administration (FDA
BAA-13 00119).
Role of the Funder/Sponsor: The funders had no
role in the design and conduct of the study;
collection, management, analysis, and
interpretation of the data; preparation, review, or
approval of the manuscript; and decision to submit
the manuscript for publication.
REFERENCES
1. Gregg EW, Li Y, Wang J, et al. Changes in
diabetes-related complications in the United States,
1990-2010. N Engl J Med. 2014;370(16):1514-1523.
2. Pogach L, Aron D. Balancing hypoglycemia and
glycemic control: a public health approach for
insulin safety. JAMA. 2010;303(20):2076-2077.
3. Lee SJ. So much insulin, so much hypoglycemia.
JAMA Intern Med. 2014;174(5):686-688.
4. Lipska KJ. Improving safety of diabetes mellitus
management. JAMA Intern Med. 2014;174(10):1612-
1613.
jamainternalmedicine.com
2017 American Medical Association. All rights reserved.
Downloaded From: http://archinte.jamanetwork.com/ by a Columbia University User on 08/21/2017
5. Workgroup on Hypoglycemia, American
Diabetes Association. Defining and reporting
hypoglycemia in diabetes: a report from the
American Diabetes Association Workgroup on
Hypoglycemia. Diabetes Care. 2005;28(5):1245-1249.
6. Majumdar SR, Hemmelgarn BR, Lin M, McBrien
K, Manns BJ, Tonelli M. Hypoglycemia associated
with hospitalization and adverse events in older
people: population-based cohort study. Diabetes
Care. 2013;36(11):3585-3590.
7. Lipska KJ, Ross JS, Wang Y, et al. National trends
in US hospital admissions for hyperglycemia and
hypoglycemia among Medicare beneficiaries, 1999
to 2011. JAMA Intern Med. 2014;174(7):1116-1124.
8. Huang ES, Laiteerapong N, Liu JY, John PM,
Moffet HH, Karter AJ. Rates of complications and
mortality in older patients with diabetes mellitus:
the diabetes and aging study. JAMA Intern Med.
2014;174(2):251-258.
9. Budnitz DS, Lovegrove MC, Shehab N, Richards
CL. Emergency hospitalizations for adverse drug
events in older Americans. N Engl J Med. 2011;365
(21):2002-2012.
10. Signorovitch JE, Macaulay D, Diener M, et al.
Hypoglycaemia and accident risk in people with
type 2 diabetes mellitus treated with non-insulin
antidiabetes drugs. Diabetes Obes Metab. 2013;15
(4):335-341.
11. Stahn A, Pistrosch F, Ganz X, et al. Relationship
between hypoglycemic episodes and ventricular
arrhythmias in patients with type 2 diabetes and
cardiovascular diseases: silent hypoglycemias and
silent arrhythmias. Diabetes Care. 2014;37(2):516-
520.
12. Whitmer RA, Karter AJ, Yaffe K, Quesenberry
CP Jr, Selby JV. Hypoglycemic episodes and risk of
dementia in older patients with type 2 diabetes
mellitus. JAMA. 2009;301(15):1565-1572.
13. McCoy RG, Van Houten HK, Ziegenfuss JY, Shah
ND, Wermers RA, Smith SA. Increased mortality of
patients with diabetes reporting severe
hypoglycemia. Diabetes Care. 2012;35(9):1897-1901.
14. Zoungas S, Patel A, Chalmers J, et al; ADVANCE
Collaborative Group. Severe hypoglycemia and risks
of vascular events and death. N Engl J Med. 2010;
363(15):1410-1418.
15. Karter AJ, Subramanian U, Saha C, et al. Barriers
to insulin initiation: the translating research into
action for diabetes insulin starts project. Diabetes
Care. 2010;33(4):733-735.
16. Laiteerapong N, Karter AJ, Liu JY, et al.
Correlates of quality of life in older adults with
diabetes: the diabetes & aging study. Diabetes Care.
2011;34(8):1749-1753.
17. Nicolucci A, Pintaudi B, Rossi MC, et al. The
social burden of hypoglycemia in the elderly. Acta
Diabetol. 2015;52(4):677-685.
18. Peusens G, De Jonghe K, De Rood, I, et al.
Phytoplasmas in pome fruit trees: update of their
presence and their vectors in Belgium. Commun
Agric Appl Biol Sci. 2015;80(2):143-148.
19. Amiel SA, Dixon T, Mann R, Jameson K.
Hypoglycaemia in type 2 diabetes. Diabet Med.
2008;25(3):245-254.
20. Meulstee M, Whittemore R, Watts SA.
Development of an educational program on
prevention of hypoglycemic events among elderly
(Reprinted) JAMA Internal Medicine Published online August 21, 2017
Original Investigation Research
veterans with type 2 diabetes. Diabetes Educ. 2015;
41(6):690-697.
21. Rondags SM, de Wit M, Snoek FJ. HypoAware:
development and pilot study of a brief and partly
web-based psychoeducational group intervention
for adults with type 1 and insulin-treated type 2
diabetes and problematic hypoglycaemia. Diabet
Med. 2016;33(2):184-191.
22. Elliott J, Rankin D, Jacques RM, et al; NIHR
DAFNE Research Study Group. A cluster
randomized controlled non-inferiority trial of 5-day
dose adjustment for normal eating (DAFNE)
training delivered over 1 week versus 5-day DAFNE
training delivered over 5 weeks: the DAFNE
5 1-day trial. Diabet Med. 2015;32(3):391-398.
23. Seaquist ER, Anderson J, Childs B, et al;
American Diabetes Association; Endocrine Society.
Hypoglycemia and diabetes: a report of a
workgroup of the American Diabetes Association
and the Endocrine Society. J Clin Endocrinol Metab.
2013;98(5):1845-1859.
24. Heinemann L, Devries JH. Evidence for
continuous glucose monitoring: sufficient for
reimbursement? Diabet Med. 2014;31(2):122-125.
25. Miller DR, Pogach L. Longitudinal approaches
to evaluate health care quality and outcomes: the
Veterans Health Administration diabetes
epidemiology cohorts. J Diabetes Sci Technol.
2008;2(1):24-32.
26. Huang ES, Liu JY, Moffet HH, John PM, Karter
AJ. Glycemic control, complications, and death in
older diabetic patients: the diabetes and aging
study. Diabetes Care. 2011;34(6):1329-1336.
27. Ginde AA, Blanc PG, Lieberman RM, Camargo
CA Jr. Validation of ICD-9-CM coding algorithm for
improved identification of hypoglycemia visits.
BMC Endocr Disord. 2008;8:4.
28. Miller SI, Wallace RJ Jr, Musher DM, Septimus
EJ, Kohl S, Baughn RE. Hypoglycemia as a
manifestation of sepsis. Am J Med. 1980;68(5):
649-654.
29. Moons KG, Altman DG, Vergouwe Y, Royston P.
Prognosis and prognostic research: application and
impact of prognostic models in clinical practice. BMJ.
2009;338:b606.
30. Royston P, Moons KG, Altman DG, Vergouwe Y.
Prognosis and prognostic research: developing a
prognostic model. BMJ. 2009;338:b604.
31. Altman DG, Vergouwe Y, Royston P, Moons KG.
Prognosis and prognostic research: validating a
prognostic model. BMJ. 2009;338:b605.
32. Steyerberg EW. Clinical Prediction Models:
A Practical Approach to Development, Validation,
and Updating. New York, NY: Springer; 2009.
33. Justice AC, Covinsky KE, Berlin JA. Assessing
the generalizability of prognostic information. Ann
Intern Med. 1999;130(6):515-524.
34. Breiman L, Friedman JH, Olshen RA, Stone CJ.
Classification and Regression Trees. Monterey, CA:
Wadsworth & Brooks/Cole Advanced Books &
Software; 1984.
35. Fonarow GC, Adams KF Jr, Abraham WT, Yancy
CW, Boscardin WJ; ADHERE Scientific Advisory
Committee, Study Group, and Investigators. Risk
stratification for in-hospital mortality in acutely
decompensated heart failure: classification and
regression tree analysis. JAMA. 2005;293(5):572-
580.
E9
 Research Original Investigation
36. Lewis RJ. An Introduction to Classification and
Regression Tree (CART) Analysis. 2000.
http://citeseerx.ist.psu.edu/viewdoc/download
?doi=10.1.1.95.4103&rep=rep1&type=pdf. Accessed
July 6, 2017.
37. Hosmer DW, Lemeshow S. Applied Logistic
Regression. New York, NY: John Wiley & Sons; 1989.
38. Chambless LE, Cummiskey CP, Cui G. Several
methods to assess improvement in risk prediction
models: extension to survival analysis. Stat Med.
2011;30(1):22-38.
39. Moons KG, de Groot JA, Bouwmeester W, et al.
Critical appraisal and data extraction for systematic
reviews of prediction modelling studies: the
CHARMS checklist. PLoS Med. 2014;11(10):e1001744.
40. Meurer WJ, Tolles J. Logistic regression
diagnostics: understanding how well a model
predicts outcomes. JAMA. 2017;317(10):1068-1069.
41. Walter LC, Brand RJ, Counsell SR, et al.
Development and validation of a prognostic index
for 1-year mortality in older adults after
hospitalization. JAMA. 2001;285(23):2987-2994.
42. Moffet HH, Adler N, Schillinger D, et al. Cohort
profile: the Diabetes Study of Northern California
(DISTANCE)objectives and design of a survey
follow-up study of social health disparities in a
managed care population. Int J Epidemiol. 2009;38
(1):38-47.
43. Levey AS, Bosch JP, Lewis JB, Greene T, Rogers
N, Roth D; Modification of Diet in Renal Disease
Study Group. A more accurate method to estimate
glomerular filtration rate from serum creatinine:
a new prediction equation. Ann Intern Med. 1999;
130(6):461-470.
44. Senior PA, Bellin MD, Alejandro R, et al; Clinical
Islet Transplantation Consortium. Consistency of
quantitative scores of hypoglycemia severity and
glycemic lability and comparison with continuous
glucose monitoring system measures in
long-standing type 1 diabetes. Diabetes Technol Ther.
2015;17(4):235-242.
45. Olsen SE, Bjrgaas MR, svold BO, et al.
Impaired awareness of hypoglycemia in adults with
type 1 diabetes is not associated with autonomic
E10 JAMA Internal Medicine Published online August 21, 2017 (Reprinted)
2017 American Medical Association. All rights reserved.
Downloaded From: http://archinte.jamanetwork.com/ by a Columbia University User on 08/21/2017
Identifying Patients at Risk of Hypoglycemia-Related Use
dysfunction or peripheral neuropathy. Diabetes Care.
2016;39(3):426-433.
46. Rondags SM, de Wit M, van Tulder MW,
Diamant M, Snoek FJ. HypoAwarea brief and
partly web-based psycho-educational group
intervention for adults with type 1 and
insulin-treated type 2 diabetes and problematic
hypoglycaemia: design of a cost-effectiveness
randomised controlled trial. BMC Endocr Disord.
2015;15:43.
47. Shepard JA, Vajda K, Nyer M, Clarke W,
Gonder-Frederick L. Understanding the construct
of fear of hypoglycemia in pediatric type 1 diabetes.
J Pediatr Psychol. 2014;39(10):1115-1125.
48. Rondags SM, de Wit M, Twisk JW, Snoek FJ.
Effectiveness of HypoAware, a brief partly
web-based psychoeducational intervention for
adults with type 1 and insulin-treated type 2
diabetes and problematic hypoglycemia: a cluster
randomized controlled trial. Diabetes Care. 2016;39
(12):2190-2196.
49. Pogach L, Aron D. The other side of quality
improvement in diabetes for seniors: a proposal for
an overtreatment glycemic measure. Arch Intern Med.
2012;172(19):1510-1512.
50. Tseng CL, Soroka O, Maney M, Aron DC,
Pogach LM. Assessing potential glycemic
overtreatment in persons at hypoglycemic risk.
JAMA Intern Med. 2014;174(2):259-268.
51. McCoy RG, Lipska KJ, Yao X, Ross JS, Montori
VM, Shah ND. Intensive treatment and severe
hypoglycemia among adults with type 2 diabetes.
JAMA Intern Med. 2016;176(7):969-978.
52. Lipska KJ, Kosiborod M. Hypoglycemia and
adverse outcomes: marker or mediator? Rev
Cardiovasc Med. 2011;12(3):132-135.
53. Rodriguez-Gutierrez R, Lipska KJ, McCoy RG,
Ospina NS, Ting HH, Montori VM; Hypoglycemia as
a Quality Measure in Diabetes Study Group.
Hypoglycemia as an indicator of good diabetes
care. BMJ. 2016;352:i1084.
54. Rodriguez-Gutierrez R, Ospina NS, McCoy RG,
Lipska KJ, Shah ND, Montori VM; Hypoglycemia as a
Quality Measure in Diabetes Study Group. Inclusion
of hypoglycemia in clinical practice guidelines and
performance measures in the care of patients with
diabetes. JAMA Intern Med. 2016;176(11):1714-1716.
55. Bonds DE, Miller ME, Dudl J, et al. Severe
hypoglycemia symptoms, antecedent behaviors,
immediate consequences and association with
glycemia medication usage: secondary analysis of
the ACCORD clinical trial data. BMC Endocr Disord.
2012;12:5.
56. Seligman HK, Bolger AF, Guzman D, Lpez A,
Bibbins-Domingo K. Exhaustion of food budgets at
months end and hospital admissions for
hypoglycemia. Health Aff (Millwood). 2014;33(1):
116-123.
57. Basu S, Berkowitz SA, Seligman H. The monthly
cycle of hypoglycemia: an observational
claims-based study of emergency room visits,
hospital admissions, and costs in a commercially
insured population. Med Care. 2017;55(7):639-645.
58. Berkowitz SA, Karter AJ, Lyles CR, et al. Low
socioeconomic status is associated with increased
risk for hypoglycemia in diabetes patients: the
Diabetes Study of Northern California (DISTANCE).
J Health Care Poor Underserved. 2014;25(2):478-490.
59. Frier BM, Jensen MM, Chubb BD.
Hypoglycaemia in adults with insulin-treated
diabetes in the UK: self-reported frequency and
effects. Diabet Med. 2016;33(8):1125-1132.
60. stenson CG, Geelhoed-Duijvestijn P, Lahtela
J, Weitgasser R, Markert Jensen M,
Pedersen-Bjergaard U. Self-reported non-severe
hypoglycaemic events in Europe. Diabet Med. 2014;
31(1):92-101.
61. Sarkar U, Karter AJ, Liu JY, Moffet HH, Adler NE,
Schillinger D. Hypoglycemia is more common
among type 2 diabetes patients with limited health
literacy: the Diabetes Study of Northern California
(DISTANCE). J Gen Intern Med. 2010;25(9):962-968.
62. Lipska KJ, Warton EM, Huang ES, et al. HbA1c
and risk of severe hypoglycemia in type 2 diabetes:
the Diabetes and Aging Study. Diabetes Care.
2013;36(11):3535-3542.
jamainternalmedicine.com