Pilot Testing of the Pediatric Screening Inventory in the Primary Care Setting

Objective: The present study was a preliminary evaluation of a parental self-report questionnaire appropriate for the pediatric primary care setting (The Pediatric Screening Inventory; PSI). The PSI assesses three domains: parenting skills, child behavior problems, and behaviors related to medical compliance. Methods: Piloting of the PSI involved a randomized block design that assigned pediatric primary care providers (n=214 visits) to either utilize the PSI (experimental condition) or conduct assessment as usual (control condition). Three hypotheses comparing the control and experimental conditions were tested. Results: The PSI was found to statistically significantly improve rates of detection and intervention of targeted problem domains in the pediatric primary care setting, while not having any observed adverse effects on either parent or provider satisfaction. Conclusions: These findings are consistent with the hypothesis that screening and providing effective and efficient behavioral interventions in the pediatric primary care setting can lead to positive outcomes.


Acta Psychopathologica
ISSN 2469-6676 children and adolescents for psychological concerns in pediatric primary care settings (AAP, 2009). However, there are neither widely-established physician training programs nor visit-specific protocols to guide the process of screening for and appropriately addressing behavioral concerns in pediatrics (Drotar, 2002) [8].
Moreover, significant discrepancies have been reported between parental concerns about child psychological well-being and/or family functioning and what parents actually address with their physician (Briggs-Gowan, Horwitz, Schwab-Stone, Leventhal, Leaf, 2000; Burklow, Vaughn, Valerius, Shultz, 2001) [9,10]. Ninetyone percent of parents report that it would be appropriate to discuss family problems with their pediatrician and 78.8% report that it would be appropriate to discuss disciplinary concerns (Briggs-Gowan, et al., 2000) [9]. However, only about 8% of parents actually spontaneously address their concerns with their pediatrician and only 27-40% discusses their concerns with the pediatrician when asked specifically about behavior problems during their office visit (Glascoe, MacLean, Stone, 1991;; Taaffe Young et al., 1998) [6,11,12]. Fifty-five percent of parents of children meeting criteria for a psychological disorder do not address their behavioral/emotional concerns about their child with their child's physician (Briggs-Gowan et al., 2000) [9]. Therefore, parental underreporting is a significant impediment to the detection and treatment of behavioral problems in the primary care setting which may logically be remediated through the use of screening.
Despite the potential benefit of screening, there appear to be significant barriers to widespread implementation. For example, many questionnaires have been developed and tested for the assessment of child behavior problems. However, most of these instruments were developed for research purposes and are cumbersome and impractical in the reality of the primary care setting (Stancin & Palermo, 1997) [13]. As a result, most primary care practices do not use screening devices routinely (Jensen et al., 2011) [14]. In a survey of pediatricians, only one-third of respondents reported ever having used a screening instrument in their practice and 77% believed that such instruments "are rarely useful because (they) don't feel competent to deal with these issues," (Cheng, Savageau, Bigelow, Charney, 1996) [15].
As a result of failing to screen appropriately, there is considerable evidence suggesting that 46%-80% of a patient's behavioral concerns are not known to their pediatrician (e.g., Costello et al., 1988; Sheldrink, Merchant, Perrin, 2011) [16,17] and when detected, these concerns will not be adequately addressed (e.g. Steele, Lochrie, Roberts, 2010; Wildman & Stancin, 2006) [18,19]. It has been estimated that less than half of children identified by primary care physicians as having "mental health problems" received referrals to a tertiary care facility (Bernal, 2003) [20]. This statistic is particularly disheartening given the finding that one of the strongest predictors of a family's decision to seek psychological services for their child is the recommendation of their primary care physician (Lavigne et al., 1998) [21]. The estimated costs associated with the known chasm between prevalence and detection rates are high for the individual child, the family, and society, given that the most effective and efficient method of service delivery is early detection and intervention (NIHCM, 2009).

Purpose
The purpose of the current research was to create and conduct preliminary evaluation of a parental self-report assessment questionnaire called the Pediatric Screening Inventory that is designed to improve on assessment as usual by: (1) improving the comprehensiveness of primary care screening; (2) being practical for use in the primary care setting (i.e. little administration time; ease of scoring); (3) assessing domains of functioning on continua, rather than being bifurcated artificially as either meeting or not meeting diagnostic criterion, and (4) subsequently increasing prevention and intervention efforts within primary care visits and referrals to other psychological or physical health specialists.

Hypotheses
It was hypothesized that: 1. The Pediatric Screening Inventory would result in increased detection of problems in child and/or family functioning in comparison to assessment as usual.
2. The Pediatric Screening Inventory would be associated with an increased rate of behavioral or parenting intervention during the primary care visit as well as an increased rate of referral.
3. The Pediatric Screening Inventory would have both greater physician and parent satisfaction than assessment as usual.

Development of the Pediatric Screening Inventory
The Pediatric Screening Inventory was developed using a rational method of test construction to develop items relevant to the three domains of interest (child behavior problems, parenting practices/family environment, and physical health factors/medical treatment adherence). For child behavior problems, items were developed to assess both internalizing and externalizing types of behavioral problems (e.g., Achenbach, 2012  [23]. Three parenting domains were determined to be of utmost importance to assess on the PSI: self and family management (monitoring/supervision, home environment, including access to resources and safety, and parental emotional well-being), setting limits and using discipline (discipline, consistency, and behavior control), and parent-child relationship (parental involvement and use of positive parenting techniques). Finally, to increase its relevance to a medical setting, questions related to adherence to previously prescribed medical treatments and general indices of physical health were developed.
An initial item pool of 53 items encompassing all three domains Acta Psychopathologica ISSN 2469-6676 of the PSI was developed. Two types of focus groups were held to gain feedback on the initial item pool. First, five focus groups with pediatric medical professionals were held (n=41). Secondly, two parent focus groups were conducted (n=26; 25 mothers and 1 father). Finally, expert reviewers were also asked to provide feedback on early drafts of the PSI (n=4). Three of the expert reviews are nationally recognized experts in behavioral pediatrics with extensive training and experience providing psychological services in pediatric primary care environments. The fourth expert reviewer is the Chief of Behavioral Pediatrics at a major university offering specialized training in the field. Feedback from all focus groups and expert reviews was incorporated into the final version of the PSI; six revisions of the PSI took place prior to beta testing. (Table 1) Administering the PSI to a Development Sample The second phase of this project involved piloting the PSI in the pediatric primary care setting. The piloting of the PSI took place over approximately a five month period.

Participants
Fifty-six physicians treating children and families in the greater Lansing, Detroit, and Ann Arbor, Michigan areas were invited to participate in this phase of the study. Sixteen physicians consented to participate, however, and data collection was only feasible for four physicians due to logistic constraints of their practices. Two hundred and sixteen adult parents of children aged 1-4 also participated in piloting the instrument in the context of their pediatric primary care provider's office.

Procedures
All procedures for this study were reviewed and approved by the Internal Review Board at the University of Nevada, Reno prior to data collection. A randomized block design was employed for data gathering and analysis; physicians were the blocking factor. Physicians were randomly assigned to either use the PSI during a patient visit or to follow usual assessment procedures during the visit. Therefore, each physician participated in both the experimental and control conditions. Data were analyzed using SPSS with logistic regression and ANOVA models.

Recruitment
Parent participants were recruited to participate when they presented their child for medical care and met the criterion for inclusion. Research packets with a letter of invitation as the cover page were given to each parent as they checked in with the receptionist, consistent with HIPPA requirements. Parents indicated their consent by completing the questionnaires, rather than by completing an informed consent document, in order to maintain anonymity. Provider questionnaires were paired with parent questionnaires by using a code number which was printed on all questionnaires pertaining to a particular visit.

Experimental Condition Procedures
In the experimental condition, parents bringing their children to primary care visits (either well-child or illness/injury) were given a three page packet of research materials as they reported to the receptionist. A brief (1-3 sentences) introduction inviting parents to participate was the first page of this packet. The second page of the packet was the Background Questionnaire, created for the purpose of this study, which asked basic demographic questions. The third and final page of the packet was the PSI (double-sided), the instrument being evaluated. It is estimated that completing the Background Questionnaire and the PSI took parents no more than 5-7 minutes prior to their physician visit.
Parents were asked to give the completed research materials to their provider when they entered the examination room. It was up to the provider to decide whether they used the PSI during the visit; therefore, there was no direct experimental manipulation of any physician behaviors during the visit. At the completion of the visit, either the provider or the parent deposited the research materials in a secured marked box which was present in each of the exam rooms. It is noteworthy that none of the participating pediatric practices had a behavioral health specialist on staff at the time of the study.
Following the visit, both providers and parents were asked to complete brief questionnaires about what happened during the visit. Specifically, the first question asked whether or not any psychosocial intervention occurred during the visit. Secondly, parents were asked if any referral for specialized services was made during the visit. Third, they were asked if the PSI changed the outcome of the office visit (providers only). Finally, both parents and providers were asked to rate their satisfaction with the visit overall on a Likert-style scale from 1-100, with 100 anchoring greatest satisfaction.
Parents were given the Post-Visit Questionnaire by the receptionist as they checked-out of the clinic. Therefore, their providers did not have access to their responses. Parent Post-Visit Questionnaires were then deposited in a marked box at the receptionist's window. For physicians, the Post-Visit Questionnaire was attached to the chart for each visit. These were then deposited by the physician in a marked box located in a convenient place in the clinic (e.g. records room, lounge, etc.). It was estimated that completing post-visit questionnaires took less than 2 minutes each for providers and parents.

Control Condition Procedures
The same procedures were followed in the Control Condition as in the Experimental Condition, except that parents were not asked to complete the PSI, therefore, physicians were not exposed to the screening data.

Demographics
A total of 214 parent-provider dyads participated in the piloting of the PSI across both conditions (108 controls, 105 experimental visits). Ninety-four percent of parent respondents were the biological parents of the child they were bringing for medical care, 3.4% were adoptive parents, and the remaining respondents were grandparents, foster parents, and/or legal guardians (1.1%, .6%, and .6%, respectively). As they were functioning in the capacity of parent, accounted for the majority of the respondents bringing a

Acta Psychopathologica ISSN 2469-6676
Child Behaviors Not a concern A concern Daily Activities and Behaviors

Acta Psychopathologica ISSN 2469-6676
homes. The majority of parent respondents reported that they are currently married (86.4%); with 5% each reporting that they are either divorced or has never been married. Ninetysix percent of parent respondents reported that they are their child's primary caregiver. When questioned as to the reasons that they were presenting their child for pediatric primary care, 40% of respondents reported that their child was sick, 31% were attending a well-child visit, 21% were coming for a follow-up visit from an illness or injury, and 6.3% were obtaining a physical.

Provider Practice Data
As described above, four physicians participated in PSI pilottesting. These subjects reported that they had been providing medical care for children on average 11.3 years (range 4-22 years). When asked how comfortable they are providing advice or guidance to parents regarding their parenting practices on a child for a primary care visit, and for the sake of simplicity, all of these groups will be referred to as parents.
Eighty-four percent of parent respondents were female. Most parents have some concerns about raising their family. Please check if the following common problems are a concern for you or anyone else who helps to take care of your child. This page deals with common concerns about children and the back side deals with common concerns about parenting.

Acta Psychopathologica ISSN 2469-6676
scale of 1-10 (1= not at all comfortable, 10=very comfortable), the mean rating was 7.3, "moderately comfortable" (range [6][7][8]. When asked how comfortable they felt providing advice or guidance to parents specifically regarding appropriate child behavior or child behavior problems, the mean response of the physician participants was 6.8, or "moderately comfortable" (range 6-7). When asked how much training they had had in behavioral pediatrics on a scale of 1-10 (1=not very much, 10=in excess), the average rating was 5.3, or "sufficient training" (range 3-6). When asked whether or not they felt that their training was adequate in behavioral pediatrics on a scale of 1-10 (1=not at all, 10=very much so), the mean rating for the four physicians was 4.3 (range 3-5), a little less than "sufficiently adequate." Finally, when asked to rate the adequacy of their knowledge of local behavioral healthcare resources on a scale of 1-10 (1=not adequate, 10=very adequate), physicians gave a mean rating of 5 (range 3-7), "moderately adequate."

Descriptive Statistics of the PSI
Eighty-seven of the 105 parent respondents in the experimental condition completed the entire PSI. The PSI contains 50 items and each item could be rated either a concern or not a concern by the parent respondent. Seventy-eight percent of parents completing the PSI reported at least one concern, with the mean number of concerns being four, the median three, and the mode one.
The most frequent concerns reported by parents completing the PSI were balancing responsibilities (e.g., home and work), endorsed as a concern by 24.1% of respondents, followed by temper tantrums (23%), and obeying, financial stability, and consistency of discipline (20.7% each). Concerns relating to domestic violence, showing affection to their child, and following up with medical appointments were not endorsed by any parent participants.
When parents were asked following their visit whether or not using the PSI made it easier to talk about their concerns during their visit, 58.7% reported that it was helpful. When asked if the PSI reminded them to initiate discussions or ask questions about their child's behavior during the visit, 61.3% reported that it had. When parents were asked if they would like it if their pediatric primary care provider always used the PSI when they came to an appointment, responses were approximately evenly distributed, with 48.6% responding that they would like that practice and 51.4% responding that they would not.
When providers were asked whether or not the PSI had any impact on the interventions they provided, participants indicated that only in 7.6% of visits that the PSI had made a difference in their behavior.

Hypothesis Testing
Hypothesis 1: Using the Pediatric Screening Inventory would result in increased detection of problems in child and/or family functioning in comparison to assessment as usual.
Hypothesis 1 was tested using a logistic regression model. The independent variables in this model were the categorical variable of condition (experimental vs. control) and the categorical variable of physician, which was coded to test for physician differences. The dependent variable was whether or not a behavioral concern was detected during the visit. Behavioral problems were detected 68 times (65% of visits) in the experimental condition and 26 times (24% of visits) in the control condition. A significant main effect for condition (sig=.000, p<.001) was found with no significant physician effect and no significant interaction. The odds of detecting a behavioral concern in the experimental condition were approximately twice that of the control condition. By these criteria, Hypothesis 1 was supported.

Hypothesis 2: Use of the PSI would be associated with (a) an increased rate of behavioral or parenting intervention during the primary care visit and (b) an increased rate of referral for services outside of the primary care setting.
Hypothesis 2 was tested using a logistic regression model. The independent variables in this model were the categorical variable of condition (experimental vs. control) and the categorical variable of physician, which was coded to test for physician differences. The dependent variables were whether or not a (a) behavioral intervention and/or a (b) referral occurred during the visit. Behavioral interventions occurred 42 times (40% of visits) in the experimental condition and 26 times (24% of visits) in the control condition. Referrals for specialty care occurred 7 times in the experimental condition (7% of visits) and 10 times (9% of visits) in the control condition.
When the logistic model for intervention was tested, a significant main effect was found for condition (sig=.011, p<.05), with no significant physician effect and no interaction. The odds of a behavioral intervention occurring in the experimental condition were 1.6 times more likely than in the control condition. When the logistic model for referrals was tested, no significant effects were found for condition or physician and there were no significant interactions. By these criteria, Hypothesis 2 was supported for interventions, but rejected with regard to referrals.

Hypothesis 3: The PSI would be associated with greater physician and parent satisfaction than assessment as usual.
To test this hypothesis, satisfaction was rated as a continuous variable on a Likert scale from 1-100 (100=very satisfied). Of a total of 170 physician satisfaction ratings, 161 ratings were "100." The mean rating of satisfaction was 97.8 and the mode and median were both 100. Likewise, of a total of 161 parent satisfaction ratings, 138 ratings were "100." The mean rating of satisfaction was 93.4 and both the mode and median were 100. Because of these ceiling effects, no significant differences were found between condition and physician and this hypothesis was rejected.

Discussion
The present study attempted to develop and evaluate a screening device to detect parental concerns regarding child behavior,

Acta Psychopathologica ISSN 2469-6676
parenting, and following medical directions in the context of routine pediatric primary care. The resulting questionnaire is a brief 50 item parental self-report measure entitled the Pediatric Screening Inventory (PSI).
When providers participating in the piloting of the instrument were asked to rate their comfort with providing behavioral guidance, they reported that they were only moderately comfortable doing so. When asked to rate the sufficiency of their training in behavioral pediatrics, physicians reported that they had received less than sufficient training on average, akin to the early findings of Jellinek (1982) [24] and Costello (1986) [25]. Also consistent with previous findings, physicians reported only moderately adequate knowledge of local behavioral healthcare resources. Taken together, these data suggest that pediatricians are not well-prepared to meet the behavioral needs of their patients and could benefit from collaboration with a behavioral health provider within the practice.
The majority of parents who completed the PSI (78%) indicated that they had a least one concern regarding their child's behavior, their parenting, and/or following medical directions. Notably, some of the more prevalent areas of concern reported by parents in this study (e.g., balancing responsibilities, financial stability) have not been previously examined in pediatric primary care, indicating that the PSI may have provided an impetus for parents to communicate genuine concerns relevant to the well-being of their child that would have otherwise not been known by their medical provider.
In this vein, a little more than half of parents surveyed indicated that using the PSI made it easier for them to remember and initiate discussions about their behavioral concerns. Correspondently, about half of parents reported that they would approve of their pediatrician using the instrument routinely. Presumably these were the same parents for whom the instrument was a useful bridge between their concerns and communicating their concerns to their child's doctor. However, when physicians were asked to report whether the PSI had changed their clinically relevant behavior during visits, the vast majority reported that it had not.
These findings suggest that while the PSI may be perceived as a useful tool by parents, the use of the PSI may not lead to any changes in provider behavior. Ultimately, this disconnection between assessment and treatment could lead to increased frustration in parents and providers, as parents would have presented concerns to their child's physician but the physician was either unable or unwilling to address these concerns, given practice and training constraints. While this question was not examined in this study, it is also possible that this apparent breach may actually serve a protective function if it were the case that increased detection may lead to increased and potentially iatrogenic treatments being applied by clinicians who are not sufficiently trained and/or skilled to do so (e.g., pediatricians not trained in behavioral interventions failing to administer empirically-supported treatments in favor of inappropriately prescribing psychotropic medications). Furthermore, the question remains when (e.g., sick visits, well-child visits only, etc.) and how often (i.e., yearly, every visit) the PSI should be used. In addition, future research should investigate the extent to which physician behavior changes when a collocated behavioral health consultant is present in the pediatric practice.
Overall, the PSI improved detection of behavioral concerns significantly over assessment as usual. Even in the control condition, however, the rates of detection exceeded those previously reported in the literature when parents were not prompted to report concerns (e.g., McCue Horowitz et al., 1998; Taaffe Young et al., 1998) [26,12]. What is not known is how physicians in the control condition assessed for behavioral concerns. Future research also needs to investigate the sensitivity and specificity of the PSI as it may be the case that some of the identified behaviors were false positives. Because of the randomized block design, carryover effects are also likely, which suggests that perhaps physicians in control condition visits were more likely to assess for behavioral concerns because of having also recently participated in experimental visits. These effects may, in part, explain why detection in the control condition was higher than predicted.
Overall, the rate of detection in the experimental condition matches reported rates in previous research, while the control condition (though comparable to other detection studies) falls far short of adequately detecting behavioral concerns in comparison to the experimental condition and known prevalence rates. These preliminary findings suggest that the PSI may improve current clinical practices by greatly improving physician awareness of behavioral concerns of their patients' parents.
Similar to detection, rates of intervention occurred at significantly higher levels in the experimental condition than in the control condition. However, in the experimental condition, far more behavioral problems were detected than were addressed, while in the control condition the rates of detection and intervention matched. A question raised by this finding is whether the PSI was accurately detecting behavioral concerns (those warranting intervention) in the experimental condition, but barriers in the visit impeded intervention occurring at a higher rate. For example, parents may have been identifying concerns through the use of the PSI, but factors such as limited time during the visit and/or physician reluctance to treat may have interfered with an intervention actually taking place. Additional research would be necessary to determine what variables may have interfered with physicians' ability to intervene when behavioral problems were detected, but these findings may have significant implications for the system of pediatric primary care. For example, if it is the case that insufficient training and/or time during visits are factors that interfere with the delivery of appropriate behavioral interventions; this may suggest a need for a more radical change in the structure of typical pediatric primary care. One solution to the dilemma of having inadequate resources in primary care to provide appropriate behavioral care would be to include in the treatment team a co-located pediatric behavioral health specialist, such as a specially-trained psychologist.
It is also interesting to note that the intervention findings do not support the physicians' reports that their behavior was unaltered by using the PSI, given that rates of intervention were much higher in the experimental condition. Was it the case that physician behavior was influenced by the PSI and yet they did This article is available from: www.psychopathology.imedpub.com

Acta Psychopathologica ISSN 2469-6676
not perceive it to be so? Physicians may have increased their rate of intervention because of the stimulus cues inherent to participating in the research, without the PSI serving as a specific cue. It is also plausible that physicians consulted the PSI but perceived that they would have been able to detect and address the same concerns with usual methods of assessment, leading to their self-report that the PSI did not impact their intervention. Unfortunately, this question cannot be answered from the current data and will require additional scientific inquiry.
To summarize the findings on the ability of the PSI to increase the rates of behavioral intervention in pediatric primary care, the outcomes of the statistical tests for significance indicate that the PSI condition far exceeded the control condition in rates of intervention. However, this apparent change in behavior may or may not be attributed to the use of the PSI, particularly given the reports of physicians that the PSI did not impact their interventions.
Across both the experimental and control conditions, referrals for specialty care occurred very rarely. Furthermore, no differences were noted between the two groups on this variable. These findings are particularly interesting in the context of the findings previously described indicating that relatively high levels of behavioral concerns and low levels of behavioral intervention were reported. One would expect that if there was a gap between clinical concerns and intervention during the visit that referrals would fill that gap, however, this was not borne out empirically. One possible explanation for the low level of referrals was the physicians' self-report of only moderately adequate knowledge of local behavioral health resources. If this was the case, then training in local resources should theoretically rectify the problem. However, if there were other reasons that physicians were not referring their patients for additional specialty care, such as insurance limitations, a belief that behavioral interventions are ineffective, or a dearth of available resources in the community, possible solutions may be much more difficult to develop. Moreover, across both conditions (experimental and control) there appeared to be a ceiling effect in the ratings of physician and parent satisfaction. Satisfaction ratings were uniformly much higher than anticipated. In fact, this invariance in the data precluded meaningful statistical analysis of the satisfaction data.

Limitations of the Current Study
Although the sample size was sufficient for preliminary testing of the PSI as evidenced by the significant findings in two important areas (detection and intervention), it was not sufficiently large to draw conclusions about the utility of the PSI or the generalizability of the data to other populations of providers or parents. Specifically, only a small number of providers participated in this study, all of whom shared the same two offices and self-selected to participate based on their self-described interest in behavioral pediatrics. It is quite possible that lower rates of assessment and intervention would be found in other samples of pediatric providers. Furthermore, all of the participating physicians were pediatricians; however, many physicians providing care to children and families are general practitioners and as such would have different training histories and may have different practice patterns. Therefore, the findings with regard to physician behavior have only limited generalizability.
At this stage of instrument development, it would be premature to recommend that the PSI be disseminated for use in pediatric primary care. However, the results were promising and no ill effects related to using the PSI (such as decreased satisfaction) were reported. Based on the findings reported above, it is reasonable to suggest that the PSI continue to be used in pediatric clinics in the context of continued instrument development.
There are several logical steps which need to be taken in order to continue to develop the PSI. First and foremost, the psychometric properties of the PSI need to be examined. First, the PSI needs to be administered to a larger and more diverse sample of parents.
In doing so, it would be helpful to establish test-retest reliability by administering the instrument to the same set of parents twice with a one-week time interval in between (enough time to reduce carry-over effects but a small enough time frame that we would not expect meaningful changes in concerns). Following preliminary analysis of reliability, each of the 50 items should be carefully examined independently and in relationship to the other items, likely utilizing exploratory factor analysis. Items which are endorsed infrequently or correlate highly with other items should be evaluated for possible exclusion. In addition, the factor analytic process should also examine the extent to which the items correspond to the three main areas of interest on the PSI.

Conclusions
In summary, support was found for the two main hypotheses in this study. The PSI may be useful in improving the detection of behavioral problems and the occurrence of behavioral interventions in pediatric primary care. No evidence was found to support the PSI increasing the rate of referrals for specialty care nor that the use of the instrument improves either parent or physician satisfaction with pediatric primary care visits. However, its best use may be in integrated primary care in which behavioral health consultants are present. Collectively, these data suggest that the PSI may have some utility as a new screening tool in pediatric care and additional research is warranted to further explore the psychometrics properties and utility of the PSI.