SOCW 6311 wk 11 Discussion 1: Translating Knowledge From an Evaluation Report

‘Keeping families and children in mind’: an evaluation of
a web-based workforce resourcecfs_731 192..200

Andrea Reupert*, Kim Foster†, Darryl Maybery‡, Kylie Eddy§ and Elizabeth Fudge¶
*Senior Lecturer, Department of Rural and Indigenous Health, Monash University, Moe, Victoria, †Associate Professor,

Mental Health , University of Sydney, Camperdown, NSW, ‡Associate Professor of Rural Mental Health,

Department of Rural and Indigenous Health, Monash University & Gippsland Medical School, Moe, Victoria, and

§Workforce Development Officer, ¶Project Manager, Children Of Parents with a Mental Illness (COPMI) national

initiative, North Adelaide, South Australia, Australia

A B S T R AC T

This study outlines pilot evaluation data of the web-based training
resource ‘Keeping Families and Children in Mind’, designed for clini-
cians who work with families where a parent has a mental illness. The
resource was developed from scoping existing workforce packages
and in consultation with consumers, carers, researchers and mental-
health clinicians. Preliminary evaluation data were collected from an
urban and a rural site in Australia via focus group interviews and pre-
and post-training questionnaires to ascertain the experiences of those
who participated in the training. Additionally, training facilitators
were invited to maintain journals in order to identify planning and
implementation issues when using the resource. Post-training, partici-
pants emphasized the need to work collaboratively with others, as
well as the importance of acknowledging and working with the family
members of consumers, especially children. Also, participants
reported positive changes in knowledge, skill and confidence when
working with families affected by parental mental illness. Facilitators
highlighted technology issues and the need to work interactively with
participants when using the resource. Recommendations regarding
policy and future research conclude this paper.

Correspondence:
Andrea Reupert,
Department of Rural and Indigenous
Health,
Monash University,
PO BOX 973,
Moe, Victoria,
Australia
E-mail: [email protected]

Keywords: children, evaluation,
families, parental mental illness,
web-based workforce training

Accepted for publication: August 2010

I N T R O D U C T I O N

Mental illness is a family affair, particularly where a
parent, with dependent children, has a mental illness.
Several studies indicate that children where a parent
has a mental illness may be at twice the risk of devel-
oping a mental illness diagnosis compared to other
children (Black et al. 2003; Park et al. 2003; Cunning-
ham et al. 2004; Leschied et al. 2005; Edwards et al.
2006). Other studies highlight the range of behav-
ioural, interpersonal, academic and other difficulties
that children of parents with a mental illness might
face (Rutter & Quinton 1984; Farahati et al. 2003;
Maughan et al. 2007; Reupert & Maybery 2007). An

epidemiological study has estimated that between 21
and 23% of all families have, or have had, at least one
parent with a mental illness (Maybery et al. 2009).
Thus, given the prevalence of families affected by
parental mental illness and the potential difficulties
they face, it is important that the mental-health work-
force is appropriately skilled at identifying and subse-
quently intervening with children and their parents.
This paper describes a web-based training resource,
‘Keeping Families and Children in Mind’ designed for
the mental-health workforce, and the results of a pilot
evaluation of the resource.

While there is ample evidence highlighting the need
for early intervention, children living in families where

doi:10.1111/j.1365-2206.2010.00731.x

192 Child and Family Social Work 2011, 16, pp 192–200 © 2010 Blackwell Publishing Ltd

a parent has a mental illness have been described as
‘hidden’, because clinicians are often unaware that
consumers are parents with dependent children
(Fudge & Mason 2004). In the USA, it is suggested
that agencies take a categorical approach and focus on
either the child (e.g. in terms of child protection) or
the adult (e.g. for his or her mental-health needs)
(Nicholson et al. 2001). Maybery & Reupert (2006)
found that while many Australian adult mental-health
clinicians want to work with all family members, they
report clear skill and knowledge limitations, a finding
also confirmed in a Finnish study with psychiatric
nurses (Korhonen et al. 2008). Slack & Webber (2008)
found that even though many adult mental-health
workers favour supporting children of consumers,
they did not necessarily consider it their role to do so.
Maybery & Reupert (2009) summarize workforce
barriers in terms of (i) policy and management; (ii)
inter-agency collaboration; and (iii) clinician attitude,
skill and knowledge.

Notwithstanding these barriers, acknowledging and
working with family members has been shown to be
beneficial to the consumer, his or her children and
other family members (Glynn et al. 2006; Beardslee
et al. 2008). For example, it has been found that a
family-focused intervention was effective in reducing
the exacerbations in schizophrenia, improving medi-
cation compliance and reducing or eliminating sub-
stance abuse (Glynn et al. 2006). Family-sensitive
practice is beneficial to the consumer as well as other
family members, by reducing a family’s subjective
burden of care and increasing their level of self-care
and emotional functioning (Glynn et al. 2006). Fur-
thermore, acknowledging and working with children

of parent consumers improves family functioning and
children’s understanding of their parent’s disorder as
well as a reduction in children’s internalizing symp-
toms (Beardslee et al. 2008). Given the efficacy of a
family-sensitive approach, it is imperative that training
is designed and developed in ways that addresses
the current skill and knowledge gaps found in the
workforce.

In response to the training needs of the mental-
health workforce, a resource ‘Keeping Families and
Children in Mind’ was developed by the Australian
National COPMI (Children of Parents with a Mental
Illness) initiative through scoping existing workforce
packages and then identifying main themes and issues
across these packages (Reupert et al. 2009). Addition-
ally, the resource was developed using a Delphi
process (see Note 1) with 14 experts consisting
of consumers, carers, researchers and mental-health
clinicians (Whitman et al. 2009). These experts
responded to questions about curriculum content and
teaching processes in three Delphi ‘rounds’ until con-
sensus was reached. In this process, experts were
asked to summarize themes (generated from previous
Delphi rounds) that resulted in the final six core
modules of the resource (see Table 1).

The resulting Keeping Families and Children in Mind:
COPMI Mental HealthWorker Education Resource deliv-
ers an interactive, audio and video material using Web
2 technology. Clinicians might focus only on those
modules that are of interest and/or need or undertake
all six modules. The resource includes a variety of
educative web pages and links, and video and audio
inserts of families describing what it is like to live with
parental mental illness as well as clinicians reporting

Table 1 The six modules of ‘Keeping Families and Children in Mind: COPMI Mental Health Worker Education
Resource’

1 Mental Health and Families – introduces a family where a parent experiences mental illness. Information is also provided
by parents, children and workers about factors that contribute to mental health and illness, stigma and mental illness and
family support.

2 The Parent – introduces a second family. Information is provided about the impact of mental illness on parenting, the
impact of parenting on mental illness and the recovery process.

3 The Child – provides an opportunity to reflect on the experiences of children in two families where a parent experiences
mental illness. Information is provided on risk and protective factors that influence child well-being, including the impact
of parental mental illness on child development. There are demonstrations of how talking to children can assist in their
understanding of what is happening at home.

4 The Family – highlights the importance and influence of the family unit in the recovery of a parent who experiences mental
illness, including influences on family functioning, family resilience and working with families using a strengths-based
approach.

5 Carers – presents the perspective of family carers with a particular emphasis on the issues faced by young carers and
grandparents who provide care for children of parents who experience mental illness.

6 Putting it into Practice – provides opportunities for learners to reflect on their work practice at an individual and systems
level in regards to supporting families where a parent experiences mental illness. It provides practical examples of what
workers are currently doing across Australia and an extensive list of accessible resources.

‘Keeping families and children in mind’ A Reupert et al.

193 Child and Family Social Work 2011, 16, pp 192–200 © 2010 Blackwell Publishing Ltd

about their experiences working with such families.
The resource also provides scenarios about fictitious
families that encourage mental-health clinicians to
reflect on their clinical practice. Clinicians are able
to access the resource in a web-based, self-paced
mode, or alternatively, attend facilitator-led training
using the resource in a group format. The resource is
freely accessible at http://www.copmi.net.au/worked/
index.html

At the point of writing, several Australian state
mental-health services have indicated that they will
incorporate this resource as part of a large ‘roll out’
of training, with similar indications from overseas
researchers and trainers (personal correspondence to
the authors). Consequently, as there are likely to be
hundreds and perhaps thousands of clinicians who
will use this resource, it is essential to report initial
data about the utility of the resource. Currently, a
further evaluation of the large-scale roll out is
planned, although it will take some time to collect and
analyse a larger data set. Thus, this paper summarizes
preliminary efficacy data from the piloting of the
training resource.

M E T H O D

Training was conducted at a rural and an urban site,
in two Australian states (Victoria and Tasmania). The
workshops were each presented by two female facili-
tators with extensive backgrounds in mental health
and workforce training. The training for the rural site
took place over two half days with 23 participants,
while training in the urban site went for a full day
and included 14 participants. Training participants
came from a range of services including child and
adolescent and adult mental-health agencies, non-
government agencies and hospitals, and from disci-
plines including psychiatry, social work, education
and consumer and carer groups. The purpose of the
training, at both sites, was to introduce participants to
the broad issues related to families where a parent has
a mental illness, and to identify where to access infor-
mation about mental illnesses for families and clini-
cians. Participation in the training was not dependent
on participation in the evaluation, and ethics approval
was provided by the Monash University Standing
Committee on Ethics in Research Involving Humans.

Data were drawn from three sources. First, partici-
pants were invited to participate in focus group
interviews to gain their views on the resource
and, second, to complete anonymous pre- and
post-program questionnaires to quantify short-term

changes in learning. Finally, training facilitators
completed journals in order to identify planning and
implementation issues.

Focus group interviews

Across the two focus groups held at the two sites, there
were a total of 28 participants, eight men and 20
women. Focus group questions aimed to determine
potential changes in knowledge, attitude and practice,
for example:

• What did you learn, if anything, as a result of being
shown the resource?

• Has the resource changed any attitudes or ways of
looking at consumers you previously had, if at all?

• In what ways, if any, do you think the resource will
change the way you practice?
With permission, the focus groups were audiotaped

and subsequently transcribed. Data were then analy-
sed using an open coding system, attaching labels to
lines or paragraphs of data and then describing the
data at a concrete level, before moving to a more
conceptual level (Anfara et al. 2002), first within each
focus group transcript and then across the two tran-
scripts. This descriptive and iterative analytic process
aimed to meaningfully classify codes into themes.

Questionnaires

Twenty-seven participants (20 females and seven
males) completed identical pre- and post-training
questionnaires that assessed change in participant
confidence, knowledge and skill when working with
the parents, children and families. A coding system
was employed to ensure anonymity but also to allow
matching of pre- and post-data for analysis. The
specifically designed ‘Workforce Questionnaire’ was
employed. The questionnaire was initially developed
in an adult mental-health clinician sample (see
Maybery & Reupert 2006) and is currently being
employed across five Victorian Health regions and by
researchers in British Columbia (Canada) in a large
Australia-Canada benchmarking questionnaire of
family-focused practice. A brief version of the ques-
tionnaire, with 27 items, was employed here. The
questionnaire has excellent content and construct
validity and unpublished data highlight good internal
reliability of subscales ranging from Cronbach alphas
of 0.87 to 0.72 for relevant subscales. Participants
responded on a scale from 1 (strongly disagree) to 7
(strongly agree) for each item.

‘Keeping families and children in mind’ A Reupert et al.

194 Child and Family Social Work 2011, 16, pp 192–200 © 2010 Blackwell Publishing Ltd

Facilitator journals

The four training facilitators were invited to complete
various pre-determined questions in a journal at two
time points, before the workshop, to capture planning
issues, and then after the workshop, to highlight imple-
mentation matters. Pre-training questions included:

• What issues did you encounter in planning the
training? How did you overcome them? Or what
was required to overcome them?

• How did you decide which aspects of the resource
to use/not use?

Post-training questions included:

• What worked well when using the resource? What
didn’t work so well and why?

• What, if anything, do you think needs to change
about the resource?

• What facilitation skills were particularly important
in using the resource in workshop format?

• What issues did you encounter when using the
resource?
Facilitators were instructed to not censure them-

selves, nor be concerned about spelling or grammar.
They were not required to include their name on the
journal. Journals were analysed using the same the-
matic analysis approach employed with the focus
group transcripts.

R E S U LT S

Focus groups

Three themes emerged of participants’ experiences of
the training in terms of (i) views on the resource; (ii)
impact of the training on attitude and practice; and
(iii) possibilities for implementation.

Views on the resource

Participants expressed very positive views on the
resource, identifying it as a quality production with
comprehensive content and as a valuable resource for
clinicians from a range of child and adult healthcare
settings, particularly in rural and remote settings. The
emphasis on working together with families, rather
than simply focusing on diagnosis and assessment,
was appreciated. A strength of the resource was its
interactivity and the life-like depiction of issues expe-
rienced by children and families:

I thought that the narratives and the case studies were excel-

lent around mental health and placing that in the context of

the family . . .

Participants particularly enjoyed the video clips,
which they considered very effective for learning, ‘. . .
it holds your attention much more than just reading
. . . it brings it to life’.

Some participants considered the resource to be
quite lengthy and that it took time to learn to navigate
and become familiar with it. They acknowledged that
working through the resource in addition to their
workload would require commitment, ‘. . . if you
don’t block out time, you’ll just be too busy’.
However, the flexibility and ability to choose which
modules and content to read were helpful, ‘. . . the
advantage is that you can go in and pick the bits you
need’.

In the workshops, the opportunity to interact and
discuss issues with others, including consumers and
carers, and listen to their insights and practices, added
to the experience.

. . . one of the things that I got the most out of was having a

consumer presence in the training session. It was actually him

and his carer . . . so you were able to get the carer’s perspec-

tive, the consumer’s perspective, and that was good.

The training facilitators’ ability to identify each
group’s needs and tailor the workshop accordingly
was an important aspect of training. Reliable internet
access was integral to its effectiveness, with broadband
access and slow running of computers, a frustration
for some participants.

Impact of the training on attitude and practice

The most prominent impact of the training was par-
ticipants’ heightened awareness of the need to care
for all family members and the importance of taking
time to sit and talk with families. For some, training
reaffirmed that they were on the right pathway in
what they were already practising, while for others,
the resource challenged their attitudes towards
families.

. . . it makes you aware of what you’re bringing to the inter-

action [with families]. Am I bringing pre-conceived ideas and

biases . . . which may or may not be an accurate view?

Training highlighted the need for clinicians from a
range of backgrounds and settings to collaborate in
care: ‘I think what’s come out of it, very much so, was
the importance of everyone who’s working with that
family, working together . . .’ On their return to work,
participants felt their raised awareness would encour-
age them to look further into a family’s needs, and
ensure that they scheduled appointments where they
could see the children as well as the consumer parent.

‘Keeping families and children in mind’ A Reupert et al.

195 Child and Family Social Work 2011, 16, pp 192–200 © 2010 Blackwell Publishing Ltd

The training added to their knowledge about chil-
dren’s understandings of mental illness at different
developmental stages, and the need to tailor informa-
tion and address issues accordingly.

. . . because I have limited experience in working with young

carers . . . it really opened my mind to some of the issues for

young carers . . .

In addition to being more mindful of children and
families, participants identified the need to develop
policies around family-inclusive practice in their
service. They acknowledged that while the resource
might impact on individual practice, unless manage-
ment supported child and family-inclusive practice as
core business, the impacts could be limited.

. . . the practice will change the individual, and ideally if

they’re supported by the management structure . . . it spreads

through the organisation . . .

Possibilities for implementation

Participants saw a range of possibilities for imple-
menting the resource within their organization as well
as in other organizations and contexts. This included
using the resource in professional development ses-
sions for staff over a 12-month period, including it as
part of new staff inductions, and using it as a training
resource for students on clinical placement:

. . . I can see the potential for it to change practice within my

organisation.We have a carer support program . . . and I’d like

to deliver [a module] to the carer support team.

Participants made a number of suggestions for
improving the resource, such as adding further infor-
mation on mental illness and drug and alcohol use for
users who might not have a background in mental
health, and adding links to relevant local and/or
regional resources and services for clinicians and
family members. Finally, participants suggested that
the resource should include a section on collaborative
practice and possible protocols for how agencies can
work with each other with families.

Questionnaire

Table 2 presents participant mean scores, standard
deviations and paired sample t-statistics for pre- and
post-responses to items about clinicians’ confidence,
knowledge and skill concerning family issues. Almost
all participant responses to knowledge (see Note 2)
items moved in the expected direction (note that some
items are negatively worded) and just under half of the

26 items showed a significant change – again all in the
expected direction. Items shown in bold are signifi-
cant at 0.05 level and actual P-values are shown in the
right-hand column for each item.

Significantly different variables generally reflect
changes that occurred at both the rural and the urban
sites.

Program facilitator journals

Planning issues

The main planning issues for facilitators were related
to information technology and the knowledge to
appropriately download and save different aspects of
the resource. Working with others with technological
expertise was or would have been useful. Facilitators
indicated that having a thorough knowledge of par-
ticipants’ background (profession and organization)
was important in deciding which aspects of the
resource to employ.They also suggested that the work-
shop needed to incorporate interactive as well as
didactic components.

Implementation issues

According to facilitators, the family videos were a
powerful and effective way to encourage participant
discussion and reflection. Problems experienced with
the resource centred primarily on facilitators’ own
facilitation skills (time management, co-facilitating,
providing too much information, not outlining learn-
ing objectives) rather than the resource per se. All
agreed that the ability to work with groups and
manage group discussions were important facilitation
skills when using the resource. Facilitators employed
small and large group discussion groups to expand on
key learnings when using the resource. Handouts, case
notes, and, in particular, local and national services,
were used in conjunction with the resource. Overall,
modules were used in a theme-based way, related to
the needs of training participants and their particular
clientele.

D I S C U S S I O N

Overall, pilot evaluation data of the ‘Keeping Families
and Children in Mind’ resource demonstrated high
participant satisfaction and significant self-reported
changes in knowledge, skill and confidence. Thus,
results demonstrated the utility of the resource as a
training tool for clinicians working with families,

‘Keeping families and children in mind’ A Reupert et al.

196 Child and Family Social Work 2011, 16, pp 192–200 © 2010 Blackwell Publishing Ltd

with both participants and training facilitators valuing
the interactivity and flexibility of a web-based
resource. Participants, including those from adult
mental health, highlighted the need, post-training,
to work with all family members including children.
This is an important finding given that previous
research has highlighted clinicians’ reluctance to work
with the children of parent consumers (Slack &
Webber 2008). Participants also stressed the necessity
to collaborate with other agencies when working with

families, another substantial result, due to the lack of
inter-agency and inter-sectoral collaboration in this
area (Darlington et al. 2005). While the resource was
not compared in terms of delivery (that is, self-paced
mode vs. a group setting), the focus group data high-
light the significance for clinicians in being trained
alongside others, including consumers and carers, in
order to share insights, experiences and practices.

The quantitative questionnaire data tentatively
shows an improvement in participants’ knowledge,

Table 2 Scores and paired sample t-statistics for pre- and post-responses to items regarding clinicians’ confidence,
knowledge and skill when working on family issues

Questionnaire item
Pre Post M

t PM (SD) M (SD) diff

I am knowledgeable about how parental mental illness impacts on children and
families.

5.65 (0.85) 6.04 (0.77) -0.39 -2.30 0.03

I am not confident working with families of consumer-parents. 3.00 (1.87) 2.32 (1.68) 0.68 1.71 0.10
I am not knowledgeable about the key parenting issues for consumer-parents. 3.04 (1.68) 3.00 (2.04) 0.04 0.12 0.90
I am knowledgeable about the key things that consumer-parents could do to

maintain the well-being (and resilience) of their children.
4.96 (1.28) 5.38 (1.65) -0.42 -1.15 0.26

I am knowledgeable about the role of family carers and their influence on
recovery for consumers.

5.38 (1.06) 5.81 (1.17) -0.42 -1.23 0.23

I am not knowledgeable about the role of young carers in families where parents
experience mental illness

3.27 (1.66) 2.23 (1.56) 1.04 2.78 0.01

I am not confident working with children of consumer-parents. 2.92 (1.77) 2.50 (1.84) 0.42 1.17 0.25
I am knowledgeable about how the role of parenting impacts mental illness. 5.36 (1.04) 5.84 (0.80) -0.48 -2.39 0.03
I am not confident working with consumer-parents about their parenting skills. 3.35 (1.79) 2.58 (1.79) 0.77 2.08 0.05
I do not have the skills to work with consumer-parents about how parental mental

illness impacts on children and families.
3.23 (1.53) 2.31 (1.49) 0.92 3.27 0.00

I provide education sessions for adult family members (e.g. about the illness,
treatment).

4.63 (2.09) 4.42 (2.01) 0.21 0.41 0.69

I am skilled in working with consumer-parents regarding their parenting. 4.38 (1.84) 4.92 (1.77) -0.54 -2.01 0.06
I provide education sessions for children (e.g. about the illness, treatment). 3.94 (2.30) 4.06 (2.25) -0.12 -0.32 0.76
I am skilled in providing psychosocial-education to adult family members about

the mental illness.
4.00 (2.02) 4.86 (1.80) -0.86 -2.42 0.03

I regularly have family meetings (not therapy) with consumer-parents and their
family.

4.26 (2.26) 4.74 (1.88) -0.47 -1.21 0.24

I consider information from the carer or family when diagnosing and/or
treating the consumer-parents.

5.37 (1.50) 5.79 (1.47) -0.42 -2.04 0.06

I provide emotional support for family members and children. 5.48 (1.53) 6.09 (0.85) -0.61 -2.37 0.03
I do not refer children of consumer-parents to child-focused (e.g. peer

support) programs (other than child and adolescent mental health).
3.19 (1.78) 3.00 (2.05) 0.19 0.43 0.67

I do refer consumer-parents and their families for family therapy or
counselling.

5.09 (1.50) 5.48 (1.34) -0.39 -1.20 0.24

I provide written material (e.g. education, information) about parenting to
consumer-parents.

4.75 (1.59) 5.38 (1.47) -0.63 -3.72 0.00

I regularly provide information (including written materials) about mental health
issues to the children of consumer-parents.

4.10 (2.19) 5.29 (1.76) -1.19 -4.23 0.00

I often consider if referral to parent support program (or similar) is required by
consumer-parents.

5.09 (1.53) 5.65 (1.34) -0.57 -2.02 0.06

Rarely do I consider if referral to peer support program (or similar) is required
by my consumer-parent’s children.

2.35 (1.47) 2.13 (1.52) 0.22 0.59 0.56

I don’t provide information to the carer and/or family about the
consumer-parent’s medication and/or treatment.

3.82 (2.15) 3.77 (2.09) 0.05 0.09 0.93

I am aware of locally based resources for consumers who are parents. 4.96 (1.51) 5.76 (0.83) -0.80 -3.18 0.00
I am aware of locally based resources for families with a mentally ill parent. 5.38 (1.33) 5.85 (0.78) -0.46 -2.13 0.04
I am aware of resources available to the public about mental health and illness. 5.65 (1.13) 6.12 (0.71) -0.46 -2.07 0.05

‘Keeping families and children in mind’ A Reupert et al.

197 Child and Family Social Work 2011, 16, pp 192–200 © 2010 Blackwell Publishing Ltd

skill and confidence when working with families and
children, post-exposure to the resource. Almost half of
the 26 items showed a significant change, in the
desired direction.

Overall, while promising, these data are primarily
composed of self-reported data obtained immediately
after training occurred. In addition, the quantitative
findings must be considered in light of a small and
limited data set. The nature of the analyses under-
taken in the questionnaires promoted the likelihood of
Type 1 errors. There was neither a control group nor
a long-term follow-up of participants to ascertain
long-term behaviour change. It might also be assumed
that there was a sample selection bias, as those who
attended the training were aware of and motivated to
undertake family-sensitive practice. Thus, it remains
to be seen whether and how behavioural changes …