Development of the Programs Guide
Selection of programs
When compiling this guide, the KidsMatter Early Childhood team identified a large number of programs that were suitable for inclusion. To assist us with the selection process, a reference group comprising of experts in children’s mental health established a list of explicit inclusion criteria that align with the KidsMatter Early Childhood children’s mental health framework. These criteria are:
- The program was explicity designed for early childhood (children birth – 5 years of age), or their parents and carers, or early childhood staff.
- The program and staff training associated with it is accessible and widely available for purchase in Australia.
- The program can be implemented in early childhood services.
- The program contains sufficient information and training to allow it to be implemented.
- The program has explicit and documented aims and social learning outcomes for young children.
- Program authors consent to have their program included in the guide.
Inclusion of a program in this guide does not constitute endorsement by KidsMatter Early Childhood. Rather, inclusion merely acknowledges that the program meets the criteria described above. It is also important to note that the absence of a program from this guide does not neccessarily mean that it failed to meet these criteria. Rather, some programs were not included at the request of the author.
Program Effectiveness Criteria
One of the most important questions to ask about each program is “does it work?” To answer this question, the KidsMatter Early Childhood team, in conjunction with a panel of experts in children’s mental health, developed a star rating system. This rating system was designed to make it easy to compare programs, because it provides uniform information about the effectiveness of each program. These criteria are based on those used to ‘grade’ the quality of evidence that has come from evaluations of a program and are outlined below.
Program logic
This refers to the theoretical basis of a program. It determines the likelihood that a program will work effectively. Generally speaking, programs that are based on already established and tested theories are more likely to be effective than those that do not have a specific theoretical basis.
Research-based evidence
This refers to scientific research or studies that document the level of success a program has had at achieving its aims. KidsMatter Early Childhood only considered evidence from peer-reviewed sources (e.g., papers published in academic journals) for this criterion, because the quality of this research is easier to assess than that orignating from other, less informal, sources.
Research-based evidence is further broken down into the following categories:
-
Single impact, where there is a single research study and/or practice-based evidence documenting the program’s effectiveness;
-
Multiple impact, where there are multiple studies and/or substantial practice-based evidence documenting the program’s efficacy;
-
Maintenance of improvements, where there is at least one study documenting maintenance of improvements over time; and
-
Generalisation of outcomes, where improvements are observed in settings and groups beyond those the program was designed for.
Practice-based evidence
This refers to evidence of a program’s effectiveness that comes from applying it in the field, such as in early childhood services. Specifically, it gives an indication of how well a program works in practice, what has been reported about how it works and what outcomes have been observed following its implementation.
Star rating system
The KidsMatter Early Childhood star rating system provides an index of a program’s effectiveness by assessing it’s theoretical underpinnings and the quality and quantity of any research- or practice-based evidence available for it. The number of stars a program receives is determined by the number of criteria it meets- the more criteria met, the more stars awarded. For example, a high rating (e.g., 5 stars) means that the program has a sound theoretical base and there is a substantial amount of high quality evidence (either research- or practice-based, or sometimes both) available to suggest its effectiveness. In other words, a high rating indicates the robustness of program outcomes and the likelihood that outcomes similar to those already observed in research or in practice will occur upon implementation of the program in similar settings.
The table below shows the assessment used to determine a program’s effectiveness rating.
Star Rating |
Rating Description
|
 |
|
| |
|
| |
|
| |
|
| |
|
There are a number of reasons why a program may not have received a high effectiveness rating in this guide. Some programs might not have been formally evaluated but have been shown to work well in early childhood settings. Other programs may have been extensively evaluated with primary school-aged children but no evaluation has yet been undertaken with younger children. As such, neither of these reasons should be taken to mean that a program is ineffective or would not be useful in an early childhood setting.
For an introduction to the programs guide, click here.
For an introduction to the program summaries, click here.
For information about the development of the guide, click here.
For information about how to select a program for your service, click here.
