Each program that appears in the guide has been reviewed and summarised by the KidsMatter Early Childhood team using information and materials supplied by the program’s author(s). The summary of each program includes:
 
  • an overview of the program, specifying what type it is (e.g., universal, or prevention) and the target audience (e.g., children or parents and carers);
  • details of the KidsMatter Early Childhood component(s) to which it aligns; 
  • the aims and content of the program (i.e., skills taught);
  • details of how the program is implemented, including information about its structure (e.g., group or individual sessions), methods of delivery (e.g., role play or DVDs), who can deliver it (e.g. early childhood staff) and the training requirements to do so, and an estimate of cost;
  • research about the effectiveness of the program and its outcomes, wherever available; and
  • contact details for authors, materials and facilitator training (if required).
Early childhood services can use this information to determine if a program will meet the specific needs of their children, parents and carers or staff.
 

Development of the programs guide: Selection of programs

When compiling this guide, the KidsMatter Early Childhood team identified a large number of programs that were suitable for inclusion. To assist us with the selection process, a reference group comprising of experts in children’s mental health established a list of explicit inclusion criteria that align with the KidsMatter Early Childhood children’s mental health framework. These criteria are:
 
  • The program was explicitly designed for early childhood (children birth – 5 years of age), or their parents and carers, or early childhood staff.
  • The program and staff training associated with it is accessible and widely available for purchase in Australia.
  • The program can be implemented in early childhood services.
  • The program contains sufficient information and training to allow it to be implemented.
  • The program has explicit and documented aims and social learning outcomes for young children.
  • Program authors consent to have their program included in the guide.
Inclusion of a program in this guide does not constitute endorsement by KidsMatter Early Childhood. Rather, inclusion merely acknowledges that the program meets the criteria described above. It is also important to note that the absence of a program from this guide does not neccessarily mean that it failed to meet these criteria. Rather, some programs were not included at the request of the author.
 
One of the most important questions to ask about each program is “does it work?” To answer this question, the KidsMatter Early Childhood team, in conjunction with a panel of experts in children’s mental health, developed a star rating system. This rating system was designed to make it easy to compare programs, because it provides uniform information about the effectiveness of each program.  These criteria are based on those used to ‘grade’ the quality of evidence that has come from evaluations of a program and are outlined below.
 

Program logic

This refers to the theoretical basis of a program. It determines the likelihood that a program will work effectively. Generally speaking, programs that are based on already established and tested theories are more likely to be effective than those that do not have a specific theoretical basis.
 

Research-based evidence

This refers to scientific research or studies that document the level of success a program has had at achieving its aims. KidsMatter Early Childhood only considered evidence from peer-reviewed sources (e.g., papers published in academic journals) for this criterion, because the quality of this research is easier to assess than that originating from other, less informal, sources.
Research-based evidence is further broken down into the following categories:
 
Single impact, where there is a single research study and/or practice-based evidence documenting the program’s effectiveness;
Multiple impact, where there are multiple studies and/or substantial practice-based evidence documenting the program’s efficacy;
Maintenance of improvements, where there is at least one study documenting maintenance of improvements over time; and
Generalisation of outcomes, where improvements are observed in settings and groups beyond those the program was designed for.
Practice-based evidence 
This refers to evidence of a program’s effectiveness that comes from applying it in the field, such as in early childhood services. Specifically, it gives an indication of how well a program works in practice, what has been reported about how it works and what outcomes have been observed following its implementation.
 
The KidsMatter Early Childhood star rating system provides an index of a program’s effectiveness by assessing it’s theoretical underpinnings and the quality and quantity of any research- or practice-based evidence available for it. The number of stars a program receives is determined by the number of criteria it meets- the more criteria met, the more stars awarded. For example, a high rating (e.g., 5 stars) means that the program has a sound theoretical base and there is a substantial amount of high quality evidence (either research- or practice-based, or sometimes both) available to suggest its effectiveness. In other words, a high rating indicates the robustness of program outcomes and the likelihood that outcomes similar to those already observed in research or in practice will occur upon implementation of the program in similar settings.
 
The table below shows the assessment used to determine a program’s effectiveness rating.
 
Star rating system used in programs guide

Star Rating

Rating Description

one starone starone starone starone star

All effectiveness criteria were met.

Program has sound logic AND multiple studies and/or evidence from more than source in the field reporting improvement in target behaviours.

At least one study demonstrates maintenance of improvements AND generalisation of those to other settings and/or groups.

one starone starone starone star

Program has sound logic AND multiple studies and/or evidence from more than one source in the field reporting improvement in target behaviours.

At least one study demonstrates maintenance of improvements OR generalisation of those to other settings and/or groups.

one starone starone star

Program has sound logic AND one study and/or evidence from one source in the field reporting improvement in target areas.

At least one study demonstrates maintenance of improvements OR generalisation of those to other settings and/or groups.

one starone star

Program has sound logic AND one study and/or evidence from one source in the field reporting improvement in target areas.

one star

Program has sound logic but no research- or practice-based evidence of it’s effectiveness was available at the time this guide was compiled.

 
There are a number of reasons why a program may not have received a high effectiveness rating in this guide. Some programs might not have been formally evaluated but have been shown to work well in early childhood settings. Other programs may have been extensively evaluated with primary school-aged children but no evaluation has yet been undertaken with younger children. As such, neither of these reasons should be taken to mean that a program is ineffective or would not be useful in an early childhood setting.