4.A.3
Evaluating consultation and supervision: needs, formative and summative
ASPPB's competency statement 4.A.3: Consultation and supervision evaluation strategies and techniques (e.g., needs assessment, outcome evaluation, formative and summative evaluation)
Evaluation asks different questions at different times. A needs assessment comes before a service is designed; formative evaluation gathers information to improve a service or a supervisee while there is time to change; summative evaluation judges results at the end, for decisions such as funding or completion of training. Matching the method to the question, and deciding what level of outcome counts, keeps evaluation useful rather than ceremonial.
Key ideas
- Needs assessment
- A systematic look at the gap between current and desired conditions, and its causes, before a program or consultation is planned. It uses existing data, surveys, interviews or forums, and it shapes what the service should be.
- Formative and summative
- Scriven's distinction: formative evaluation informs improvement during the work, such as midpoint feedback to a trainee; summative evaluation reaches a judgment at the end, such as a final rating or a renewal decision.
- Process and outcome
- Process evaluation asks how a service is delivered relative to its plan; outcome evaluation asks whether intended changes occurred; impact evaluation compares results with a condition in which the program did not take place.
- Levels of training outcome
- Kirkpatrick's four levels separate participants' reactions, their learning, their behavior on the job and results for the organization or its clients. Satisfaction alone says little about the higher levels.
- Goal attainment scaling
- Kiresuk and Sherman's method sets a scale of possible outcomes for each individual goal before work begins and converts attainment to a standard score, so different goals can be compared and combined.
Worked example
A county funds psychologists to consult with child care centers about children whose behavior worries staff. The county asks a psychologist to design the evaluation for the service's first two years.
- Before expanding, assess need: which centers ask for help, what concerns staff raise and how often children are asked to leave care.
- Build in formative checks during year one, such as brief feedback from center directors after each consultation, so the service can be adjusted.
- Plan a summative evaluation for the end of year two that looks beyond satisfaction to staff practice in classrooms and outcomes for the children.
- Use individualized goals, scaled in advance, for the varied concerns each center brings, so the results can be combined across centers.
Decide what question each stage must answer, then choose the evaluation type and outcome level that answers it.
Check yourself
An evaluator checks whether a consultation service reached the sites it planned to, how many visits each site received and whether consultants followed the service's model. This is best described as:
Sources
- Scriven (1967), The methodology of evaluation, in Tyler, Gagne and Scriven (Eds.), Perspectives of Curriculum Evaluation, The formative and summative distinction
- Kidder and colleagues, CDC Evaluation Framework Work Group (2024), CDC Program Evaluation Framework, 2024, MMWR Recommendations and Reports, 73(6), 1-37 (PMID 39316770), Types of evaluation (formative, process or implementation, outcome, impact, economic); stage of development and the choice of evaluation type
- Kiresuk and Sherman (1968), Goal attainment scaling: A general method for evaluating comprehensive community mental health programs, Community Mental Health Journal, 4(6), 443-453 (PMID 24185570), Abstract: a measurable scale for each goal set before treatment and converted to a standardized T score
- Kirkpatrick and Kirkpatrick (2006), Evaluating Training Programs: The Four Levels, Third Edition, Reaction, learning, behavior and results
- Rossi, Lipsey and Henry (2019), Evaluation: A Systematic Approach, Eighth Edition, Needs assessment as the first evaluation question