Practice Evaluation and Research

1.Techniques for evaluating progress and effectiveness of intervention, treatment, and service plans

• Measure progress against the goals and objectives in the plan, starting with a baseline before intervention.
• Single-system (single-subject) designs track one client over time: AB (baseline, then intervention), ABA and ABAB (withdrawal and reintroduction strengthen causal inference) and multiple baseline designs across behaviors, settings or clients.
• Tools include standardized measures repeated over time (such as the PHQ-9 or GAD-7), goal attainment scaling, behavior counts, client logs and rating scales.
• Routine outcome monitoring, or feedback-informed treatment (for example, the Outcome Rating Scale and Session Rating Scale), uses session-by-session client feedback to catch lack of progress early.
• Review results with the client and change the plan when progress stalls; distinguish practical (clinical) significance from statistical significance.

In practice: A worker records the number of panic attacks a client has each week for three weeks before starting CBT, then continues counting during treatment, plotting an AB graph they review together every two weeks.

2.Methods of evaluating agency programs (e.g., needs assessment, formative/summative assessment, cost- effectiveness, cost-benefit analysis, outcomes assessment)

• A needs assessment, done before a program is designed, uses social indicators, surveys, key informant interviews, focus groups and community forums to identify the size and nature of a need.
• Formative evaluation happens during a program to improve it (including process evaluation of whether it is delivered as intended); summative evaluation happens at the end to judge its overall worth and outcomes.
• Outcome assessment asks whether participants changed in the intended ways (knowledge, behavior, condition), ideally against a comparison.
• Cost-effectiveness analysis compares costs with outcomes expressed in natural units (cost per person housed, per relapse prevented); cost-benefit analysis puts both costs and benefits in dollars and yields a benefit-cost ratio.
• Choose the method that fits the question: is there a need, is the program running as planned, does it work, and is it worth what it costs?

In practice: A board asks whether a new youth employment program should continue. The evaluator uses summative outcome data on job placement and a cost-effectiveness analysis of cost per youth employed, compared with the previous program.

Advanced Generalist items go deepest on program evaluation and costs; Bachelors items focus on recognizing the types and their purposes.

3.Basic and applied research design and methods, data collection and analysis methods, and methods to assess reliability and validity in social work research (e.g., objective, subjective, qualitative, quantitative)

• Basic research builds knowledge; applied research solves practical problems. Quantitative designs include experimental (randomized controlled trials: random assignment to treatment and control), quasi-experimental (comparison groups without random assignment), pre-experimental (one group, pretest and posttest) and correlational or survey designs; qualitative designs include ethnography, phenomenology, grounded theory, narrative and case study; mixed methods combine both.
• Sampling: probability samples (simple random, systematic, stratified, cluster) support generalization; nonprobability samples (convenience, purposive, snowball) are common in qualitative work and with hard-to-reach groups.
• Levels of measurement: nominal (categories), ordinal (ranked), interval (equal distances, no true zero) and ratio (true zero); they determine which statistics can be used.
• Reliability is consistency: test-retest, inter-rater, internal consistency (Cronbach’s alpha, commonly 0.70 or higher acceptable) and parallel forms. Validity is accuracy: face, content, criterion (concurrent, predictive) and construct. A measure can be reliable but not valid, but not valid without being reliable.
• Threats to internal validity include history, maturation, testing, instrumentation, statistical regression to the mean, selection and attrition; external validity concerns generalizing to other people and settings.
• Statistics: descriptive (mean, median, mode, standard deviation) and inferential (t-test, chi-square, ANOVA, correlation from minus 1 to plus 1; a p value below 0.05 is the usual threshold); correlation does not prove causation. Qualitative rigor is judged by credibility, transferability, dependability and confirmability, using member checking and triangulation.

In practice: An agency compares clients who chose a new group with those on the waitlist and finds better outcomes in the group. The worker notes that without random assignment, selection bias is a threat: more motivated clients may have chosen the group.

This AKS appears on the Masters, Advanced Generalist and Clinical exams; Clinical items lean toward reading treatment research, Advanced Generalist items toward program research.

4.Quality assurance (e.g., program reviews and audits by external sources)

• Quality assurance checks that services meet set standards; continuous quality improvement goes further, using data to improve processes, often through Plan-Do-Study-Act cycles.
• External reviews include accreditation (for example, CARF, The Joint Commission, the Council on Accreditation, Accreditation Canada), audits by funders and regulators, and licensing inspections.
• Internal tools include record audits, peer review, utilization review, client complaint processes, performance indicators and incident reporting.
• Serious incidents call for a critical incident review or root cause analysis that looks for system causes rather than blame.
• Findings must lead to action plans, staff training and follow-up checks, or quality assurance becomes paperwork.

In practice: An external audit finds that a third of files lack signed consent forms. The program manager sets up a corrective plan with a revised intake checklist, staff training and a monthly file audit until compliance reaches the target.

This AKS appears on the Advanced Generalist and Clinical exams.

5.Methods of establishing program objectives and outcomes

• Goals are broad statements of intended change; objectives are specific, measurable, achievable, relevant and time-limited.
• Process objectives describe activities and outputs (number of sessions held, people served); outcome objectives describe changes in participants (knowledge, attitudes, behavior, condition) in the short, intermediate and long term.
• A logic model or theory of change shows how inputs and activities are expected to lead to outcomes, which makes objectives and indicators clear.
• Each objective needs an indicator, a target, a data source and a timeline, agreed with stakeholders including clients and funders.
• Do not report outputs as outcomes: serving 200 families says nothing about whether those families are better off.

In practice: A family support program replaces its objective, serve 100 families, with: by June, 70% of participating families will report reduced parenting stress on a standardized scale, measured at intake and at six months.

This AKS appears on the Advanced Generalist and Clinical exams.

6.Program evaluation and related impact on services

• Evaluation findings should be used to improve, expand, modify or end programs; utilization-focused evaluation (Patton) designs the evaluation around its intended users and uses from the start.
• Involve stakeholders, including clients and front-line staff, in planning the evaluation and interpreting results, which increases relevance and use.
• Report findings honestly, including negative or mixed results, and share them with staff, clients, boards and funders in usable forms.
• Break down results by group (race, gender, age, language) to see whether a program works equitably, and look for unintended consequences.
• Check implementation fidelity before concluding that a program does not work; poor outcomes may reflect poor delivery.

In practice: An evaluation shows a parenting program works well for English-speaking parents but most Spanish-speaking parents drop out. The agency hires a bilingual facilitator, adapts the materials, and re-evaluates the next cycle.

This AKS appears on the Advanced Generalist and Clinical exams.