Pilot Study · Social Validity Research

Nohum Monosov, MS Ed, BCBA, LBA · Real Change ABA, Lakewood, New Jersey

Measuring What Matters at Home

Ask most providers of applied behavior analysis (ABA) how they measure progress and they will show you graphs: how often a behavior happens, how many prompts a child needs, what percentage of trials he gets right. That data has its purpose. But it lives inside the therapy room, and a family’s life happens outside it. A graph can show that a child sat at a table for eleven minutes. A parent can tell you the family finally ate dinner together.

Measuring what the family experiences is the oldest goal the field has. When Baer, Wolf, and Risley (1968) set out the founding dimensions of ABA, the first one they named was that the behavior worked on must matter to the person and to the people around him. Ten years later, Wolf (1978) argued that the judgment of whether a program is worth anything belongs to the people it is meant to help. He called this social validity, and placed it in his subtitle as the field finding its heart.

The field has not been consistent about it since. A review of the Journal of Applied Behavior Analysis covering 1999 to 2016 found that only about 12% of eligible studies reported any social validity measure at all (Ferguson et al., 2019). Most published work, in other words, does not ask.

There is good reason to measure it directly rather than assume that standard clinical measures already capture it. In a study of 154 children who received two years of ABA, progress on individualized, patient-centered goals showed little correlation with gains on a standard adaptive-behavior measure (Choi et al., 2022). The two kinds of measurement answer different questions, and a family’s experience of change is the one standard scores can miss.

In my own conversations with payers during authorization review, questions have increasingly moved toward change families feel at home — whether there is something a child can do now that he could not do before that makes family life better, such as sitting down to a family dinner, behaving in a store, playing a game with siblings, or expressing himself in a way he never had. That is, almost word for word, the question this study set out to answer.

This study was undertaken on the recommendation of Robert J. Bernstein: ask the parents, before and after, whether the work was changing anything in the child’s actual life.

Method

Participants

Nineteen children receiving services at a single ABA practice were rated by a parent at the start of treatment. Three completed no second rating and were excluded, leaving 16 children in the analysis. All were male. Because two of the 16 are siblings rated by the same parent, the 16 cases represent 15 respondents.

Measure

Parents completed a 13-item questionnaire developed by the practice, covering areas of everyday functioning (Table 1). For each area, parents rated how much difficulty their child was having on a four-point scale: not at all, a little, somewhat, or a lot. Parents who fell between two ratings were instructed to select both adjacent options, which were scored at the midpoint. Two open-ended items followed, asking the parent to describe the main challenge they wanted their child to improve in and any progress they had observed.

Every item on the instrument measures a difficulty. No item measures growth, skill acquisition, independence, or self-management. This property proved consequential and is addressed in the Discussion.

Procedure

Follow-up was targeted at approximately six months after the initial rating. Most baselines were collected before treatment began; some were collected after treatment had started, with the parent rating current functioning; one parent was asked to describe functioning prior to treatment. Each child’s first rating was compared with his last under a rule fixed before any case was reviewed. Two documented exceptions were applied: one child’s middle entry was excluded as being of unknown origin, and one child who discontinued and resumed treatment approximately two years later is included for his first course of treatment only.

Results

Change by Domain

The change parents reported concentrated in a specific place. The largest gains were in connection, communication, and self-concept. The areas that moved least were academics, school, and listening to parents (Table 2).

Listening to parents is the closest item on this scale to a measure of compliance, and it is among the areas that did not move. The practice’s approach is designed to support a child in connecting, understanding himself, and expressing what he needs, rather than to drill compliance or tutor academic content. Gains appeared where the approach is aimed and were absent where it is not. This alignment is the clearest signal in the data.

Severely Rated Difficulties

Parents rated 62 areas as causing a lot of difficulty at baseline. By follow-up, 38 of those 62 (61%) had improved. This proportion was stable under sensitivity analysis: removing the single strongest case yielded 58%, and removing the strongest and weakest cases together yielded 60%.

Ten of the 16 children showed lower overall difficulty scores. Three improved across nearly every rated area, and no child declined to a comparable extent (Figure 1). An area rated worse indicates that the parent’s rating moved; it does not necessarily indicate that the child lost ground, as the following section addresses.

Increased Ratings in Areas of Exposure

Several parents rated particular areas as more difficult while describing progress in their open-ended responses. In those cases the pattern was specific rather than diffuse. One child’s increased ratings fell entirely in areas requiring exposure to others — social functioning, school, public settings, and siblings — while his parent described maturity and independence. Another child moved from not at all to somewhat on school difficulty over 17 months while his parent described growing responsibility and self-care.

A child who does not leave the house cannot exhibit difficulty in public. As participation increases, opportunities to struggle increase with it, and an instrument that records only difficulty will register that increase as decline. This is a hypothesis rather than a finding, and it is the kind of question a pilot study exists to surface. An instrument tracking participation alongside difficulty could test it directly.

Parent Narrative Responses

Eight of the 16 parents answered the optional progress item, and all eight described gains. One described a child “more aware of what is socially acceptable, more tuned in to the feelings of others.” One wrote that her son had “matured a lot, not impulsive, knows safety precautions, takes care of himself well.” One described a child “learning to calm himself down when very angry, more helpful in the house, generally more cooperative and less aggressive with parents.” One reported that her son had begun taking showers independently. Another wrote that he “expresses himself better and has better relationships.”

The domains parents named — independence, self-care, responsibility, safety awareness, impulse control, maturity, and social awareness — are almost entirely absent from the questionnaire, which asks only about difficulty. Parents consistently described growth the instrument had no place to record.

Discussion

Parents who live with these children reported change, and reported it most in the areas the practice’s approach is built to reach. Gains landed where the method aims and remained absent where it does not, and they appeared in parents’ own words as growth that no behavioral graph would have recorded.

The most useful result may concern the instrument rather than the children. A measure composed entirely of difficulty items cannot record acquisition, and parents repeatedly supplied in narrative what the ratings could not hold. Future measurement in this practice should pair difficulty ratings with items capturing growth and participation.

Limitations

This is a pilot study and its findings should be read accordingly. Sixteen children at a single practice were rated by their own parents without a comparison group, so the study describes where parents observed change rather than establishing what produced it. Parent report is a meaningful measure of family experience but not a precise instrument, and the brief scale was built for accessibility rather than fine resolution. That coarseness operates in both directions, which is why the findings reported here rest on larger and more consistent patterns rather than on any single rating. The purpose of a pilot is to identify signals worth testing under a stronger design; the domain-specific pattern, with gains concentrated in connection, communication, and self-regard, is the finding most worth confirming in future work employing a comparison group and an instrument built to capture growth alongside difficulty.

References

• Baer, D. M., Wolf, M. M., & Risley, T. R. (1968). Some current dimensions of applied behavior analysis. Journal of Applied Behavior Analysis, 1(1), 91–97. https://doi.org/10.1901/jaba.1968.1-91
• Choi, K. R., Lotfizadah, A. D., Bhakta, B., Pompa-Craven, P., & Coleman, K. J. (2022). Concordance between patient-centered and adaptive behavior outcome measures after applied behavior analysis for autism. BMC Pediatrics, 22, Article 305. https://doi.org/10.1186/s12887-022-03383-2
• Ferguson, J. L., Cihon, J. H., Leaf, J. B., Van Meter, S. M., McEachin, J., & Leaf, R. (2019). Assessment of social validity trends in the Journal of Applied Behavior Analysis. European Journal of Behavior Analysis, 20(1), 146–157. https://doi.org/10.1080/15021149.2018.1534771
• Wolf, M. M. (1978). Social validity: The case for subjective measurement or how applied behavior analysis is finding its heart. Journal of Applied Behavior Analysis, 11(2), 203–214. https://doi.org/10.1901/jaba.1978.11-203

Table 1: Areas of Everyday Functioning Rated by Parents

Area	Area
Getting along at home	Repetitive behaviors
Getting along socially	Tantrums
Handling school	Behavior in public
How he feels about himself	Getting along with siblings
Listening to parents	Making friends
Expressing his needs	Academics
Holding a back-and-forth conversation

Note. Each area was rated on a four-point scale from not at all to a lot. Parents falling between two ratings were instructed to select both adjacent options.

Table 2: Direction of Change by Area, First to Last Rating

Area	Direction of change
Expressing his needs	Improved most
Repetitive behaviors	Improved
Making friends	Improved
Getting along socially	Improved
Tantrums	Improved
How he feels about himself	Improved
Holding a conversation	Improved
Behavior in public	Improved
Getting along at home	Improved
Academics	Little change
Handling school	Little change
Listening to parents	Little change
Getting along with siblings	Little change

Note. N = 16. Areas are ordered by magnitude of mean change in parent-rated difficulty.

Figure 1: Areas Rated Better, Unchanged, and Worse for Each Child

Note. N = 16. Each bar represents the 13 rated areas for one child, comparing the first questionnaire with the last. Children are ordered by number of areas rated better. An area rated worse indicates movement in the parent’s rating and does not necessarily indicate deterioration in the child.