Evidence-Based Adult Education System (E-BAES) Return-on-Investment Pilot
May 31st, 2024 | Blogs
From June 2023 through April 2024, World Education led a community-engaged research initiative on behalf of the Adult Literacy and Learning Impact Network (ALL IN). This opportunity funded short-term research projects to unearth instructional and programmatic innovation, make visible teacher knowledge, and offer insights aligned to the National Action Plan for Adult Literacy’s goals of access, quality, and/or uptake of adult learning opportunities. This blog is one of five reports from the funded research teams of this initiative.
By Margaret Becker Patterson, Jungeun Lee, and Kate Antanovich
The Challenge
Two questions in U.S. adult foundational education (AFE) are seldom asked: do adult learners really get what they come for from AFE programs, and is investing in their learning a solid investment? Rigorous return-on-investment (ROI) studies in AFE rarely occur because research resources are sparse and conducting ROI in the AFE field is complex. The few studies conducted over the past 30 years are limited in scope and utility.
The aim of this ROI pilot research was to inform the design of a future national ROI study. The ROI pilot allowed AFE researchers in the Evidence-Based Adult Education System (E-BAES), a national initiative of the Open Door Collective and Literacy Minnesota, to think differently from standard ROI approaches. This process engaged AFE practitioners and adult learners actively in the research. The knowledge, perspectives, and expertise of AFE practitioners validated and added insights to learner survey responses, considered inputs and costs, and started the process to develop a field-informed design for ROI study.
Instead of starting with the usual ROI relationship between investment (i.e., what AFE spends) and measurable economic outputs (i.e., what society gains economically from AFE), the ROI pilot started with a learner-centered focus – by identifying what goals and outcomes from AFE matter most to adult learners. Adult learners contributed to the pilot via survey (“The Adult Learner Survey” or ALS), completed in November 2022, by providing the reasons they attend AFE, the outcomes they make, and the most important priorities drawing them to AFE.
Then the pilot built on and confirmed ALS findings with descriptions of the realities facing adult learners as provided by AFE practitioners in the pilot’s focus group discussions. Through this activity, AFE practitioners in the pilot triangulated learner results from their perspective. They also added insights into potential cost factors and considered how an ROI study might be designed and developed.
Setting and Methodology
The ROI pilot provided an opportunity to complete analyses of November 2022 ALS survey results, explore the perspectives of AFE practitioners on learners’ goals and outcomes in AFE, and gain AFE practitioner recommendations on further ROI study. E-BAES researchers first employed quantitative, then qualitative research methods addressing 4 research questions (RQ):
- What goals do adult learners report, via the ALS, for participating in AFE programs and what outcomes from learning/ work/ community/ family/ personal domains do they report?
- After reviewing ALS findings, what do AFE practitioners understand as learners’ reasons and outcomes from their own professional experiences with learners?
- What recommendations do adult learners and AFE practitioners have on the highest priority topic(s) for ROI study in AFE and how do practitioners recommend determining potential costs of ROI?
- Based on pilot feedback, what primary areas should be tracked in an ROI study?
E-BAES researchers analyzed ALS results (n=793 adult learners from 43 states) to address RQ1. Goals and outcomes in ALS were measured in five predetermined domains: learning, work, community, family, and personal. To address RQ2/3, E-BAES researchers conducted 7 regional focus groups with AFE practitioners and a state directors’ group in February and March 2024. Focus groups included AFE instructors and administrators (23 seasoned practitioners, with a median 9 years of experience in AFE – ranging from 2-40 years). Starting with recommendations from state directors, the research team recruited a local director and several instructors in that local program for Zoom-hosted focus groups from various regions of the US. Focus group questions delved into AFE practitioners’ perspectives on ALS results and potential priorities for ROI study.
Observations to Date
Adult Learner Survey
Among the five predetermined domains, a learning goal was the most prevalent reason for participating. More than half of adults taking ALS (52%) participated to learn – to strengthen their foundational skills, such as reading, writing, and math. Additional top goals for participation in AFE programs were:
- gaining confidence in what they know (48% – personal domain)
- learning skills to keep the job they have (48% – working domain)
- making life even more satisfying (48% – personal domain)
- helping to support their child(ren) to learn (47% – family domain)
- being able to plan for and go after career goals (45% – career domain).
Across all five domains, these top goals indicate that adults have diverse drivers for entering AFE.
The six top outcomes that learners reported from AFE participation reflected learning, personal transformation, and community involvement:
- learning to do things in everyday life (45% – personal domain)
- speaking/understanding English (43% – learning domain)
- making life even more satisfying (43% – personal domain)
- gaining confidence (42% – personal domain)
- earning an HSD, GED, or HiSET credential (39% – learning domain)
- making a positive difference in the community (39% – community domain).
Focus Groups
Focus group data largely confirmed findings from ALS. AFE practitioners mentioned additional learner goals such as getting an HSE, learning English, improving work skills, and widening career opportunities. At the same time, they stressed learners’ personal transformation and development of a new self-vision. Other benefits AFE practitioners identified as important to learners were feelings of belonging and developing personal networks, which learners use for support with academic or family matters.
Overall, focus group conversations revealed that benefits of participation in AFE are particularly profound for marginalized learners, such as those learners failed by the traditional educational system, immigrants, incarcerated people, and rural residents. Working with these groups, adult educators noted that they can foster a better engaged citizenry that is “more informed and can have conversations and participate in what’s going on around town, can advocate for themselves. So… we’re growing folks who are empowered in their thinking …”
AFE practitioners’ and learners’ recommendations for prioritized ROI topics
Recommendations for topics to be investigated in a larger-scale study covered AFE in general, economic mobility, and family literacy. One practitioner noted a need to “look at the whole variety of what we do for the students” as “probably the biggest investment.” On economic mobility, practitioners emphasized learners having a career, not just “go to McDonald’s and get employed, but to become more career minded and trying to get a job or a career that would support them.” Focus group participants agreed that ROI studies focusing solely on employment metrics are not enough. As one state director confessed: “We look at return on investment, we look at the number of individuals that completed the high school credential and then did that individual… Did they get a job? And try to formulate a number… but we know… that’s a gross underestimate of return on investment.” Focus group participants also commented on the importance of family literacy and its multigenerational benefits, such as breaking the cycle of poverty and low educational achievement patterns.
AFE practitioners’ recommendations on costs and design in ROI study
AFE practitioners next spoke about costs, including cost per learner that programs must pay, intangible costs (versus measurable costs), adult learner income/pay, and costs that adult learners have to pay to be in AFE (transportation, childcare, and not being at work to earn money while in class). Many of these costs are not generally considered in calculating ROI. Focus group participants argued that a traditional ROI approach fails to measure individual returns, such as increased confidence and abilities; longitudinal family outcomes, such as positive influence on the next generation’s educational achievement and career prospects; and societal returns like increased literacy rates, decreased recidivism, and civic involvement.
Focus group participants argued that any study of ROI needs to account for the complexity of AFE and the heterogeneity of learners and their goals. Therefore, future research needs to include a variety of AFE programs (adult basic education (ABE), English learning (ESL), workforce training, etc.). An ROI study must take into consideration the specifics of the population adult educators serve, whether working adults, immigrants, or incarcerated learners.
Implications for Research, Policy, and Practice
From ALS, E-BAES researchers identified four quantifiable priority areas that learners identified as important and should be considered for further study of ROI, including:
- Contributing to community
- Supporting family (child’s learning and family health /safety)
- Gaining learning outcomes (English learning, foundational skills, and HSD/HSE credentials)
- Making progress on career outcomes (planning a career and gaining skills for a new job)
Focus group participants suggested that a future ROI study pay attention to the following outcomes:
- Whether AFE programs hire their own graduates
- Parental involvement in children’s education (via partnerships with a K-12 system)
- Learners’ volunteering rates in the community
- Immigrant integration and their ability to navigate life in the new country
- Learners’ educational trajectories beyond AFE programs (via partnerships with a higher education system)
- Long-term learner career changes
Of course, there are additional considerations for the E-BAES team as they design the ROI study: determining the priority area and subpopulations, measuring long-term impacts of AFE, data sharing with K-12 and higher education systems, and gaining learners’ trust in data collection.
Despite these and other methodological challenges, focus group participants agreed that rigorous study of ROI in AFE is needed, not only to illustrate the impact of AFE on individuals, communities, and society, but also to advocate for more resource allocation to AFE. Such a study, program directors noted, could provide a more nuanced understanding of costs and benefits of AFE compared with the better funded K-12 education system.
