Evaluating a Kriah Intervention Starts at the Beginning

Rabbi Yoni Gold

Many schools are identifying kriah as an area for growth. Reading remains a foundational part of chinuch, making finding a solution a major concern. There are many options that schools can implement. A school might approach the problem by fundamentally changing its methodology for teaching Hebrew, introducing a new curriculum with updated materials and teacher training. On the other hand, a school might take a more limited view and only increase the amount of individual reading time for each student. There is a spectrum of tactics that any institution might take.

Regardless of the direction a school chooses to take, there are several major considerations that should not be overlooked. The need for change may feel so urgent that administrators find themselves charging down a path before they look up and reflect on their actions. The need to help students drives schools to move, but if important questions are only asked after the fact, then administrators might have trouble piecing together the answers. To think about these questions, here is a list of some important considerations.

What is the problem?

To make a decision and to make changes, one first needs to clarify the reason for making any change at all. If the problem is related to kriah, then schools need to have more than a vague sense that students aren’t reading well. It might be true that students are struggling, but what is the real problem? Or, better yet, is there really a problem at all?

Teachers are a vital source of information, but relying only on their anecdotal evidence can leave the problem ill-defined. The basics of reading include accuracy and fluency, but teachers are not always using “kriah” to mean the same thing. When middle school rebbeim and moros talk about reading in their class, are they referring only to fluency, or are they also including students’ ability to translate Hebrew? Is there a difference between students’ skills with modern Hebrew and biblical Hebrew? With nekudot and without? These differences can lead teachers to have conflicting reports on student skills. There can be confusion when teachers are sitting in the same room, with some complaining that students have trouble figuring out the pesukim they learn in class, while others say that they seem to be fine reading the Hebrew newspaper articles in their workbooks. Being specific about the problem should help guide the way forward.

It is also important to clarify the extent of the problem, looking at both when the problem begins and how widespread it is. While some students may be leaving elementary school without the requisite skills, unable to identify the correct sounds for each letter and nekudah, it is still possible that they acquired these skills as they should have but have lacked sufficient practice and reinforcement over the following years. Middle school teachers might assume that students never learned what they were supposed to, pointing to a change in the way reading is taught in early elementary school, but the solution might be more middle school Hebrew reading assignments. Related to this, a school should ascertain if these reading problems affect a significant percentage of students or if they are limited to a smaller number. It is possible that the teacher reports can exaggerate the extent of the problem, since the bad tends to stick out in their memory more than what’s good. (It is also important to note that a small but significant percentage of any group of students will struggle with reading, and if it’s these students who the teachers are noticing, that may indicate change is needed in the area of special reading support and intervention, while the general reading instruction program is working well.)

How will this intervention solve the problem?

After deciding what the problem is, a school is ready to look towards change. Administrators need to clarify why any proposed change or intervention will solve the problem that they have uncovered. School leaders can utilize a logic model to identify and outline the necessary steps for implementation, as well as potential pitfalls. A logic model outlines the resources and plans, explicitly labeling the connections between an organization’s activities and its desired outcomes. A map like this should explain why this plan will address the problem being faced by the school. Here are a couple of simplified examples using invented school names:

Example 1

Jewish Academy noticed that students in younger grades are reading at grade level but are falling behind before they reach middle school. They decided to allocate more reading time for students in fourth and fifth grades. To accomplish this, they mapped out the steps and the expected student outcomes. Teachers in fourth and fifth grades will include 20 minutes of reading out loud each week, which will lead to students getting more practice, and students’ fluency will increase.

This plan focuses on the grades that seem to need help and includes the changes that will be made. It includes what the teachers will do, what the students will do, and what change is hoped for in the students. But it also brings to the fore some issues that might reveal themselves later. Where will teachers find the time for these reading assignments? Will teachers need any ideas for what type of reading activities to use?

Example 2

Hebrew Central found that students are leaving second grade with subpar fluency. They recently invested in a new curriculum for first and second grades but have not yet seen the outcome benefits from using it. Administrators discovered that teachers who have been there for many years are still using their own materials and have not been using the new methodology. They spoke with the teachers and found out that they were overwhelmed by the number of new books. The outline of their plan includes the following steps: Teachers will be trained in the new curriculum, and they will receive additional planning time to co-plan how they will utilize the curricular materials, ensuring that teachers will employ the new materials and methods, and students’ fluency will increase.

In this case, identifying the problem led to a different type of solution. The school is not looking to change directions, but to make better use of resources that are currently available. Since the problem involved teachers feeling overwhelmed, dedicated planning time for this initiative is also included. Some of the potential issues that this plan raises are: Which details are most important to include in the training? Where will the additional planning time come from?

When putting a model for change on paper, it can be helpful to look backward and forward. Looking backward at the problem aligns the intervention with one’s needs. If the teachers are using the curriculum and students need more time to practice, then implementing more teacher training misses the point. Looking forward and picturing the outcomes will help schools evaluate whether it is working.

How will we know if it’s working?

Through a change model, schools can identify specific steps that need to be evaluated along the way towards their goal. Evaluating any change initiative can be difficult, but it is even harder when it is not planned in advance. When designing the evaluation, consider the following two key ideas: take measurements at multiple stages of the process and create parallel assessments.

There are many reasons that a plan could fail. Administrators could measure student reading skills at the beginning of the process, implement a new curriculum, and then measure reading skills at the end of the year. If scores have changed, then they can celebrate their success. However, if they have not reached their goal, they are left with a series of questions. Where did the plan fail? Was this the wrong curriculum for this school? Did teachers not use the new strategies in class? Did teachers misunderstand how to implement the curriculum? 

Using a model to map out the steps toward change can help a school identify multiple stages to assess progress. It is not only important to measure outcomes but also to assess fidelity to implementing the plan. If the logic model outlined teacher training, classroom implementation, and student outcomes, all three parts of the plan should be evaluated. Schools should consider what it means to evaluate what teachers have learned at staff trainings and put such evaluations in place. Administrators should measure teacher implementation through lesson plans, classroom observations, teacher self-reports, or check-in meetings. The final measure of success is indeed the students’ reading skills, but strategically incorporating evaluations at multiple steps of the process can identify where progress stalled. 

Another element that can be easily overlooked is the reading assessment tool. There are reading scans and curriculum-based assessments available, which in some cases may be enough. However, if pre-made tests are unavailable because of cost or are not provided along with the curriculum, or schools are looking to add some additional checkpoints for assessing progress, that should be considered as part of the plan. If these schools wait until they are in the middle of implementation, piecing together ad-hoc reading tests, this can lead to questions about how to decide on what passages to use, whether to use nonsense words or only existing words, and how representative or reliable each test is. Many of these decisions can be made in advance, which can also help clarify the expected outcomes of the change model.

Another reason to plan in advance is that the test data will be more informative when it can illustrate change over time, so finding or creating multiple versions of a test should be considered. This can be time-consuming, so it might be helpful to spend time in advance creating those assessments. Schools can draft a pre-test, mid-year test, and end-of-year test, and pilot them with a group of students to see if students are getting similar scores, trying to establish that they are truly parallel tests. Doing this at the start of the process can help avoid the inevitable questions later on about whether the students are changing or the test has changed.

Playing the Long Game

When students are struggling with kriah, schools need to address the problems as quickly as possible. Yet, addressing the issue effectively might require slowing down the instinct to start moving. While administrators should not get stuck in this planning phase – paralysis by analysis – at the same time, laying the groundwork correctly will allow students to thrive. Schools should recognize that playing the long game sometimes requires waiting and planning. If schools clearly articulate their needs, map out the interventions that will address those issues, and identify clear strategies for assessing their outcomes, then they will be setting themselves up, and most importantly, their students, for success.

Yoni Gold is the Director of Data Strategy and Instructional Excellence at the Associated Talmud Torahs of Chicago. In this role, he provides professional development to Chicagoland schools and creates processes for understanding student data and PD programs. You can reach Yoni at ygold@att.org.

• Print
• Email