"With the Using Data Process, (teachers) were able to take the data and talk about it. It was about 'this is where our students are, so what do we need to do to help them bridge the achievement gap? And what changes do we have to make in instructional practice to get there?"

Gloria Lemerise, Dean of Instruction

Data Tip #1: "Every member of a school community can act as a data leader."

(Love, Nancy et al., The Data Coach’s Guide to Improving Learning for All Students, 2008, p.7) 

Schools are working hard to provide data that works for teachers and students. In fact, your school may have invested in a powerful data warehouse that provides you with access to reports that may include state test scores, benchmark assessment scores, and other assessment data. You may see aggregate and disaggregated scores for your state, district, school, and class, as well as scores for your individual students. You may wonder, “How can I use all these numbers to help me? How can they help my students?”

 Read more...

Using data effectively starts with teachers who understand that the benefits of data are not all on the data dashboard. Access to well-organized data is just the beginning of an ongoing and collaborative process that investigates the current status of student learning and instructional practice. In this process, any member of the school community can act as a leader by celebrating accomplishments, challenging current practices, encouraging learning communities, staying focused on goals, communicating ideas, and actively engaging others in decision making and instructional improvement. So, lead the way—take steps to work together with your colleagues to use your data to find:

  • successes to celebrate;
  • learning problems to address;
  • teaching practices to change.

Action Steps

To get started,

  • Request a meeting with grade-level or subject-area colleagues to discuss the data sets provided by your school or district.
  • Referring to your data, ask yourselves,What am I doing well? How can I amplify what I'm doing well?
    1. Who isn't learning? Are there student groups not being served?
    2. What, specifically, aren’t some students learning?
    3. What in my practice could be causing this?
    4. How can I be sure my assumptions are correct?
    5. What can I do to improve? How do I know that it worked?
    6. What do I do if the students still don't learn?
  • Work in pairs to pinpoint one or two priority learning challenges you feel need to be addressed. 
  • Identify whether specific student groups are struggling with the identified challenges more than others.
  • Present your findings to the larger group to discover similarities and differences. 
  • Together, make inferences about what might be the causes for these learning challenges. 
  • From here, develop a plan for how you can continue to analyze multiple data sources (including test scores, attendance records, student work, and student observation) to confirm or refute your inferences about possible causes. 

Now, you have taken the first steps as a data leader by making meaning of your data and beginning the discussion, “What can we do differently, and how will we know if it works?” Great teaching begins with using data! 

Written by: 
Diana Nunnaley, Using Data Director
Mary Anne Mather, Using Data Facilitator

Data Tip #2: "Making predictions beforeanalyzing new data raises awareness about existing assumptions that can influence accurate interpretation of that data."

(Love, Nancy et al., The Data Coach’s Guide to Improving Learning for All Students, 2008) 

School administrators have made a commitment to data-informed decision making. Often this means that periodically you will see reports that probably include state test results, benchmark assessment scores, and more. Your first impulse might be to scan the results and draw conclusions: who is doing well, who needs help, and what you can do about it. But before your 'take action' impulse kicks in, STOP!

 Read more...

Before you even take a peek at the new data you have in hand, predict what you expect the data to tell you.  This first-step strategy can help guide your analysis of the data and contribute to a bigger pay-off down the road by helping you to more clearly pinpoint student learning problems, their causes, and next steps. 

As educators, we know that making predictions is an effective strategy for teaching new concepts to students. It activates prior knowledge and uncovers understandings and misconceptions—anchoring new learning to familiar concepts. In much the same way, making predictions about student achievement data offers a starting point for navigating new data and engaging in dialogue about what it tells you. In fact, predicting is the first step in a four-phase data-discovery process called Data-Driven Dialogue (Wellman & Lipton, 2004). This structured process enables a Data Team to explore predictions, present a visual representation of the date, make observations, and generate inferences and questions before forming solutions.

If you are ready to make some predictions, here’s how: 

Action Steps

  • Reflect back on the content and skills represented in your new data set.
  • Think about how, when, and for how long that material was taught. Were all students in attendance? Were they engaged with the material? Did they complete assignments? Did you need to provide remedial opportunities?
  • Now, make predictions about what the data is going to tell you. Record each prediction as a list on chart paper or in a journal.
  • Organize your predictions in categories. For example:
    --Overall results you expect to see for all students,
    --Results for specific student groups, such as your third period class or your English language learners,
    --Results for specific standards or skills,
    --Results compared to previous years,
    --Results related to attendance records.
  • Once you have a complete list, review your predictions and look for patterns in your thinking or assumptions about students as individuals or as groups.
  • Stay in tune with your assumptions. When you look at your data, you will gain insights by comparing what you see with what you thought it would report.

Using data in a meaningful way starts with teachers who understand that data are just the beginning. And to predict is the first step on the pathway to making data-informed instructional decisions that can lead to results. The next step is “go visual”—making graphic representations of the data at hand. We’ll talk more about that in our next Using Data Tip from TERC!

Written by: 
Diana Nunnaley, Using Data Director
Mary Anne Mather, Using Data Facilitator

Data Tip #3: "Go Visual"

(Love, Nancy et al., The Data Coach’s Guide to Improving Learning for All Students, 2008, p.7) 

Teachers have access to rich and varied student data, often provided in a variety of computer-generated documents with lots of numbers. Where does a data team begin their dialogue about what the numbers show? How can the team integrate multiple sources of data to tell a coherent story? How can a data team bring to life pages of numbers, so that the data can paint a picture about student learning? One way to illuminate the stories within the data is for data teams to create their own visual display of the data. We call it "Go Visual!"

 Read more...

"Go Visual!" is the second stage in a four-phase process that guides data teams through deep discussion about data and helps them derive meaning from the data. 

 

Data teams work together to create large, visually vibrant displays of data that combine information from multiple sources, make comparisons across student demographic groups, or capture several timeframes. These visuals can illuminate subtle changes in achievement over time. They can pinpoint achievement gaps that may, or may not, reinforce assumptions about who is doing well and why. Most importantly, by creating visual data and then making observations about this data, the team gains ownership of the story the data tells. The shared understanding among the data team that results from Going Visualcan lead to a culture of group responsibility for improvement.

If your team is ready to Go Visual with your data, these steps will get you started: 

Action Steps

  • Identify several data sources that relate to one another (demographics, state test scores in a selected subject area, district benchmark test scores in a particular subject area, etc.).
  • First, make predictions. On chart paper, list your team's ideas about what you think that this data will tell you. (To learn more about predicting click here.) Based on the team's predictions, select data that will help illuminate your assumptions. Include aggregate and disaggregate data from multiple sources and across 2-3 years.
  • Discuss what format or organization will best illustrate the data you have selected. Consider these and other similar questions:
    1. Do you want to compare students that represent various demographics, i.e, special needs, free and reduced lunch, gender groups, attendance groups?
    2. Are you interested in how your students compare with other students in the district or state?
    3. Would it be useful to show multiple years of data on one chart?
    4. What format will best display the data story —bar graph, pie chart, line graph?
  • Together, using large sheets of paper and colorful markers, create a set of posters or graphic illustrations that capture your data.
  • Post the data posters next to your list of predictions and begin the discussion about whether the data confirm your predictions.
  • These visuals can form the beginning of a data wall, which will be a source of ongoing dialogue about using data for meaningful change.

Going Visual is a powerful step in helping a data team make sense of data. Creating visual data as a collaborative team contributes to greater understanding and ownership of the story the data convey. And Going Visual paves the way for deep and rich observations about the data, and then discussions about inferences, causes and effects, and solutions that will greatly impact improvement.

Written by:
Diana Nunnaley, Director
Mary Anne Mather, Facilitator

Data Tip #4: "Make Data Observations: Before Identifying Solutions, Get All the Facts on the Table."

(Love, Nancy et al., The Data Coach's Guide to Improving Learning for All Students, 2008)

Teachers are natural problem solvers. When we see evidence in our data that groups of students are underachieving, we are anxious to find solutions. But data analysis is most effective if a team takes the time to observe and record as many details as possible about what the data reveal. The Using Data process advocates a 'hold your horses' mindset that can help teachers to better pinpoint a student learning problem before jumping to explanations, interpretations, and quick-fix solutions.

 Read more...

Observe is the third stage in a four-phase dialogue process* that guides deep discussion toward uncovering accurate meaning from the data. (See more information about Step 1: Predict and Step 2: Capture Predictions.)

The Observation phase of the four-phase dialogue process requires strong discipline! Assign a group dialogue monitor to avoid moving the discussion too quickly to 'because' and 'we should'. 

Observations might start with phrases such as, “I notice that…, I see that…I’m struck by...I’m surprised that…” 

Sample Observations

What makes a good observation statement? Here are some questions to guide you to make refined and specific data observation statements:

  • Does each statement communicate a single idea about student performance?
  • Is the statement short and clear?
  • Does the statement incorporate numbers (the data)?
  • Does the statement focus just on those direct and observable facts contained in the data, without explanation or inference?
  • Does the statement use relevant data concepts such as mean, median, mode, range, or distribution?

Depending on the type of data you are looking at, the observations might resemble the examples below.

Sample aggregate data observation:
I notice that at the school level, student performance in math increased from Year 1 to Year 2 (44 percent to 47 percent) and then declined in Year 3 (to 33 percent).

Sample disaggregate data observation: 
I see that in the most recent year of data at the school level, 44 percent of sixth-grade African American students performed at the lowest performance level in English language arts, compared with 36 percent of Hispanics and 35 percent of white students.

Sample student work data observation:
I’m surprised that our regular education and special education students had the same difficulty with the vocabulary used in this open response science question.

Follow these action steps to discover all the facts your data can reveal as your data team makes observations. 

Action Steps

  • First, gather your data team members. They might be a grade-level or vertical team, a subject-area department, or your school leadership team. 
  • Together, study a visual representation of the data you want to analyze. Allow some quiet 'think time' to allow members to digest and make sense of what they see. Provide some think-time prompts, such as:
    --What important points seem to pop out?
    --What are some patterns and trends that are emerging?
    --What seems surprising or unexpected?
  • Share observations about the data. Stick to 'just the facts'. A round robin brainstorming strategy works well when making data observations. It encourages all data team members to look closely at the data and have a voice at the table.
  • Capture each observation on chart paper. Continue the process until all possible observations have surfaced and are captured. 

After capturing a complete set of observations, now the team is ready to generate possible explanations for what they observed. Our next Data Tip will discuss Step 4 in the four-phase dialogue process: Making Inferences.

Written by:

Diana Nunnaley, Director
Mary Anne Mather, Facilitator

Data Tip #5: "Make Inferences and Question Your Data's Story"

“Make data observations. Then generate possible explanations that inform next steps to finding the best teaching and learning solutions.”
(Love, Nancy et al., The Data Coach’s Guide to Improving Learning for All Students, 2008.)

Before generating solutions, be certain that you fully understand the problem. As a data team, take the time to verify what learning problems are revealed in your data—and why—before suggesting solutions. After making observations about the data and listing details about what you see in it, draw inferences about why these observations are revealed. Ask yourselves, “Why are we seeing this result?” and/or “What else do we need to know to be sure of this observation?” Making inferences and asking questions before finding solutions is a classic example of the 'go slow to go fast' strategy. It gets you on track for making sure the problem you are solving is one you actually have!

 Read more...

Infer/Question is the fourth stage in a collaborative four-phase dialogue process that guides deep discussion toward deriving accurate meaning from your students’ learning data.

The following action steps will help you and your data team share inferences about the story your data are telling. These inferences will inform important next steps toward pinpointing a valid student learning problem and its true cause. 

Action Steps

  • After capturing observations about your data, make inferences and question your data’s story. Begin to generate possible explanations for what you observe by considering these questions:
    --What inferences and explanations can we draw about our observations?
    --What questions do we need to consider?
    --What tentative conclusions might we draw?
    --What additional data might we explore to verify our explanations? 

Begin your inferences with phrases such as: “I wonder if…, Might this situation exist because…, I would like to know if…We really should explore…, A question I have is…”

Inference statements link back to the observations you made about your data, and might look like the following:

“We really should explore whether district scores improved more than our school scores because some schools are on a year-round schedule.”

“I wonder if mathematical reasoning is not emphasized enough in our curriculum.”

“I’m surprised that our regular education and special education students had the same difficulty with the vocabulary used in this open response science question.”

“Our observations of disaggregate data indicate a high mobility rate. Do we have programs for kids who come to our school in the middle of the year to help them catch up?”

  • Next, work to find the answers to your questions or to confirm your inferences by identifying additional data and indicators you can collect. For example, drill down and look at disaggregate, strand, or item data. Or consider analyzing common grade-level assessments, student work, or even survey data. Does the new data confirm your inferences? Does it change your thinking?
  • Lastly, as your team completes the four-phase dialogue process for analyzing data, consider these three questions to help you define next steps:
    --What are the implications of what we just learned?
    --What actions do we need to take next?
    --Who needs to know?

Now that your data team has clarified inferences about your data, you can focus on using this information to pinpoint very specific student learning problems and generating solutions that can truly impact your students’ achievement. 

Written by:
Diana Nunnaley, Director
Mary Anne Mather, Facilitator

Data Tip #6: When Analyzing Causes, Ask "Why? Why? Why?"

“Learn from yesterday, live for today, hope for tomorrow. The important thing is not to stop questioning.” Albert Einstein       

Once a data team has analyzed several data sources to pinpoint a student learning problem, they often feel ready to leap into action and solve it. But the data team should first engage in a collaborative process of causal analysis to identify the 'root' cause of the problem, to ensure that the solution they propose addresses the true problem and produces the desired results. 

One tool data teams can use to support a root-cause analysis is called “Why? Why? Why?”—a questioning technique used to explore cause-and-effect relationships. “Why? Why? Why?” helps a group look deeply, beyond the symptoms of a problem, to find underlying causes by asking “Why?” at least three times. Each time the question is asked, the team is probing more deeply into the root cause.

 Read more...

For example, suppose your team learns that math scores on the state test noticeably improved for all students except those in the bottom quartile. On the first round of “Why?” team members respond that many of the bottom-quartile students are special education students. Asking “Why?” a second time, they speculate that the new math curriculum, which is closely aligned with the state test, is just too hard for some students. When asking “Why?” a third time, they consider that often the special needs students are pulled out of class for individual instruction and may not be getting access to the new curriculum. This lack of access could be the root cause!

Are you ready to give “Why? Why? Why?” a try?

Action Steps

  1. Define a student learning problem. Be sure to analyze at least three data sources to accurately pinpoint the problem.
  2. Clearly state the student learning problem in writing on chart paper (or use the Why? Why? Why? form
  3. Engage in collaborative dialogue with your data team. Ask “Why” do we have this problem? Then record one response beginning with “Because…”
  4. IMPORTANT: Next, discuss whether your cause needs confirmation. What other data can be consulted?
  5. Continue this process by repeating Steps 3 and 4 three or more times. 
  6. Discuss the data-confirmed causes of your learning problem. Which one seems to be the 'root' cause—the one that, if changed, will yield results? Now your team is ready to start generating solutions. But be careful—the “Why? Why? Why?” process has some limitations.

Limitations
The “Why? Why? Why?” process is not scientific. Different groups might identify different root causes based on only their current knowledge or experiences, which have inherent limitations. That’s why Step 4 is important. Think of “Why? Why? Why?” as a good starting point for launching the dialogue that will move your team toward a better understanding of the problem, before you target a solution.  

Written by:
Diana Nunnaley, Director 
Mary Anne Mather, Facilitator

Data Tip #7: Finding Time For Data Inquiry

“Time for teacher collaboration is not a luxury... It is a necessity for schools that want to improve.” (Love, N., Ed.,Using Data to Improve Learning for All, 2009)

Recently, teams of teachers in a Florida school district learned TERC’s Using Data process of collaborative inquiry. After their professional development sessions, these data teams returned to their schools to apply the process they had learned and dig deeper into their own data with colleagues. As this work progressed, one teacher expressed an epiphany: “I thought we were learning a quick way to 'fix' things. I now realize that this a process that takes time!”

Meaningful data analysis requires that data teams study multiple data sources to pinpoint student learning problems, find root causes for emerging problems, and launch a plan to tackle these problems. Data teams understand that there is not a 'quick fix' approach to understanding and closing learning gaps—this work takes time.

 Read more...

For most schools, finding time to build a culture of data inquiry requires rethinking how time is allocated during the school day and the school year. Some ideas include creative use of specialists, block scheduling, reallocation of teacher contract time, quarterly release-time for data teams, and summer data retreats. Following are some ideas and strategies for maximizing time for data use:

Take Action to Find Time*

Freed-up time. This strategy entails freeing teachers from their regular instructional time to participate in data-focused professional development or data-analysis activities. It is achieved by hiring substitute teachers or by recruiting administrators, parents, or other volunteers to serve as subs. Volunteers can also cover teachers’ recess and lunch duties.

Restructured or rescheduled time. This solution requires a formal alteration of the instructional time—the school day, the school year, or teaching schedules. For example, strategies for creating time include switching to a team teaching approach, a year-round school schedule, or a revised weekly schedule that allows for early student-release days.

Common time. Many schools encourage common teacher-preparation and planning time, rather than individual prep time. This enables teachers to meet as grade-level or subject-area teams. When coupled with a lunch break, for example, common meeting time can result in as much as 90 minutes of uninterrupted time.

Better-used time. Many schools require teachers to meet for regular staff, department, and/or grade-level meetings. By choosing to use electronic formats, schools can communicate about administrative issues more efficiently, saving face-to-face meeting time for teachers to engage in data inquiry. In addition, by reassessing existing professional development plans, leaders may find ways to allocate more time to data analysis and collaborative problem-solving, which can lead to great professional gains overall. 

Purchased time. Some schools and districts are able to reallocate existing funds and occasionally provide stipends for teachers to engage in improvement planning activities outside the school day.

Where there is a will…

Educational leaders have begun to recognize the power of collaborative inquiry around data to improve learning. They understand that changing the school schedule to make time for teacher collaboration is a requirement for collaborative inquiry, and they work hard to find creative solutions to the time crunch. The growing number of schools that now schedule time for teacher collaboration during the school day proves that where there is a will, there is a way!

Written by:
Diana Nunnaley, Director 
Mary Anne Mather, Senior Facilitator

Data Tip #8: "Triangulate, Triangulate, Triangulate"

“When we looked at our state criterion-referenced tests (CRT) for sixth grade, life science was our weakest strand. We couldn’t believe that. We thought we had a pretty strong life science program. It wasn’t until we looked at our own local assessments and saw the same weakness that we became convinced that we had to take a closer look at what we were teaching and how.” (Love, Nancy, Ed., Using Data to Improve Learning for All, p.9)

All too often state test results may be the only source consulted when targeting specific areas for improvement. However, decisions about instructional changes that reflect only this single data source might lead to errors in your decision-making.

If you want your data to lead you toward making meaningful changes, an important principle to follow is triangulation. Triangulation means using three independent data sources to examine apparent issues or problems. You might ask, “Why bother with the extra work of triangulating?” Consider this analogy:

 Read more...

A third-grade teacher asks Mary to look through the front panel of the classroom terrarium and list everything she sees. Mary diligently makes a thorough list and begins to return to her seat when the teacher asks her to take a second look through the side panel of the terrarium. She immediately sees several plants and animals obscured in the front panel view by rocks and shrubs. By using this second “window,” Mary now has a more complete picture. Then the teacher asks Mary to peer through the top of the terrarium to see if there is anything else. Mary is able to add to her list before she sits down. Her three-window analysis reveals a far more comprehensive picture than any one window alone.*

The notion of using multiple windows or perspectives also applies to understanding and applying information from student achievement data. 

Action Steps

  • Since state test data are the most widely publicized and tend to attract the most attention, this is a good starting point. It’s beneficial to thoroughly examine aggregate and disaggregate state data, including digging down into strand and item data, if available.

  • Carefully note specific weak achievement areas. Is this weakness across-the-board or for specific demographic groups? Be sure to note achievement gaps between demographics groups.

  • When district benchmark or performance assessment data become available, similarly analyze this data and compare results to your state data observations. Do these assessments show similar gaps? For the same populations?

  • Now carefully examine student work samples that focus on concepts noted as weak areas in the other tests. What are specific things students can and cannot do/explain? If these samples are common grade-level assessments, they can reveal even more insights. 

  • Based on your comparisons of findings across data sources, you are ready to consider action. You may realize you need to adjust alignment between your curriculum and assessments, or provide re-teaching of some skills, or address needs for professional development. Or you may find that looking at additional data is required. 

Although we have suggested three types of data to consult during the triangulation process, consider all the data sources available to you. End-of-unit tests, informal formative assessments, classroom observations, and teacher and student surveys each offer unique perspectives. A variety of data sources can support or contradict previous data findings and clarify insights about problems and their causes. Use all the rich resources available to you to help understand what changes will offer the most gain.

Triangulation has the following benefits:
• It can compensate for the imperfections of some assessments.   
• When multiple measures yield the same results, it can increase your confidence in the results and assure that you know where to focus reteaching or curriculum adjustment.
• When multiple measures fail to yield the same results, it will raise important follow-up questions.

Written by:
Diana Nunnaley, Director
Mary Anne Mather, Facilitator

Data Tip #9: "Disaggregate Your Data to Make the Invisible Visible"

“Disaggregation is a practical, hands-on process that allows a school’s faculty to answer the two critical questions: ‘Effective at what? Effective for whom?’ It is not a problem-solving (process), but a problem-finding process.” (Lezotte and Jacoby, Sustainable School Reform, 1992)

If you want to tap one of the most powerful uses of data, disaggregate! Disaggregation means looking at how specific subgroups perform. Typically, formal student achievementdata is aggregated, or reported for the population as a whole—the whole state, school, grade level, or class. Disaggregating can bring to light critical problems and issues that might otherwise remain invisible.

For example, one district’s state test data showed that eighth-grade math scores steadily improved over three years. When the data team disaggregated those data, they discovered that the math scores for boys improved, while the scores for girls actually declined. Another school noticed increased enrollment in their after-school science club. However, disaggregated data indicated that minority students, even those in advanced classes, weren’t participating. 

Here are some examples of questions that disaggregated data can help to answer:

 Read more...

• Is there an achievement gap among different demographic groups? Is the gap getting bigger or smaller?
• Are minority or female students enrolling in higher-level mathematics and science courses at the same rate as other students?
• Are poor or minority students over-represented in special education or under-represented in gifted and talented programs?
• Are students at certain grade levels doing better in core subjects?
• Are students whose teachers participate in ongoing professional development
in reading, math, or science doing better in these subjects than students whose teachers do not participate?
• Are the school’s most recent curriculum and instruction adjustments improving the performance of students in the lowest quartile?

To answer these or other questions, carefully consider what disaggregated data is available and what additional data you need. Develop a data collection plan that includes a wide variety of data that can be disaggregated, such as state and local performance assessments, samples of student work, enrollment data for advanced courses, special programs, and professional development, as well as student and teacher survey results. 

Action Steps
Following are tips to help you get started with disaggregating test data:

  • Thoroughly understand your school's demographics in order to select the relevant variables for disaggregation. NOTE: Some schools benefit from disaggregating data within demographic groups, such as Hispanic students born in the continental U.S. compared to those who are foreign-born.

  • Request state and district test data reports that are disaggregated relevant to your student population.

  • Explore technology tools that will help collect, analyze, and report disaggregated data more easily.

  • Note relevant demographic data as you collect other information about student learning.

  • Ask for support from district data experts or the companies that provide your data system. Let them know the types of disaggregated reports that will best serve your needs.

  • Drill down — dive into the data using the four-phase data-driven dialogue process described in TERC's previous tips (see tips #2 through #6). 

As noted by Lawrence Lezotte and Barbara Jacoby in their publication, Sustainable School Reform, “Disaggregation . . . is not a problem-solving but a problem-finding process.” Once you have a clear understanding of who knows what and the learning problems that exist, you can make changes to programs and instruction to target these specific learning gaps.

*Segments excerpted from Love, N. Using Date/Getting Results: A Practical Guide for
School Improvement in Mathematics and Science. (2002). Christopher-Gordon Publishers, Inc., p.39-42. 

Edited by: 
Mary Anne Mather, Facilitator

 

To learn more about TERC's Using Data professional development, please fill contact us.