Skip to content
11 May 2012

The MNPS Evaluation Intelligencer

Story or data analysis: Which make the most sense to you?

I’ve started a daily (for now, at least) newspaper over at Paper.li. I’ll post on these pages my comments on the lead articles.

Issue #12
The lead article this morning has a political slant, yet it touches on what we experience here in Research, Assessment & Evaluation (REA) at MNPS. How do we get across the story our data is telling us about teaching and learning? Our data coaches are on the front line of this conundrum.

It is hard to convenience teachers as well as some of our administrators the facts and implications research, assessment and evaluation data are “telling us.”

The Forbes article notes:

Society has become more data- and technology-driven, and more complex. Problems are bigger, and their workings more opaque. The world is increasingly filled with black boxes that we, and even the elites who are paid to manage them, half-understand. And there is an inevitable temptation to reject that half (or two-thirds, or 95%) understanding in favor of something you know, and see, and can put your arms around. Meanwhile, what we expect, what we see, and what is really going on continue to diverge. Big problems, be they market bubbles or flawed levees, go ignored until they blow up.

What is the nearby article on the NAEP science results telling you?

Posted 11 May 2012 † Jack § Commentary § Education § Evaluation § Testing § Uncategorized ‡ Comments (0) ° Tagged: ,
06 May 2012

No Place for the Adventurous

This is the title I’ve given to my editorial in today’s edition of the The MNPS Evaluation Intelligencer, my latest fascination with Web 2.0. Something I’ve getting up at 5 am to edit and shape.

I threw this comment out today about “Public Education’s” Central Office. It may not be a place that welcomes fresh, novel savvy evaluation thinking– such as exhibited by the “cutting-edge” workers in the Information-Based Economy mentioned in an article on the opening page of the newspaper.

The lead article on power dynamics offers an explanation:

Forcing narrow understandings of what constitutes proper evidence – and the correct ways to obtain this – closes the doors on relevant options to understand and assess processes of societal change. We need better ways of ‘measuring’ transformation –to assess in ways that are respectful, fair and useful about the changes being experienced and can enable the design of programmes that support positive social change.

But to do so requires drawing attention to the politics of evidence: how power is associated with the kinds of questions being asked and certain ways of knowing that are shaping evaluation approaches; and how organisational requirements and procedures may result in over-bearing, even oppressive management that puts pressure on grantees to use tools and methods inappropriate to what they are trying to achieve and that prevent learning and adaptation.

I would substitute “educational” for “societal” to convey our situation currently in the public education establishment. As a nation we’ve been engaged with educational reform since the 1950s and really obsessed with it since the 1980s. Yet we seem to be barely treading water. I trace this to a loss of creativity in our LEA central offices. Here in may have begun in the 60s with the courts taking over decision making in order to desegregate the schools. And the loss of autonomy for central office is accelerating. Just read The New Institutionalism in Education edited by Heinz-Dieter Meyer and Brian Rowan or the recent paper given by Christopher Thorn and Douglas Harris at the 37th annual conference of the Association for Educational Finance & Policy (AEFP) in March in Boston. I think “The Accidental Revolution” is quite sobering. True. It’s speculation, but it’s informed speculation. They forecast the ever increasing influence of the corporate publishers and foundations in imposing their view of what our children’s education is to be.

And room for alternative thinking? Well, we’ll see.

Posted 06 May 2012 † Jack § Commentary § Education § Evaluation ‡ Comments (1) ° Tagged:
30 April 2012

A look at who’s driving education: A boon or a boggle or a boondoggle?

A recent presentation by two Wisconsin professors involved with assisting local educational agencies (LEAs) in implementing accountability systems based on value-added measurements has given me pause and it might with you as well. The paper, “The accidental revolution: Teacher accountability, value-added, and the restructuring of the American school system,” was given last month in Boston at the 37th annual conference of the Association for Educational Finance & Policy (AEFP). The paper by Christopher Thorn & Douglas Harris is quite sobering. Granted it is speculative, but it is informed speculation.

They state their contention early:

The balance of power in school decision-making has shifted away from teachers, unions, and schools of education– what some call “the establishment”– toward testing companies, data managers, district department heads, school principals, and state and federal policymakers.

These changes have been substantially intentional. The goals of test-base accountability advocates have been to increase accountability for school leaders and individual educators, and to provide analytic tools to help them respond to those pressures and improve learning. Perhaps without even realizing it, however, educator accountability is restructuring roles and relationships throughout school districts and yielding entirely new organizations with which districts interact in complex ways. As we show through a series of case studies, this amounts to a major restructuring of the American school system, and one that has been largely accidental.

They go on to substantiate their claims with examples of Race to the Top (RTTT) and Teacher Incentive Fund (TIF) initiatives and the impending Common Core assessments from the two federal consortia, Partnership for Assessment of Readiness for College and Careers (PARCC) and SMARTER Balanced Assessment Consortium (SBAC).

Here in Metro Nashville-Davidson County we are on the front lines of this restructuring and many of our teachers, principals and central office staff know what it is to be caught up in the whirlwind of complying with federal, state, and foundation grant mandates.

The name of the AEFP session was “Value-Added & the Realities of Data.” If anyone wants a copy, you may download it here. Look for Session 1.11.

Posted 30 April 2012 † Jack § Education § Evaluation § Testing ‡ Comments (0) ° Tagged:
25 February 2012

Do we need a vision?

Just wanted to store this quote from David Snowden. I am sure to refer to it often in my work here at Metro. It is among the Gurteen quote collection.

In the idealistic approach, the leaders of an organization set out an ideal future state that they wish to achieve, identify the gap between the ideal and their perception of the present, and seek to close it.

This is common not only to process-based theory but also to practice that follows the general heading of the “learning organization.”

Naturalistic approaches, by contrast, seek to understand a sufficiency of the present in order to act to stimulate evolution of the system.

Once such stimulation is made, monitoring of emergent patterns becomes a critical activity so that desired patterns can be supported and undesired patterns disrupted.

The organization thus evolves to a future that was unknowable in advance, but is more contextually appropriate when discovered.

This is pictured in Snowden’s story of the birthday party for his son.

Posted 25 February 2012 † Jack § Evaluation § Management ‡ Comments (0) ° Tagged: ,
22 February 2012

Notes from inside the system:

Marking “Dick and Jane” absent

I’ve rejoined the research unit (Research, Assessment and Evaluation, RAE) of Metropolitan Nashville Public Schools. I came in October of last year. It’s not often one has a chance for a “do over” in this business. I realized that I need to stay focused on the task I’ve been hired to do and for which I want to do. The task is to evaluate our schools. Not evaluate in the sense of sorting them into the “good” pile and the “bad” pile, or into the “achieving” schools and the “failing” schools, but an evaluation that attempts to find what’s working for who, when and under what circumstances.

I have a tendency to get off track, though, and I’ve found myself doing so: Getting caught by the ever voracious workings of a central office. I just have to get out into the schools more. So I thought by an occasional post I could keep myself accountable for my mission and accountable to the teachers and the students for getting their story right. My touchstone is this quote by Phillip Schletchty in a now largely forgotten book he wrote in the 1970s: “Inside the structure of the school, students and teachers live most of their school lives in classrooms. What they do there and how they do what they do determine what will be learned in school.”

I was in a work area for literacy coaches at Haywood Elementary two months ago. We were talking about the changes to the school since I had visited a half-dozen years ago when one of the coaches mentioned the school had gotten a request recently for a teacher to volunteer to coordinate a school garden. What seemed to me an innocuous request was viewed by the staff, she said, as if it was the proverbial straw that was liable to break the camel’s back.

There are 200 more student at Haywood than when I was there last. There now are over 850 students enrolled in pre-K through 4th grade with some 60 teachers housed in a sprawling 62,000 square-foot single-story structure and ten outlying portables. Although the portables and wings have been added to the 50-year-old school building to accommodate the increased enrollment, some school facilities haven’t kept up, such as the cafeteria. School lunches begin being served at 10 am, two hours after the school day begins, and ends at 1 pm. How do the earlier eaters get through the rest of the day I wondered. And the time it takes to march kids to and from the cafeteria as well as other centers must take many minutes away from classroom instruction.

And the enrollment make-up of the school is more diverse than ever, but the racial designations doesn’t tell the whole story. There are 11.5 percent Asians (A), 10.4 percent Black (B), 60.9 percent Hispanic (H) and 17.2 percent White (W). The A’s, Asians, are not the Japanese, Chinese, Korean students of the “Tiger Mom” variety, but rather recent refugees from Burma, Nepal. Likewise the B’s are as likely to be from Somalia, the Congo. the Sudan as they are African-Americans; and the W’s are as likely to be Kurds and Ethiopians as they are white Southerners. The ethnic diversity is emblematic of the district at large. There over 80 different languages spoken by children and youths in our schools. When the office at Margaret Allen Middle School routinely sends announcements, notes and letters home they are written in three languages: English, Spanish, and Arabic.

At Haywood, having English-learning students is only one side of the coin. Some of their students are recent refugees and haven’t had the preparation for school we think of as occurring normally in the family. I was taken to a “new comers” class where a handful of kids were being taught their colors, shapes, names of common objects. There are two such classes at Haywood one for grades 1 and 2 aged children and one for grades 3 and 4 aged children. Over 90 percent of the enrollment are on the free-and-reduced-lunch status, and close to three-quarters are LEP, “limited english proficient.”

All Haywood teachers are having to become EL certified. Rather than shaking their heads in dismay, though, they seem generally to relish their work at the school and seeing their charges learn and become competent. They spend extra hours in instruction during the day and after school and on the weekend and working with both pupil and parent. They adapt to the new models of literacy and math instruction, conduct the required DEA tests, running records and Dibbles, and meet with and take direction from the data coaches, the instructional coaches, the literacy coaches. And they had just completed their teacher evaluation and in-class observation in what seemed to be a marathon of lesson plan preparation, minutely and exactly done to a “T,” of pre-observation meetings with the principal and of the post-observations.

So when it came to a request for a voluntary gardner or garden coordinator, or whatever it is, Haywood teachers look at one another quizzically and wonder where and how that time could be worked in their already crowded schedule and how it would assist their students in getting ready for and coping with middle school.

Posted 22 February 2012 † Jack § Education § Evaluation ‡ Comments (0) ° Tagged: ,
26 December 2011

The church and cockroaches in changing times

How about guerilla church?

Are you familiar with K and r strategy? Me neither until last week, then I haven’t been able to keep it out of my analytical mind, applying the concept all over the place.

I was reading someone last week, I forget who exactly who. Probably Phillip Schlechty. I’ve been digging into him a lot these past weeks as I prepare a design to evaluate metro schools. Anyway… I was reading along when mention was made of someone or something being a “K-strategist.” Now I remember. It was Schlechty and he was writing about textbook publishers. Not knowing what a K-strategist was I Googled. Among the first handful of entries was an article by a Bob Campbell in the September 3, 1981 New Scientist.

He explained that terms, K strategist (capital K) and r strategist (lower-case r), have escaped from evolutionary biology are are being applied in social realms such as banking and book publishing. He says,

Briefly, the r strategist produces large numbers of offsprings which tend to be small and live a relatively short time; the parents take little or no care of the young…. The K strategist produces larger offspring which are usually carefully brought up, have high interspecific competitive ability and live a relatively long time. As there is a large investment in each offspring the K strategist produces relatively few young. …In nature a typical r strategist is an insect pest, “small and randy,” like the locust, quick to exploit a new environment; a K strategist might be an elephant closely adapted to a particular environment which it exploits to a maximum.

You can see how this concept can apply to publishers. The textbook and standardize test guys invest a lot of time and effort in producing a text book or a test. They are not about to let their produce whither after one season, so they perpetuate the textbook and test and they become permanent and invasive. Look at our school system, for instance, the state uses Pearson for it statewide tests (TCAP) and many LEAs adopt Pearson textbooks as their curriculum.

But consider. The cockroach, a r strategist par excellence, has survived many environmental changes. Some estimate that they are over 350 million years old. So where’s the dinosaurs?

Anyway. It has had me thinking among other things about the state of the church. I see around me here in Nashville a number of church fragments being formed. Fragments such as home groups, home churches (check out Jerry and Cindy Bryant’s H2H), small-small congregations (check out Midtown Fellowship). Couple this with what we experience around us as the our rapidly changing society and culture, then you too may see that a large monolithic church needing lots of money to sustain buildings, grounds, and staff may not be what is needed at this time.

What may be needed is not necessarily cockroach churches, for we have a caring and loving God. He would not spawn something without nurture and care. But, maybe, more of a guerilla church which relies upon the Holy Spirit for direction more than a bureaucratic hierarchy or an oligarchy of the elite anointed.

For those queasy about the term, “guerilla,” a guerilla church doesn’t use terrorist tactics or advocate violent overthrow or confrontation, but rather is a church using divine power.

— —
Actually I was reading a section in Heinz-Dieter Meyer & Brian Rowan’s The New Institutionalism in Education.

Posted 26 December 2011 † Jack § Evaluation § church § the ‡ Comments (0) ° Tagged:
30 August 2011

Last week’s Gibson raids may impact owners of vintage guitars: From yesterday’s Slashdot

Appearing in yesterday’s Slashdot discussions: ” Environmental Enforcement Agents Targeting Guitars”

etrahedrassface writes “According to the Wall Street Journal, Federal agents again raided guitar maker Gibson this past week, seizing several pallets of wood and computer documents. At the heart of the issue is the wood that is being used in guitars and whether or not it comes from sustainable sources. The company insists it is being harassed and made to ‘cry uncle’ to the government’s enforcement laws. The article notes that exotic fret and tone woods are protected in order to prevent the equivalent of ‘blood diamond like trade,’ but the ramifications now extend to guitar owners. If you play a vintage guitar, or a hand-built guitar made of old stock woods that were legally obtained years ago, you better not fly with it. John Thomas, a law professor at Quinnipiac University and a blues and ragtime guitarist, says, ‘there’s a lot of anxiety, and it’s well justified.’ Once upon a time, he would have taken one of his vintage guitars on his travels. Now, ‘I don’t go out of the country with a wooden guitar.’”

Discussion link

Wall Street Journal article link

Posted 30 August 2011 † Jack § Reportage ‡ Comments (0) ° Tagged:
25 August 2011

Things are not in place to improve our schooling

Things are not in place in Metro Nashville to improve public schooling and do better for our children and youth no matter how many tweaks we make to our calendar, programs, curriculum, instruction, sanctions and incentives. Anywhere from 40 percent to 65 percent of the components of achievement— depending on subject and grade level— occur with parents, families, neighborhoods. (I found a good synthesis of studies on the variance components of achievement tests at New Zealand Ministry of Education site in 2006. Try looking for “Best Evidence synthesis.”)

Schools and school systems are limited as to what they can change. When kids enter school they have already a physical, personality, ethnicity, and gender shape. They bring with them conditioning from their

  • Parents. What are their parenting styles? Do they live with both parents or do they come from a single-parent household? What are about their parents ages? Were they teenagers when they begat children? Young adults, middle-aged, mature or just plain o-l-d? What do the parents invest financially toward their children’s up bringing? What about the time they spend with their children? Are they/have they been divorced? What are their work habits? Their level of education? Cultural identity? Genetic endowments?
  • Family. How many siblings are there? Is it a “blended” family? What is their family structure?
  • Neighborhood. What’s the residential makeup: houses, apartments, condos, housing development? What’s the socio-economic-status of the neighborhood? The social group(s) active and available? What’s the culture there?

So when we ask ourselves what can we learn from the success of others about educational change and improvement we should recognize it may have taken altering conditions other than purely school-related ones.

For instances, take “the old country” from which my Mother emigrated: Finland. Read the article, “Why Are Finland’s Schools Successful?” in the on-line edition of September’s Smithsonian. “Finland has vastly improved in reading, math and science literacy over the past decade in large part because its teachers are trusted to do whatever it takes to turn young lives around.”

Finns began working on public education 40 years. They started with creating a new cadre of teachers .They now have some 62,000 teachers (in 3,500 schools) with master’s degrees, which is required and available only to the top 10 percent of university graduates. The article notes:

…a Finnish child has a good shot at getting the same quality education no matter whether he or she lives in a rural village or a university town. The differences between weakest and strongest students are the smallest in the world, according to the most recent survey by the Organization for Economic Co-operation and Development (OECD). “Equality is the most important word in Finnish education. All political parties on the right and left agree on this,” said Olli Luukkainen, president of Finland’s powerful teachers union.

A principal interviewed for the story is quoted as saying, ““Children from wealthy families with lots of education can be taught by stupid teachers.”

Let me give one more excerpt that goes with my contention that we need changes in what goes on outside of school:

It’s almost unheard of for a child to show up hungry or homeless. Finland provides three years of maternity leave and subsidized day care to parents, and preschool for all 5-year-olds, where the emphasis is on play and socializing. In addition, the state subsidizes parents, paying them around 150 euros per month for every child until he or she turns 17. Ninety-seven percent of 6-year-olds attend public preschool, where children begin some academics. Schools provide food, medical care, counseling and taxi service if needed. Stu­dent health care is free.

Hey. You can read the article yourselves.

The New Republic also had a recent article on Finland, by the way, “The Children Must Play: What the United States Could Learn from Finland about Educational Reform.”

Posted 25 August 2011 † Jack § Commentary § Education § Evaluation § Individuals and Community § Kids ‡ Comments (0) °
10 August 2011

We need relationship with one another

I am beginning to see that we can/I can overcome the malaise that is America at this moment if we/I approach relationships with one another with the same thrill an abandonment these young people have in launching themselves in space: play video

Posted 10 August 2011 † Jack § Commentary ‡ Comments (0) °
24 July 2011

Five basic research questions in education studies

There seems to be five basic questions at the heart of any education research. I use to give these to students in evaluation and measurement courses I taught ages ago. These are not original to me; although I don’t remember where I picked them up. I am indebted to the original author.

Identifying your primary question is initial to any investigation, particularly those involving innovations and their effects upon the behaviors, skills and attitudes of people.

Here are the five basic research questions:

  • How many are there? This is usually called Description.
  • What is the relationship between this and that? This is called Prediction.
  • Why is it so? This is called Explanation.
  • Does it/will it work? This is called Assessment.
  • What does it mean? This is called Understanding or Meaning.

With this frame we see that research can be about:

Description: Descriptive research is “the attempt to discover facts or describe reality.” It asks questions such as how much? how often? and so on. It is intended to describe as fully as possible a particular phenomenon.

Perhaps the most extensive descriptive research is that conducted by the US Bureau of Census. The Census is conducted every 10 years. The goal of the Census is to count every person living in America and provide descriptive data on us as a population. Information is available on not just on how many Americans there are but what do we look like in terms of age, gender, workforce participation, living standards, ethnic background and so on. The amount of data collected is enormous and takes years to collate and then distribute.

It is not just the Census Bureau that conducts descriptive research. Many organizations, both private and government conduct research of this kind. Examples, include the federal, state and local education agencies, for instance, the National Center for Education Statistics (NCES). Various statistical profiles of our schools in Metro can be found on state and on the MNPS sites. The epidemiology and the health care services evaluation divisions have assembled health status information for persons who are residents of Nashville/Davidson County. On their website, you can access health data on population, mortality, natality, and adult and youth behavior risk surveys.

Prediction: The goal of research can be prediction. We may want to know the connection between certain events. This type of research is about “making projections about what may occur in the future or in another setting.”

Predictive research in education is of great significance to certain organizations and individuals. For example, knowing the dropout rate (descriptive research) does not provide us with information of why and in what circumstances certain students drop out of school. If we could identify the relationships between the children who drop out of school and their circumstances, home environments, contexts, individual characteristics, then preventive programs could be developed and target those children and conditions. Still prediction is speculation. No matter how sophisticated and/or complicated the analyses. the results of predictive studies does not give us certainty.

Explanation: Research of this kind does involve attempts to understand “why or how something occurred.” We want to know the cause-effect relationship between things or why things happen. Has X caused Y? This what realist evaluation attempts.

In education, a lot of work is based on assumptions of causality. However, there is very little research that demonstrates a relationship between learning, such as what occurs in schools, and a particular program or factor. By definition, learning is multi-casual and that makes explanation very difficult.

It is very difficult to demonstrate cause-effect, particularly when researching social phenomenon. Perhaps the most famous example of an association but not an absolutely certain connection is the link between smoking and cancer. For causation to exist, we would have to be able to demonstrate that in each and every instance of someone being a smoker, that person will develop cancer. We know from our own experience that it doesn’t happen every time. What we do know from research is that there is a high association between smoking and developing cancer. The link, therefore, is a high probability rather than a certainty because cancer can be triggered by a number of factors other than smoking.

Assessment: This type of research focuses on “methods to plan intervention programs, to monitor the implementation of new programs and the operation of existing ones, and to determine how effectively programs or clinical practices achieve their goals.”

Teachers, students, parents, and funding bodies often want to know that what we are doing is working. We may need to evaluate the success, or otherwise, of a new program or new service. Often, we have to evaluate what we have done before we can get money to continue our work. Using research methods to undertake evaluation is an important skill for workers.

Understanding/meaning: This is research that wants to know how people subjectively understand things. It is an attempt “to gain access to that personal, subjective experience.”

A lot of very important research seeks to know people’s own experience. For example, what is it like to be a mother or a teacher or a student, or all three? What is the experience of being a student in an alternative learning center? How do the families of juvenile offenders cope with what their children are doing?

Each of these questions is more concerned about what it is like for the person involved than counting or predicting or even trying to work out an explanation. Teaching and learning is essentially discourse with language. We educate children in order to change their behavior by changing their inner language. So in our work, it is very important to have people tell us what they think is really going on. This type of research will help us do that.

Posted 24 July 2011 † Jack § Uncategorized ‡ Comments (0) °