“If a judge or commission member knows there are people in the community caring and that show up to the meetings, then they will very likely make totally different decisions.”
Mrs. Camilla Bibbs-Lee was telling us a story to describe the multiple strategies of the Hamilton anti-drug coalition. We were sitting in her office, which doubles as a Weed and Seed “safe place” for kids in the MLK neighborhood of Chattanooga. Bibbs-Lee is the executive of the Community Anti-Drug Coalition Across Hamilton County, CADCA-HC. Markey and I were talking with her and Dr. Vic Bumphus, a criminology professor at nearby University of Tennessee at Chattanooga. We had come to inquire about their participation in a pilot of the revised evaluation WorkStation of the CADCA Institute.
Our conversation touched on a number of issues dealing with the monitoring and evaluation of a coalition, its activities and outcomes. They are quite complex. And reporting out is even more ticklish. What’s best to use: words or numbers? a telescope for the “big picture” or a microscope for the “up close and personal?” a blender to serve purée or a pot to serve stew? Or, in the measurement terms we like to use in social research and evaluation, is it best to use: 1) qualitative or quantitative data; 2) big picture state-and-county-level data or sub-county and community level data; 3) incorporating the multiple strategies into a composite score or keep them separate and score them individually?
Bibbs-Lee was illustrating the concurrent nature of coalition work. She had just told us her approach was having several strategic components work together at the same time. “We look at our prevention interventions from a law enforcement, community policing, and neighborhood restoration perspective,” she had said. She implied that it takes all three together to affect change.
Since the fall, the coalition has joined with the city police, fire department, and county sheriff’s office in conducting compliance checks once a month of places that sell alcohol. Compliance checks are sort of undercover “sting” operations where an under aged youth or one that looks under aged attempts to, say, buy beer at a convenience store. The coalition recruits the “buyer” from among its youth groups. This is their “enforcement” piece. Then the coalition recruits neighborhood people to show up at beer board and court hearings, just to be present and to be prominent in their coalition tee shirts. She calls it “neighborhood watch.” This is their “community policing” piece– residents and police working together. The “neighborhood restoration” piece is a slower process, but it starts with residents “actually stepping up to the plate and taking part in the process.” In the meantime the coalition is working to attract social responsible merchants and businesses to the MLK (Martin Luther King) and East Chat (East Chattanooga) neighborhoods.
A month and a half ago the coalition had invited a dozen people to a beer board hearing in the city. Also showing up, though, were ministers, parents and grandparents from East Chat. “So we had about fifteen to sixteen people sitting out there in audience observing the eight members of the beer board,” Mrs. Bibbs-Lee continued.
“The city beer board has established a procedure they follow: first time violators get a warning or three-day suspension of their license; the second time, five days; the third time, seven days; and so on. At this particular hearing this one lady had sold to the same kid at an Oakie Dokie Market on three successive occasions and she was up before the board on this third occasion. Remember we spaced the compliance checks thirty days apart. Okay the board was ready to say we gave you a three-day suspension of your license, then five, now this time we will give you a seven-day suspension. They allow folks to come up from the audience before they make the final decision. We had one guy who said, ‘No. Enough is enough.’ We had ministers to come up saying, ‘This is destroying our community.’ We had parents come up saying, ‘You know she’s selling to my kids too.’ We even had some folks come up saying, ‘You know she is selling to the same person five to six times a day.’ You know it is illegal to sell to people already intoxicated; that’s against the law. So they sent it around to get a vote on it. The motion to suspend her license for seven days failed to get a majority. A second motion pushed up the suspension to fourteen days. It passed unanimously. That is showing me there are signs it is beginning to catch on with these folks –especially with the beer board folks.
“The store is actually located across the street from a housing development. So one of the community issues we struggle with is we want to shut her down, but it is the only place people shop without catching a bus. So at the same time we are trying to get her shut down, we are trying to get other merchants to come in here and legitimately sell goods and services at reasonable prices.”
No one was cited during the compliance checks conducted the previous weekend.
Bumphus said his big concern was being able to capture such activities of the coalition in a whole way. He is a member of the coalition work group on needs assessment and evaluation. A native of the state, he has a master’s in urban studies and criminology from the University of Nebraska and holds a doctorate in criminology and sociology from Michigan State. He has conducted evaluation and research studies for a dozen years or more and finds coalition work different and intriguing.
He said that he’s done a lot of quantitative stuff, but it doesn’t have to be all quantitative when it comes to coalition evaluation. “Some of this qualitative stuff can lead us to different ways of seeing,” he said. “I’m interested in these two pieces: what the coalition is doing, actions that lead to community change and then what can be substantiated by community-level indicators that change has happened. You don’t see a lot of evaluations that pushes both of these and not in a systematic fashion.”
If there is anything we learn in research and evaluation it is to be systematic.
The evaluation that CADCA-HC is piloting consists of keeping logs in four areas: community change, services provided, resources generated, and media coverage. “Community changes” are those things that change as a result of coalition action –such as policies, practices, and procedures– and are related to a major goal of the coalition. For instance, the first time a compliance check is conducted it is a community change. Thereafter it is a service provided. The coalition forged with city and county to do compliance checks. This resulted in practice and procedure changes. For instance, the city fire department sends inspectors to check for code violations. Their thinking is that where there is one violation such as selling to minors, there will be others such as fire code violations. Another result of this was the reactivation of the county beer board.
“Services provided” are the on-going activities of the coalition. What is so useful is that a community change can also be recognized as a services provided. “Resources generated” are what we normally think of as in-kind donations: for example, the monetary equivalent of one thousand flyers printed and donated by BiLo, or police officers providing fifty hours in volunteering at the youth camp. And “media generated” includes the articles and stories of the coalition that appear in the Internet newspaper The Chattanoogian, and the interviews on Talk Radio, WGOW-FM, and the coverage by WTVC, Channel 9. The media piece does not have to be centered on the coalition, but it has to mention it even peripherally.
Dr. Bumphus said, “These are good qualitative way of looking at what coalitions are doing that you can’t measure quantitatively. I think you have to have these measure of what you are doing as opposed to simply documenting increases and decreases in some indicator. As you know just assessing increases and decreases sometime misses the picture of what is really happening. It is important to know that these relationships have been forged because they will not come out in your traditional quantitative report. Also they often get lost in the narrative.”
The Hamilton County coalition is taking part in Tennessee’s SPF-SIG project and has received funds to carry out several prevention interventions. And until the opportunity came to take part in the CADCA Institute pilot, he said he had been concerned over having goals and objectives for their SPF-SIG interventions without having a direct method by which to prove they had fulfilled them. The directions and data provide by the state SPF-SIG office at best offered only indirect or proxy ways of measuring attainment of the goals and objectives. Coalitions do a lot of things that cannot be captured simply by recording an increase or decrease in percents of results from a countywide survey given every two years.
To be comprehensive as well as to be palatable to most people, your evaluation reports have to include some parts that are quantitative and some parts that are qualitative, according to Bumphus.
Under the direction of Bumphus the coalition is keeping quantitative data on the compliance checks. Names and locations of the businesses where the checks are conducted are logged into an Excel spreadsheet. Also recorded are items such as the dates of the compliance checks, dates of rechecks, whether the businesses were cited for violations, and if cited, the disposition of the beer board. Later he’ll import the spreadsheet into his SPSS stat package for analysis.
Discussion of the quantitative data from the compliance checks led us into the topic of the levels of measures needed to assess the work of the coalition. “You need community-level surveys to track community-level change,” he said. He has conducted community surveys. Some have been the door-to-door interview type where the respondents have been selected from a “frame” with probability sampling; others include handing a questionnaire to every fifth person (or some n) coming though the doors of a community center.
Coalitions working with the SPF-SIG have access to data and indicators for needs assessments and evaluation. However, there are no data or indicators below the level of the county. The problem is that I may have good data and information from valid and reliable measures but it is, at best, only countywide and I as a coalition am working in communities. My efforts may be successful but may not be evident.
(By the way, “SPF-SIG” is the acronym for Strategic Prevention Framework-State Incentive Grant, which is funded by the Substance Abuse and Mental Health Services Administration of the US Department of Health and Human Services and is administered through the Center for Substance Abuse Prevention. Now isn’t it quicker just to say “Spiff sig funded by sam sah and administered thru see sap”?)
Dr. Bumphus said that with community-level data he’s more comfortable in saying whether they accomplished their goals and objectives or not. “I feel less comfortable with the ‘big picture’ surveys. If it says my indicator went down, I couldn’t know what part my organization played a part in that. If it says it went up, I couldn’t know if I am really at fault. So I am left out because of not having the kind of indicators that really focus on the kind of problems we are working on with the four interventions we have.”
It is difficult to obtain data and information for needs assessments and evaluation at levels below national, regional, state, county. So often because of funding requirements and obligations, agencies and service providers use data that, at best, approximates their target areas and populations.
I am reminded of the story about the drunk on his hands and knees going around a street lamp in a nearly deserted parking lot. A guy walks up to him and asks, “What cha doing?” The drunk says, “I’ve lost my keys and I’m looking for them.” The man asks, “Did you lose them here?” The drunk points to a dark portion of the lot where you can see a vague silhouette of a car. “Naw, over there.” “Then why are you looking over here?” the man asks. “Because I can’t see anything over there,” the drunk replies.
Isn’t this the position federal and state grants put organization far too often?
For the Hamilton coalition the use of the special student survey prepared for SPF-SIG work and given at grade levels 6, 8, 10 and 12 in the 30 counties selected to participate in the project is an issue. Because of their large student population, schools were sampled. But, Bibbs-Lee said, none of the schools in the areas on which the coalition has focused its “Communities Mobilizing for Change on Alcohol” intervention were in the sample.
She said that if she could show CMCA works in one sub-area of the county she can transfer it to another area and it will more likely get buy-in and work. “That’s the whole idea of evidence-based practice,” she added.
The multi-dimensional nature of coalitions and their work is interesting isn’t it? And I have some ideas of how to capture this multi-dimensional effort, but that will have to wait until next time.