When the Budget Sinks, Can Student Learning Still Rise? A Fable
The following remarks were made at a meeting of the chief academic officers from the Council of Independent Colleges member schools in Santa Fe, New Mexico on November 11, 2009.
Aesop and his menagerie of foxes, hares, and turtles launched one form of storytelling, the "fable," many years ago. Such stories don't pretend to replicate reality; instead they tease us into asking questions about it and about ourselves. So rather than pretend to be an economist and provide a definitive answer to the question, "When the Budget Sinks, Can Student Learning Still Rise?" I want to tell a fable about imaginary Carlsberg College and its hard working vice president and chief academic officer, Dr. Alexander Grindstone.
Early on Dr. Grindstone, convinced that much needed to be done to improve student engagement and learning at Carlsberg, consulted experts in the field and some very high-priced consultants. Following their advice, he developed a five-point plan for raising the academic quality of Carlsberg. It would, he believed, put some muscle behind the ten cognitive and personal goals articulated in Carlsberg's path-breaking "New Curriculum for the Twenty First Century," which was adopted some years earlier. As financial resources became available, Grindstone proposed these steps:
Develop ten new general education courses focused on global citizenship in the 21st century, and restructure at least two dozen other courses that would increase student interest and engagement and differentiate Carlsberg from its stodgy arch rival down the road, Tuborg College.
Staff these courses with new nationally recognized scholars. This would increase Carlsberg's visibility, and decrease its comparatively high student/faculty ratio. Grindstone noted that this would have a positive effect on Carlsberg's ranking in the US News and World Report survey.
Reduce class size so all classes enrolled just under twenty students. The obvious educational benefits would go hand-in-hand with higher rankings.
Recruit twenty new merit scholars in each entering class. The "peer effect" of these bright and highly motivated students should inspire higher levels of engagement and learning among all Carlsberg students.
Reduce teaching load from six courses a year to five so that faculty would have more time for informal interaction with students.
These five measures, Grindstone felt, would raise Carlsberg's educational excellence and, incidentally, its place in the rankings and push it definitively ahead of its competitor, Tuborg.
The faculty welcomed the plan and President Gladhand agreed to build these steps into the next capital campaign, right up there with the new natatorium, the improved rock climbing wall, and the concierge facilities at each of the new student residential quarters. Things were looking good. In fact, when the same outside experts recommended a similar academic plan to Tuborg College, President Gladhand raised all the numbers in Grindstone's plan. The arms race was on. The tranquil valley where Danish farmers had raised their hops and barley for over a century was about to be the scene of one of our nations' great academic battles.
But then there came the late unpleasantness on Wall Street. The capital campaign was "strategically repositioned," and the Grindstone's plan was "postponed indefinitely."
That was bad enough, but as the budgetary situation grew bleaker, President Gladhand told Grindstone to cut the existing academic program, and to "stop talking about that plan; we just can't afford to improve student learning until the economy recovers."
Poor Grindstone, he had only a few years to go before retirement and desperately, sincerely, wanted to raise the not always awe-inspiring level of student engagement and learning at dear old Carlsberg. He became irritable at home and grouchy to his administrative assistant. He lost his appetite, couldn't sleep, became sarcastic in his comments, and distant in his relations with friends and colleagues.
As his depression got worse, his wife got worried, impatient, and then angry. Finally, she let him have it. "When you took this job, you didn't say your goal was to beat Tuborg in the US News rankings. You weren't losing sleep about the next capital campaign. You didn't talk money and reputation all the time like you do now. You said you wanted to see Carlsberg students develop a love of learning. I believed you. That's what you wanted then. You've lost your moral compass."
Grindstone made no response and went sullenly to his now empty bed.
The next day was cold and rainy. Grindstone couldn't seem to deal with even the most routine administrative chores. Then a strange thing happened. He was pushing aside a report from a national study of student engagement when his eye fell on a surprising sentence. "Measures of student engagement show that the difference between institutions is roughly one tenth of the difference within institutions."
He read it again, "Measures of student engagement show that the difference between institutions is roughly one tenth of the difference within institutions." Was that, he wondered, just a statistical artifact? As a closet statistician, he knew he should expect bigger variations when comparing individuals within an institution than one institution with another. But did that fact mean anything in the real world where institutions fought and boasted about their superiority? Did this mean that here really was not that much difference between Carlsberg and Tuborg or even Pilsner State?
After a minute or two he had an idea. He reached for the phone and called his friend Paul Sotherland at Kalamazoo College, knowing that Kalamazoo had done a true longitudinal study of their students' gains over four years using the Collegiate Learning Assessment. Grindstone wondered whether the Kalamazoo data would confirm or refute the claim that the big differences in student learning came within institutions, not between them. What had they found, he asked, about the top and the bottom ten percent of their students in respect to their gains in critical thinking?
Paul was characteristically forthcoming. After explaining his methodology he said, "When we looked at the gains over four years, the ten percent of our students who showed the largest positive increases gained on average 3.8 times a standard deviation. We think that is pretty darn good. We think the average student makes pretty good gains, as well, about 1.5 standard deviations. The bad news is that bottom ten percent show a decrease of more than half a standard deviation. Those students actually lost ground during their four years. As far as critical thinking was concerned, at least, they exited worse than they entered."
Grindstone was shocked. "You mean they regressed?"
"Don't be surprised," said Paul. "Charlie Blaich has found something even worse in the 49 institutions in the Wabash National Study. He found that over their first six months of college, first year students only improved on 1 of 12 measures. Worst was their academic motivation. There even the top ten percent showed declines."
Grindstone was amazed, and, strangely elated. "Paul," he said, "that may not be such bad news. If we could figure out what accounts for the variation between the top and the bottom deciles, we might be able to make a real difference in student learning."
"You're right," Paul responded, "and we are crunching some numbers right now to see if we can get an answer to that question. All we know so far is that their gains, or losses, don't correlate with entering SATs or high school GPAs."
Grindstone envied Paul. Though he had no formal training in statistics, he valued rigorous evidence. But Carlsberg was not the place to crunch numbers. Grindstone knew that his faculty—especially on the curriculum committee—were dead set against anything that looked like "assessment," or "cross institutional comparisons," or any "breach of confidentiality affecting human subjects," or "any infringement" of the first amendment rights of faculty members to teach in whatever way they choose. The rhetoric did tend to get overheated. Faculty governance was a big deal at Carlsberg and faculty did not want anything that resembled assessment. To be sure, the committee had allowed him to collect National Survey of Student Engagement (NSSE) data "for your eyes only," but refused to let Carlsberg participate in the Collegiate Learning Assessment (CLA), or even develop a program of electronic portfolios. Its chair, Professor Anne Anderson, had expressly warned him against using quantitative evidence in matters affecting teaching or learning. "You can't measure the human soul with numbers, Grindstone," she had told him. "Here at Carlsberg, if we need to know something about our students, we ask them. We talk to our students; we don't line them up against the wall and measure them."
But now a light bulb went off in Grindstone's hitherto gloomy brain. "Suppose," he thought with a flicker of a smile, "I took Anne Anderson at her word. Suppose I asked the ten members of that committee if they would be willing to talk to four seniors about their experience at Carlsberg. And suppose I gave the committee members a list of students to talk to, drawn from the ten percent that showed up as highly engaged on the NSSE survey and the ten percent that were at the bottom end. I wouldn't have to tell the committee which students they were interviewing, but when we get together to compare notes, we might find out why some of our students are so engaged while others couldn't care less about their education. "
Well, to make a long story short, the committee took the bait. Grindstone gave them a set of suggested questions, many drawn directly from the NSSE survey, though he didn't mention this minor plagiarism. He added some questions of his own about whether the seniors felt more or less engaged than when they entered Carlsberg, and he threw in one he had stolen from Dick Light at Harvard, "Suppose you were dean for a day here at Carlsberg. What would be the one thing you would change?"
When the committee convened to compare what they had heard, they remarked with amazement about the wide differences they had discovered among their students. "It's as if they were going to different schools," Indira Rajani, a newly appointed assistant professor of philosophy, remarked. Grindstone smiled. Bill Westergaard, the chair of the English department, went a step further, "It's as if they hadn't gone to college at all, and were forgetting everything they learned in high school." Grindstone gathered up his courage and then described the results of the Wabash Study that showed that at many institutions, students actually regressed in many ways, at least during their first year in college.
The committee was shocked by the Wabash report, but soon found something that shocked them even more—the students who had come in to Carlsberg with the highest ACT scores and the best high school GPAs, and lots of AP courses were among the least academically engaged seniors. Many of them had won coveted full-tuition Carlsberg Merit Scholarships. Asked why they were so disengaged, they said they simply hadn't found the work "all that exciting." The teaching really wasn't very different from what they had experienced in high school. "It's all dull lectures and rote learning for the exams," one senior remarked. "You can come to class poorly prepared; no one calls you to task if you answered questions with a load of bull. Not sufficiently challenging? You could say that. Boring? Yeah, like, sort of, you know, whatever."
"How can our students say that about Carlsberg?" Bill Westergaard fumed. "We set high standards for our students. I tell my students, "Two hours of work outside of class for every hour in class." That gives them a rigorous intellectual experience as well as plenty of time to have fun."
"But," Grindstone blurted out, "our students spend on average fifteen or sixteen hours a week in out of class study time—a lot less than most of us would guess."
"That's a very interesting figure," Anne Anderson, the chair of the committee, noted, suspiciously. "How did you find it out?"
Grindstone decided this was not time to pull out his moral compass. He lied. He said nothing about NSSE data. "Oh," he replied, "just by talking to students. They tell you things like that if you ask them. It's really good to talk to students."
The committee members nodded approvingly, and then went on to make another observation. Most students, both those at the top and those at the bottom, hadn't a clue about the "cumulative educational goals " embodied in the "Ten Learning Objectives for the Twenty First Century" embedded in Carlsberg's path-breaking curriculum that the faculty had adopted after long and bitter debate ten years earlier. These were the guiding principles of a Carlsberg education. But many of the students seemed quite unaware of them. They were equally in the dark about the term "liberal education."
"How can they fail to know something that is central to the whole Carlsberg experience?" Bill Westergaard demanded.
"How would they know?" Anne Anderson wondered. "Do you ever talk to your students about these goals or about your understanding of liberal education? I don't very often." Grindstone smiled to himself; the committee was discovering at Carlsberg exactly what the AAC&U had reported from a national survey a few months earlier. Most colleges had set cumulative cognitive and personal goals for their students, but there was little evidence that students understood what those goals were, or how they related to their course work or major requirements.
It was not surprising then that the Carlsberg students whom the committee talked to identified exactly what Harvard students had as the "one thing they would change." Like Harvard students, they were confused about the rationale and purpose of their courses and requirements. "What do they add up to? What was this education all about? What was a liberal education, anyways? What was it good for? Why would anyone think it mattered?"
It was getting late and the committee was about to adjourn with, as usual, nothing accomplished, when the chair, Anne Anderson, named a sub-committee to come back with some recommendations about how to increase student understanding of the Carlsberg curriculum, and another to see if there was something to be done about the apparent lack of engagement on the part of seniors who had come to the college on Carlsberg Merit Scholarships. She also named a third sub-committee to "look at the evidence" concerning the amount of time students spent preparing for class. Grindstone was enjoined to provide the latter group with whatever information he could find about the subject.
It was a long meeting. As Grindstone left, he felt hungry.
The next committee meeting came before the sub-committees were ready to report. Professor Westergaard had insisted it meet to consider his proposal for revising the curriculum. "After ten years, this curriculum has not done the things Carlsberg most needs—increase retention and graduation rates." He felt it needed to be replaced by one centered in his own disciplinary specialty. That provoked a lengthy and heated discussion, until Indira Rajani, the assistant professor of philosophy, asked, "How do we know that the curriculum significantly affects gradation rates?"
That was the opening Grindstone had been waiting for. He reached into his briefcase and distributed copies of a recently published pamphlet by George Kuh entitled, High Impact Practices explaining that by analyzing NSSE data, Kuh had been able to show that practices such as learning communities, internships, research assistantships, capstone courses, and international study significantly increased student engagement and retention and graduation rates. The effects were particularly strong among African American and other minority students. None of the practices, he went on to note, were tied to the content of the curriculum."
"That's impressive," Anne Anderson conceded. "How many of these practices do we have in place here at Carlsberg?"
"Practically all of them, but only a small number of our students participate in them."
"That's ridiculous," Bill Westergaard interrupted. "How much can it cost to get them up to scale? What's the matter with you, Grindstone? Don't make excuses and sit around until the next capital campaign. Get going on things we can do right now."
"You're right, Bill," Grindstone said placidly. "We will work on that, and on lot of other things we can do right now, without spending a lot of money."
"Like what?"
"Well, for one thing, a Teagle-funded Collegium at Columbia University has been finding a lot of interesting things about memory and content mastery," said Grindstone, feeling more confident. "You can find a list of their findings on the Teagle website. Don't miss the material on metacognition."
"There's evidence from Wabash that shows that things like the prompt return of written work makes a big difference to student learning. It doesn't cost any more to return the papers promptly than to let them pile up until the end of the term. New data shows a lot about student learning."
"Did you say '…data shows?'" Anne Anderson asked.
"Was there an ominous hiss in the end sibilant?" Grindstone wondered. "Sorry," he replied, "I know 'data' is plural. I should have said 'data show.'"
"This is no time for pedantry," Anne replied. "The singular is acceptable English usage, nowadays. We mustn't get bogged down in little details. The evidence you report is really interesting. What else does it tell us about student learning?"
"The Wabash Study I mentioned found that a lot of seemingly small things make big differences."
"So we can do a lot with a little? Sometimes I think that ought to be Carlsberg's motto, per minima ad maxima." (Anne was very proud of her Latin.)
Grindstone replied, "I'd have to go into detail to be really helpful, but the main headings are these:
Good Teaching and High Quality Interactions with Faculty
Academic Challenge and High Expectations
Diversity Experiences
"Nothing about one subject matter being better than another?" Westergaard asked.
"The data doesn't show—sorry, don't show—that subject matter makes a big difference when it comes to long-term cognitive capacities such as the ones in the new Carlsberg Curriculum."
"So it's not what we teach, but how we teach it?" Indira asked.
"Not entirely," Grindstone responded. "In my view, liberal education requires real mastery of content, as well as development of things like critical thinking, analytical reasoning, written and oral expression, and so on. But some practices really help develop mastery of content."
"All right," said Bill Westergaard, "give me an example that could help me in my teaching of history."
"Here's one: a historian you may know at Barnard College named Mark Carnes has developed a project called "Reacting to the Past." It's a whole repertoire of what he calls "historical games" that use role playing to get students into the mind set and the values of past cultures."
"But do they work?"
"Thanks for asking." Grindstone said, trying not to smirk. "The data are coming in and they are very impressive. We know that Reacting to the Past produces rather remarkable transformation in the students who participate."
"So you think that we should be using this historical game stuff?"
"No," said Grindstone, surprising himself by the passion in his voice. "I think we should use evidence, including quantitative data, to figure out what really works for our students here at Carlsberg, right now, and do that."
There was a long silence. Grindstone wondered whether he was about to start a search for a new job. He couldn't survive for long as CAO if the faculty wanted his scalp.
Finally, he broke the silence himself, "In fact, I don't think Carlsberg can flourish or even survive in this harsh financial environment if we don't ask ourselves very rigorously what we want to achieve and whether what we are doing produces the results we want."
There was another long silence. Then the chair drew herself up, looked him straight in the eye, and said, "Go for it Grindstone. We'll back you up."
That's the end of the fable. But I have to tell you one more thing about ancient Greek fables. They don't come with a moral. The little lessons at the end of each Aesop fable were probably added many centuries later. Fables leave it up to the reader or listener to figure out what the story is really saying.
This is such a complicated fable that I can imagine many possible morals. Someone might conclude that quantitative evidence really pays off. Somebody else might reason that clever use of interviews and other "qualitative" techniques can bypass it. Maybe both are essential in the true culture of evidence we need on our campuses right now. Or maybe the moral is that persistent leadership really pays off, that never getting discouraged by foot dragging or budgetary hassles, keeping at it, getting the little things that make a difference right—nose to the grindstone, one might say—is the key.
There are a lot of possibilities. So let me just tell you the end of the story.
Grindstone retired before the capital campaign was revived. Some years later, thinking back on his vice presidency, he surprised himself with the thought that maybe, after all, the economic crisis might have been a good thing for Carlsberg. "Those consultants had it backward. They thought that if we just increased what they called 'inputs,' that is, things like numbers of faculty and scholarship dollars, 'outcomes' would automatically improve. That distracted us from looking for the things that would really work, and if we had implemented that plan, it would have been endless capital campaigns, because each part of the plan drives long-term costs. You don't just add faculty. They need offices, support staff, travel money, and, of course, parking.
"We were forced to make do with what we had, and so we found that if we used the evidence about student learning, good things happened, for students and their education, for faculty in their job satisfaction, and ultimately, even for the budget. Oh, it wasn't dramatic. The word went out that Carlsberg was an exciting place to be. We got a few more applications each year and a higher 'yield.' Just a few more students entered in each class. But those who came tended to stay. The graduation rate went up, from not-quite 60 percent to over 75 percent. Now, Carlsberg is at its optimal size, and the budget, I hear, is solid enough to let my successor do some of the things I wanted to do. We did a lot with a little. We had to. It worked."
Note: Some of the characters in the fable are total fictions. Others, including Charlie Blaich, Dick Light, and Paul Sotherland, are real life friends who have helped me in many ways over the years, and now let me put words in their mouths. One other friend, Jillian Kinzie of Indiana University, is responsible for the light bulb that saved poor Grindstone from his depression—getting faculty colleagues to talk to students in ways that reveal a lot about their experience in college. Many thanks to all these friends, and to Rich Ekman and his colleagues at the Council of Independent Colleges, for providing me a venue to try out this way of exploring issues that seem really important to me.