Wharton's creative new method for peer grading

Grading doesn’t usually make people say “whoopee” — not unless you’re Wharton professor Peter Fader.

Or rather, he says, “WHOOPPEE” — Wharton Online Ordinal Peer Performance Evaluation Engine.

Developed by Fader and a team of collaborators, including Wharton doctoral candidate Daniel McCarthy and Wharton Computing Senior IT Director David Comroe, WHOOPPEE compiles peer reviews in order to generate a comprehensive grade for student assignments. It has been under development for several years; the latest iteration of the program was presented in October at Educause, a yearly education conference.

“[The name] clearly gets people’s attention, and it sets people’s expectations that it’s going to be something goofy, but it’s actually something that’s quite serious and quantitative,” Fader said.

Fader was inspired to begin the project by a student’s complaint about his practice of posting the best few papers for every assignment online.

“[The student] said, ‘That’s very depressing — don’t just post the best papers...post some average ones. Or, why don’t you let each of us see some random papers by random students [laughs],’” Fader said. “And then I was on a long flight back from South Africa that summer, and the idea hit me.”

Each student’s rankings, along with rankings generated by the professor and any teaching assistants, is compiled to generate a final ranking across the entire class. WHOOPPEE contains a set of algorithms designed to make the scores as accurate as possible, taking into account factors like the ranker’s own score or any large disparities from the rest of the data. In pilot runs, Fader has seen that the system of online peer rankings improves student investment and drastically drops turnaround time for grading from several weeks to around a day.

Fader said that the professors he had spoken to were overwhelmingly positive about the idea; he himself has pilot tested the program in his course Marketing 476, Applied Probability Models for Marketing. It is also being used for Marketing 211, Consumer Behavior, taught by Wharton professor Jason Riis.

While the program does have the drawback of potentially reducing feedback in terms of comments or criticism, WHOOPPEE’s creators are considering ways of incentivizing feedback by using comments marked as useful as part of a class participation grade.

“I left lots of really mean — ahem, constructive — comments on people’s papers,” Wharton senior Elliot Oblander said. “You get to learn from seeing other people’s work, and you get to be a part of the grading process. There’s a lot more transparency.”

Oblander took Fader’s class in spring of 2015, and was part of WHOOPPEE’s first test run. A year later, they worked on the other side as a teaching assistant for the class, helping tailor and improve algorithms.

Oblander aside, student response to WHOOPPEE’s first iteration was not overwhelmingly enthusiastic. Only about half of the students surveys felt that their work had been accurately assessed, and 46 percent said that they felt the process had been “significantly improved.”

Fader and his team are hoping to make students more comfortable with the idea of algorithmic, peer-generated grading.

“The first time around...[the professor and the teaching assistants] didn’t touch every paper,” Fader said. “So a lot of students didn’t get that...and they felt bad about that.”

Since then, WHOOPPEE’s algorithms have been tweaked and adjusted. The current focus is optimizing the user interface and integrating it into Canvas — another common complaint among students.

While the project is still in progress, Fader is highly optimistic about its potential.

“The hope is to roll it out across Penn, and — I hope — lots of other universities,” Fader said. “And, you know what? Why stop there?”