'Team effort' leads to futures fix

Disaster struck futures.wharton.upenn.edu, the main undergraduate e-mail system, of the Wharton School of Business last Thursday when the back-up to the computer's back-up system failed, causing a major disk crash that temporarily denied e-mail access to 2,400 Wharton students. Many Wharton Computing staff members spent all night manually reloading 37,000 user files until the back-up was able to continue the process on its own. Director of Wharton Computing Gerry McCartney said the team took many calls from students asking, "Where is the blinking system?" The problems began on Thursday at 4:45 p.m. when futures.wharton stopped accepting any logins. After the first indication of difficulties, Wharton Computing staffers used repair programs to try to "breath life" into the system, McCartney said. That effort did not work and the group determined that the disk systems were corrupt. McCartney said workers attempted to recover the system from back-up, but the back-up robot had also failed. "The crash was the damnedest bad luck," McCartney said. "The chance of everything not working is pretty extreme." In order to solve the file system problem, the team split into two groups. Some staffers worked continuously to restore copies to the files manually, while others tried to fix the back-up robot. By 10 p.m. Friday night, a majority of the files were reloaded, and the robot was able to continue the reloading process automatically. Early Saturday morning futures.wharton was fully repaired, but the team ran extra diagnostic tests before returning access to undergraduates. "We already felt embarrassed enough about the whole shutdown," said McCartney. "I decided to make sure the system was running well before the opening." Undergraduates were able to use futures.wharton beginning at 1:30 p.m. Saturday. "Wharton staff members worked unusually hard to restore the system as quickly as possible," said Wharton freshman and computing employee Jeremiah Kalan. "Specialists were brought in from other departments, and our staff spent hours on the phone with technical support staff at other companies." McCartney said small problems at the computing center often occur, but this situation was highly unusual. "Our team always practices with simulations," McCartney said. "But this crash was like a fire drill that has a broken fire truck."