FORTUNE:
Secrets of Greatness  
Are today's CEOs batting a thousand?
Earnings can be misleading, and stock prices are too hyperactive. So what's the best way to measure executive performance?
By Justin Fox, Fortune editor-at-large

(Fortune Magazine) -- Just what makes a CEO great? Is it the ability to inspire the troops with bold visions of future growth? (You know, like Jeff Bezos at Amazon back in the day.) Or is it the resolve to cut ruthlessly away at divisions and people that aren't performing? (Like Jack Welch in the 1980s at General Electric.)

Perhaps it's the financial acumen to see which measures of performance truly matter. (Alfred P. Sloan at GM, or Roberto Goizueta at Coca-Cola.) Then again, maybe it's the ability to know when to ignore financial measures to define what's really special about a company. (Steve Jobs in his second coming at Apple.)

terry_semel_yahoo.03.jpg
Yahoo! chief, Terry Semel
ghosn_carlos.03.jpg
Carlos Ghosn, CEO of Nissan/Renault
The Curious Capitalist
Research shows CEOs aren't corporate superheroes, so why do we pay them that way? (Read the blog)
JUDGING CEO PERFORMANCE
1. The stock market
It's great at assessing CEOs - if you can wait a decade or two for short-term noise to wash away.
2. The "balanced scorecard"
Combining financial measures with metrics like employee turnover has been embraced by Best Buy, Ingersoll-Rand, Xerox and others.
3. Turnover
Bad CEOs, by driving out talented staff, can do far more harm to a company than great CEOs can do good.
4. Glowing reviews
Studies show that CEOs who are treated like heroes underperform their unsung peers.
Video More video
Wynton Marsalis, Google's Marissa Mayer and Craigslist's Jim Buckmaster reveal their personal strategies for success.
Play video

Is it the discipline to shun empire building and instead focus on making your core business great? (Katharine Graham at the Washington Post Co. in the 1970s.) Or the vision to move into a new business with greater growth potential? (Katharine Graham again, buying test-prep pioneer Kaplan in 1984.)

No, no, here's what it is: the unique leadership ability to drive a company forward as no other manager could. (Think Stanley Gault of Rubbermaid, which soared under him, then foundered when he retired.) On second thought, maybe it's the ability to pick successors who are better than you. (Reginald Jones, Welch's predecessor at GE.)

Okay, you get it: There is no single recipe for executive greatness. Still, you'd think there would at least be a reliable way to measure whether a CEO is doing an excellent job. There's certainly enough at stake: the economy, our jobs, those mind-boggling CEO paychecks. So how do we tell if a chief executive is any good?

Sports analogies

To help answer that question, let's take a brief detour into baseball, the field of human endeavor in which performance is measured in its most reliable, bounteous, excruciating detail.

The sport's hierarchy of statistics has been upended in recent years by a handful of mavericks and outsiders - personified by a former night watchman at a baked-beans cannery named Bill James and brought to the attention of the non-baseball-fanatic public by Michael Lewis's 2003 bestseller, "Moneyball."

Academics have joined in too: In an article published this summer in the Journal of Economic Perspectives, two scholars concluded that baseball teams had until recently persistently undervalued hitters who walked a lot. Yet another trio of economists has written a book reassessing the use of basketball statistics.

When it comes to CEOs, there's a strong argument to be made that we have similarly been paying attention to the wrong performance indicators - short-term stock price movements, quarterly earnings, press clippings. So can we fix executive evaluation by choosing a few new stats? It may be harder than that.

"I loved Moneyball," says Alfred Rappaport, the former Northwestern University accounting professor who has been among the most influential students of corporate performance over the past half-century. "But I don't think we're talking about quite the same animals here."

One obvious problem is that with corporations only team statistics are readily available, not individual ones. But the most important difference, Rappaport says, is this: "Baseball performance statistics, once they're recorded, don't change. Corporate performance statistics are nothing more than tentative indicators in a game in progress."

Quantifying performance

In other words, you can't really judge a CEO's tenure until years after he has left office. That's fine for business historians but no help at all to board members trying to decide if a CEO deserves to stay, investors judging a company's prospects or executives themselves. Which is why right now, in business-school ivory towers and corporate boardrooms, there's a lot of talk - and even some action - about nuanced new ways to assess company leaders in real time.

It was in the 1950s that the idea of treating management as something that could be measured and quantified first came into vogue. Key works of the new management literature included Chester Barnard's "Functions of the Executive" (which was first published in 1938 but took a while to catch on), Peter Drucker's "The Practice of Management" and Sloan's "My Years With General Motors."

Barnard, a former president of New Jersey Bell, emphasized the role of an executive as communicator, evaluator of talent, and motivator. Drucker talked of the need to set clear objectives but give workers freedom to figure out how to realize those objectives. Sloan's emphasis was on how to achieve this balance by rigorous use of financial controls.

That early literature focused, as Drucker put it later, on "managing the existing rather than innovating the new and different." The direction of America's big corporations had already been set. Those that survived the Depression were usually characterized by strong internal cultures and financial discipline.

When the economy took off in the postwar years, they almost automatically churned out huge profits. The CEO was usually a lifelong company man charged with keeping the ship headed in the same direction. As Time said of its 1955 Man of the Year, GM boss Harlow Curtice, he was "as surely a product of General Motors as Dwight Eisenhower is of the U.S. Army."

Similar things could be said of other big-company CEOs in those days. Earnings mattered, but so did fitting into the corporate culture. The stock market did occasionally intrude in the form of the stock options awarded to top executives - but the grants were modest, and executive pay in general was kept in check by top tax rates upwards of 90 percent.

Until the 1970s, this way of doing things worked reasonably well. Then energy prices skyrocketed, and with them the price of everything else. Fierce competitors emerged in Germany and Japan. The company men who had ascended to top jobs at places like GM and Bethlehem Steel were mostly at a loss as to how to adapt. Their ingrown corporate cultures weren't much help. Nor were reported earnings a guide, as inflation artificially distorted them upward.

Upping the game

The stock market provided an answer. Wall Street quickly noticed that something was amiss in corporate America - the five years starting in 1972 were even worse for U.S. stocks, on an inflation-adjusted basis, than the five starting in 1929.

During those same years, the efficient-market hypothesis, which held that stock prices accurately reflected all available information, was taking the nation's business schools by storm. The lesson was clear: CEOs needed to be paying closer attention to the stock market, which provided a means for judging performance that was both straightforward and flexible.

Giving teeth to this academic talk was the hostile-takeover boom. The 1974 bid for battery maker ESB by the International Nickel Co. kicked things off. By the early 1980s, junk-bond-fueled upstarts were assaulting some of the nation's largest corporations. CEOs learned that if their stock lagged, somebody might buy their companies and throw them out.

The result was a new standard of corporate and CEO performance, what Rappaport in a 1981 Harvard Business Review article dubbed "shareholder value." The job of the CEO was to make shareholders money. For Rappaport this was a long-run goal, best accomplished by a careful focus on metrics like free cash flow and the cost of capital. Some big-company CEOs, like Welch at GE (Charts) and Reuben Mark at Colgate-Palmolive (Charts), took this talk to heart and over decades-long tenures helped create lasting value for shareholders.

But the long bull market that began in 1981 made it seem much easier than that. Stock gains came quickly and spectacularly, and the CEOs on whose watch they occurred were celebrated as superheroes, capable of transforming a corporation with a few bold strategic moves. Their stock- option-rich pay packages reflected this new status, as did the hiring process for new CEOs, which increasingly favored well-known outsiders.

In the process, the quest for shareholder value became focused on - and eventually distorted by - short-term measures. The simplest way to keep your stock rising was to meet or exceed the "consensus" quarterly earnings estimate set by Wall Street analysts. Those analysts made their forecasts after consulting with company executives, so a missed target was a possible indication that a CEO wasn't on top of his operations.

But the quarterly earnings game took up huge amounts of executive time, focused intense attention on a metric that wasn't a reliable predictor of long-run return to shareholders, and led some companies to bend or break the rules to make sure they didn't disappoint Wall Street. An awful lot of shareholder value was destroyed in the name of creating shareholder value.

A balanced scorecard

Since the market crash of 2001-02, corporate America has been groping for a new standard for CEO performance. Finance scholars now acknowledge that short-run stock movements can be extremely noisy - that is, they reveal no useful information whatsoever. The stock market remains the ultimate arbiter of long-run success, but its day-to-day movements are looked upon with suspicion.

The strengths of the 1950s "organization man" approach are meanwhile being rediscovered. In the most important management book of the new millennium, "Good to Great," independent researcher Jim Collins - the closest thing to a Bill James in this tale - identified 11 CEOs who steered their corporations to sustained leaps in stock performance and found that they were all self-effacing insiders who put their companies ahead of themselves and focused most of their early efforts on surrounding themselves with good people.

Boards and CEOs are beginning to catch on. A new approach is emerging that seeks to balance the pressures of the stock market - which still figures heavily in CEO compensation - with the recognition that long-term success is often built upon less tangible factors like employee motivation and customer satisfaction.

Mercer Delta Consulting has been doing a booming business lately, for example, helping boards evaluate CEOs on such intangibles as how they relate to the executives who report to them. "It's changed quite a bit post-Sarbanes-Oxley," says Mercer Delta chairman and founder David A. Nadler. "Boards are taking this more seriously."

Then there's the "balanced scorecard," a template for supplementing financial measures with other metrics that was dreamed up in the early 1990s by Harvard Business School professor Robert Kaplan and management consultant David Norton and is now very much in vogue.

At retailer Best Buy (Charts), CEO Brad Anderson's bonus is determined in part by customer-loyalty scores, employee-turnover improvement and something called "customer centricity store revenue." It is also becoming customary for new CEOs to take several months to articulate a strategy and set of goals, then present a plan to the board complete with the metrics by which progress toward those should be measured.

Executive search stars Jim Citrin and Tom Neff of Spencer Stuart push the CEOs they place to identify three or four key "themes" that they expect to define their tenure. Those themes are then used, Citrin says, to help determine "how you drive compensation systems, culturally who you celebrate, and what you measure."

Tallying the final score

So far, investors appear to be dubious of these developments. A recent survey by the compensation consulting firm Watson Wyatt found that institutional investors still favor total return to shareholders as the main pay-for-performance metric for CEOs (return on equity came in second). At Xerox (Charts), dissident shareholders put a proposal on this year's proxy ballot demanding an end to the company's "balanced scorecard" approach to determining bonuses - because CEO Anne Mulcahy got a bonus last year even though the stock price dropped.

At GE, Jeff Immelt's green-oriented Ecomagination initiative (or "theme," if you will) has won raves in the press and from some employees, but the stock price is still below where it was when he took over in 2000.

The really big question may be how much CEOs even matter. A 1972 study that found top executives' impact to be negligible set off an academic debate that has raged ever since. The CEOs-do-matter side has largely gained the upper hand - leaning in part on studies of the impact of basketball coaches and baseball managers on team performance - but that's not because most academics think CEOs are superheroes.

A recent study of "superstar CEOs" by Ulrike Malmendier of the University of California at Berkeley and Geoffrey Tate of the University of Pennsylvania's Wharton School found that companies run by top executives who won awards handed out by the business press between 1975 and 2002 consistently underperformed the market after being honored. (Malmendier and Tate argue, in part, that CEOs who have been anointed as superstars neglect their jobs.) An earlier examination of auto industry productivity found big impacts from changing CEOs - except at the best-run car company, Toyota (Charts).

"Good leaders can make a small positive difference; bad leaders can make a huge negative difference - because they drive people out," says Stanford Business School professor and organizational expert Jeffrey Pfeffer. "If you said to me, 'Who can fix GM?' I don't know. But there's no question that someone could make an enormous difference on the downside."

The ultimate lesson here may be that there are limited rewards to identifying the very best CEOs. Same with paying CEOs like superstars - Collins found no clear link between executive pay and success at the "good to great" companies he studied.

Enough of the superhero ethos remains that we still tell tales of management heroics: Carlos Ghosn parachuting into Nissan and turning it around. Terry Semel bringing his Hollywood savvy to sputtering Yahoo. But could Ghosn really have saved GM, as he was hoping to attempt? Did Semel really earn the $439 million he made over the past three years, especially now that Yahoo is sputtering again? Who knows?

Our obsession with finding greatness in CEOs is a hangover of sorts from the 1990s boom. It's not a bad thing in itself, but it distracts boards and investors from a more important task: identifying CEOs who just can't hack it.

Suggestions for further reading

_____________________

Four key skills to master now

10 Highest-paid executives Top of page

Want to learn more Secrets of Greatness? Get the new book