Not so long ago, Stephen Sondheim was considered a renegade composer on Broadway, his work regularly derided as unmelodic, chilly, and sour. I know a few show music fans, of a certain age and then some, who have never gotten past that verdict—but it is generally considered retrograde today. It would be stretching it to designate Sondheim mainstream, but his work is now embraced by Hollywood (including a successful Sweeney Todd film in 2008); there is now a groaning bookshelf of books about him and his work; and in the eight years I have lived in New York, there have been full-scale productions of eight of his eleven major works.

Finishing the Hat: Collected Lyrics (1954-1981) with Attendant Comments, Principles, Heresies, Grudges, Whines and Anecdotes is the first of two volumes gathering all of Sondheim's lyrics including the "trunk songs," the ones written for but not used in shows as they took shape. The format includes extensive commentary by Sondheim himself, and he turns out to be a marvelous writer—lapidary, witty, and lucidly instructive: he intends the book not as a celebration of his work per se, along the lines of collections like Ira Gershwin's Lyrics on Several Occasions, but as a primer on the craft of lyric-writing.

A perfect lesson, for instance, is his analysis of the initial line in Sweeney Todd: "Attend the tale of Sweeney Todd." The word attend sounds an appropriately archaic note for a tale about Victorian England; the word tale flags that the narrative will be fabulistic rather than literal; the alliteration of the t's feels, in its formality, vaguely sinister, as is this story of a homicidal barber. All of that in six words—this kind of thing is why some of us know almost every lyric the man has ever written by heart.

Yet what stands out most about the book is the heresies promised in the subtitle. Sondheim, so often cast as the underdog, turns out to be rather pitilessly dismissive of the work of his predecessors. He rarely revealed this scorn in his younger days, and even recently he published a list of songs he wishes he had written, suggesting a typical adoration of the types of Songs They Don't Write Anymore (in the vein of "All the Things You Are," for instance). We learn that he was just being polite, and he's chosen to come clean now that most of the usual suspects have passed on. The collection becomes an odd thing, the musings of an Old Turk.

Sondheim places a foundational value on the challenge of creating an artistic statement within the various constraints of the lyric form, including precise rhyme, singability, plausibility to the character, and the economy enforced by the brevity of song. He is especially withering on modern pop artists' lack of interest in close rhyming in favor of "soul," as if the two were incompatible.

What takes some adjustment is when he turns the same gimlet eye upon the Golden Age giants of Broadway. The judgments, to be sure, can be accurate. I, too, have always found Ira Gershwin's work self-consciously spry and often even twee: the fact that "S'Wonderful" was written in 1926 doesn't excuse its cloying quality. I was also relieved to find that I am not the only show music aficionado immune to the charms of Gilbert and Sullivan ("I never fail to be startled by the fervor of their fans," Sondheim muses).

Sondheim also notes that his mentor Oscar Hammerstein, as I have noted in these pages, had a treacly bird fetish and an overall penchant for greeting-card imagery. It is also true that Lorenz Hart's scansion was too often off (in "Too Good For the Average Man," he has "Psychoanalysts are all the rage," the kind of thing undergraduates let pass in songs for campus revues).

Too often, however, Sondheim's criticisms are based on a presentist aesthetic, as if the members of the pantheon had started their careers around 1969. He has a particular aversion to inverted syntax, as in Hammerstein's "'Round in circles I'd go" in "If I Loved You" or "Five or ten / dollars, then / I'd collect from all those yes-men" in Gershwin's "How Long Has This Been Going On?" Sondheim finds such syntax archaic and unnatural—which it is, today. However, old-style poetry was part of the warp and woof of American culture in the first half of the 20th century. People of the era memorized works like Longfellow's The Tale of Hiawatha in school, could hear Edna St. Vincent Millay headlining her own radio show in the 1930s, and read passages of light-verse doggerel in the interstices of their morning newspapers. This antique layer of English syntax was more alive to ordinary Americans than it is even to the highly literate today, and to dismiss its usage in ancient songs requires also condemning Dickens' syntax as ponderous.

Similarly, it is unfair to rate writers of this vintage according to how faithfully they reflected casual speech. Certainly this naturalness is a plus in the work of lyricists Sondheim admires (Frank Loesser, to name one). But early Golden Age lyricists were no more seeking to channel the sound of kitchen table chat than medieval painters were seeking verisimilitude. Fashion in public language before the mid-20th century stressed formality: the casual was allowed as a seasoning—as in Gershwin's line "Nice work if you can get it"—but not as a substrate. This was an era in which even knockabout vaudeville songs were often couched in antimacassar language.

Elsewhere, Sondheim's criteria are simply arbitrary. He takes Hart to task for "Your looks are laughable, unphotographable" in "My Funny Valentine" because technically, unphotographable would mean that someone was unable to have their image reproduced on film. But this willfully literal reading neglects how word meanings evolve, and inevitably leads to inconsistency: does inedible mean "unable to be taken down a throat" rather than "unpleasant to the taste"?

Sondheim lays down the law again when he insists that theater lyrics ought not be as dense with imagery or observation as straight poetry, in order to allow space for the music to make its aesthetic contribution. Interesting idea, and not at all exclusive to Sondheim—but it leads him to the eccentric notion that playwright DuBose Heyward's folkish lyrics for Porgy and Bess are the best ever written for the theater ("Summertime" is a better lyric than any in Chicago, The Most Happy Fella, or Nine?), and to reject the lyrics of poets like Maxwell Anderson and Truman Capote. This latter means that the legions of people who swear by classic songs like "Lost in the Stars" and "A Sleepin' Bee" are missing something.

I don't think so, and I find myself imagining a lyricist of today's generation criticizing Sondheim on the basis of concerns local to the past few decades. For example, in our era that so cherishes releasing the id (often termed "energy," "passion," "honesty," and so on), Sondheim's work is, while hardly chilly, doggedly restrained. His characters, unless they require decisively exotic characterization (Victorian Londoners, Japanese people, the Italian immigrant who tried to assassinate Franklin Roosevelt), tend to sound like college-educated people who read The New Yorker. I rather like that—partly because I am one of them—and simply assume I must go elsewhere for, say, an approach to love beyond Sondheim's ever-diagonal, wary one. Typical are "Love, I Hear," (italics mine), "Marry Me a Little," and "Is This What You Call Love?" (this last in a show titled Passion!).

Someone of a different mindset could read all of this as suspiciously constipated, not letting enough "hang out." To me, that would be as narrow a view of fifty years' work by a great artist as, well, Sondheim's take on Alan Jay Lerner as insufficiently passionate. Similarly contingent would be a potential critique of Sondheim I find similarly inappropriate despite my own skin color—in all of his work, he has written a mere eight bars sung by a black character, and that was in 1964.

It should be said that Sondheim is hard on himself as well, and in ways that touch on a paradox about the man and his work. Namely, the fountain of emotion, ambivalence, wit, and wisdom in his inestimable music springs from a man whose modus operandi is that of a puzzlemaker. A self-professed fan of games and crosswords who does not even much like reading, he often compares his work to the crafting of sonnets, and puts much stock in compact tenets such as Content Dictates Form.

Naturally, then, in the Gershwins' "How Long Has This Been Going On?" he zeroes in on the "inverted syntax" of "Into heaven I'm hurled"—form fails to match content, given that ordinary people do not talk that way—rather than on the intoxication conveyed by the lyric as a whole, in conjunction with George Gershwin's bluesy and slightly loopy melody.

This mechanical focus blinds Sondheim to the more visceral joys in even his own work. He wishes his "I Had a Love" lyric for Maria in West Side Story were in a foreign language, as in an opera, because if Content Dictates Form, then an uneducated girl shouldn't rhapsodize about love so explicitly. But the rest of us read the song as a ravishingly heightened reflection of what Maria is feeling even if she couldn't put it into spoken words. Or, Sondheim thinks the dry wit of his score for A Funny Thing Happened on the Way to the Forum is inappropriate to the lowdown hijinks of the script—once again, the wrong form for the content. Yet the very contrast between the brittle songs and the silly plot is the joy of the piece: this is why it is perpetually performed.

Part of the secret of great art is in how it breaks the rules, and that part is something Sondheim is temperamentally disinclined to revel in. Finishing the Hat helped me understand why his latest work, Road Show, has been mired on the ground for over ten years, never managing an open-ended run in New York. Here is a tale of the Mizner brothers in the Roaring Twenties, one a perpetual high-stakes entrepreneur and the other a sought-after interior designer. To speak to us meaningfully, these red-blooded, glamorous characters need to do more than stand around making observations, having tart little arguments, and avoiding falling too deeply in love. Yet this is all Sondheim, so concerned with precision, brevity, and cards held close, can give them.

Still, when I bought my first ten CDs in 1989, I could not imagine the cast album of Sondheim's Anyone Can Whistle not being among them—and not because it happens to be the one with the black walk-on character. Even within the limitations of his toolkit, Sondheim has created art that helps make life worth living for me and legions of fans. Finishing the Hat is a treasure in itself, but it makes me wish Sondheim had made more allowances for the limitations of the generations before him.

John H. McWhorter teaches at Columbia University and is a contributing editor of The New Republic.

Copyright © 2011 by the author or Christianity Today/Books & Culture magazine.
Click here for reprint information on Books & Culture.