Vanguard, a 35-year-old investment-management firm, has an unusual proposition for the financial-services industry: It's owned by shareholders and all of its funds are operated at cost. As a result, it has a laser-like focus on making sure every part of the business carries its weight, including advertising.

"We have to watch every penny we spend in any part of the business," said Michael Ma, manager-head of retail advertising. "No business unit is given a pass." Applied to advertising, that means everything is measured against its ability to deliver new households. Absent the ability to prove it can deliver, advertising can be a tough sell at an established company that already has nearly $2 trillion in assets.

"The logical line of thinking is , "Why do we need to advertise when we're already enormous?'" Mr. Ma said. "We can prove the value we can add to growth. We can support it with data. In a firm full of highly quantitative, financial executives, they get it."

In 2010, Vanguard made a bold play, launching "Vanguarding," a campaign with a major investment behind it. The push included a range of media, from TV to radio, ballooning the firm's measured-media spending to $61 million from $20 million the year before, according to Kantar and company figures. In 2011, Vanguard spent $54 million on measured media. Still, competitors like Fidelity and Charles Schwab traditionally outspend Vanguard. "Some worked, some didn't," Mr. Ma said of the marketer's campaign executions. "In 2011, we retrenched heavily into digital and killed almost everything else."

"We're 100% transparent with senior management when things don't work—had X loss or X negative impact," he said. "But we explain the thinking and then we kill it and move on to the next set of experiments."

Now, two years in, Vanguard has learned its lessons. The campaign is on track to see a 174% improvement in the number of net new households, while simultaneously decreasing advertising spending per acquisition by 68% since the beginning of 2010.

The company has taken smart, creative bets, finding ways to convey its message while still measuring impact—Vanguard often uses ad-specific web addresses and 1-800 numbers as a way of gauging effectiveness. The At-Cost Cafe, a food truck the company sent out selling coffee for 28 cents, or one-fifth the average price, served as a metaphor for Vanguard's at-cost investing philosophy. The promotion led to a 40% increase in brand recall.

"There isn't a day that goes by that we don't look at the results and the cost of those results," Mr. Ma said. "Sometimes it's creatively and strategically frustrating, but when you get it right, it's personally rewarding and strategically liberating."