The 1932-D is by far the rarest of the Washington quarters. There are only two Washington quarters with mintages of under one million coins. The 1932-D has an original mintage of 436,800 coins and the 1932-S has a mintage of 408,000 coins. Both coins have been highly sought after in all grades for the past 60 years. They are the key dates of the Washington quarter series and actually are two of the better known rare coins of the twentieth century.
The post World War II coin boom (starting in the 1950s) saw a lot of collectors looking for coins in circulation that they needed for their various sets. At the time, it was possible to find most of the copper, nickel, and silver coins of the 20th century in circulation. They may have been low grades, but at the time most collectors just wanted to "fill the hole" and condition was a secondary consideration. The coins that were more difficult to find were the lower mintage coins such as the 1909-S VDB cent, the 1916-D dime, and the 1932-D and 1932-S quarters. They were nearly impossible to find in circulation and were relatively expensive even in low grades.
Nowadays, condition is much more important, but the 1932-D and 1932-S quarters are still very desirable in all grades. I believe they are very similar in rarity in circulated grades. But Uncirculated examples of the 1932-S was apparently saved a little more than the 1932-D in the year of issue as the 1932-D is definitely rarer than the 1932-S in mint state condition. The 1932-D has the lowest MS65 population of any Washington quarter. As I'm writing this there is one lone PCGS MS66, a marvelous coin that sold at auction for $143,000 in 2008.
According to a notice in the June 1934 issue of The Numismatist (p. 416), collectors could still purchase Uncirculated 1932-D Quarter Dollars directly from the U.S. Treasury for "the face value of the coins and an amount sufficient to cover the mail charrges by first-class mail."