Philately is infinitely scalable. You can collect stamps without spending any money at all. You can spend millions of dollars on a single stamp. There are stamp collections for any budget. Here are a few sources for stamps.
Your Mail. You may find some initial stamps for your collection in your own mailbox. While stamps are less common on mail than they used to be (and fewer people write letters), you may still find stamps on some of your incoming mail, including stamps intended for special bulk rate or nonprofit mail uses. Older gummed stamps can be removed from their envelope paper by soaking in a bowl of warm water. Self-adhesive stamps, particularly those issued in the last 15 years, are more of a challenge to remove from envelope paper. See here for some tips on removing self-adhesive stamps from their envelopes. Of course, you can collect the stamps still on cover!
Friends and Family. Your friends and family may have stamps put aside, and they receive mail as well. Start writing back and forth and your pen pal correspondence becomes a source for collectible used stamps! Some collectors are able to acquire incoming mail from businesses, but due to privacy concerns over names and addresses this is less common than it used to be.
The post office. You can buy new US issues directly from your local post office if you want to collect mint stamps.
Stamp clubs and other collectors. If you know other collectors, you can trade with them for stamps that you want. Local stamp clubs may be a good source of other collectors who are willing to trade with you. You can find a searchable list of stamp clubs at the American Philatelic Society website, stamps.org.
Stamp dealers and stamp shows. Brick and mortar stamp shops used to be common, but now are few and far between. There are, however, still many stamp dealers who do business at stamp shows, by mail, or on the internet. You can find stamp dealer directories at the American Philatelic Society and the American Stamp Dealers Association.
Auctions. Valuable stamps and collections are frequently sold at public auction. There are a number of specialty firms that focus exclusively on the auction of rare stamps and postal history. Stamp Auction network aggregates the auctions of most of these companies. When considering buying at auction, remember that you are likely to pay a buyer’s premium (generally 15 to 20 percent) over the “hammer” price of the auction. You should carefully read each auction firm’s terms and conditions to ensure you understand the transaction, and expect that an auction firm that you have not previously done business with may require references or a credit check.
Online. Many stamp dealers sell online. In addition, there are several online forums where collectors can sell in addition to professional dealers, including eBay, hipstamp, and the APS StampStore. Be careful to read descriptions and look at images when buying online! While most sellers are honest and reliable there are always some risks to doing business with sellers not known to you.