There are many ways to formulate recipes. I can only tell you my approach using BeerSmith. First I pick the style I want to design to and read whatever I can, making notes on base malts, flavoring malts, color malts, sugars, hops, yeasts and any flavor notes that I want to include. I then look at other recipes, especially those that score high in competitions or which are highly rated by others, and refine the malt and hop lists. Designing Great Beers, as was suggested is a great resource for this as well. Though I use it mostly for the ingredient descriptions lately.
I used to go out and get a few commercial examples just to taste and determine if I wanted to brew that style and record which flavors I did get from the beers, but that was only the first time I tried an unfamiliar style.
With those notes in hand, I select that style on a blank formulation sheet, update my profiles accordingly, and then add the base malt. I increase the base malt to get the OG in line and then start adding the specialty malts for flavor. I generally have a good idea of the % of each I want to add based upon the style guide and recipes I've researched. As I am doing this, I will adjust the base malt down to keep the OG where I want it.
The I adjust the color with any color malts, add any sugars for flavor or to help contribute to dryness of finish. I'll then select the yeast, which will have an impact on FG, ABV, so I go back to those and readjust, usually just the base malt at this point.
Lastly, I will add the hops into the mix to fit the style and match the bitterness, flavor and aroma profile that I want.
The finishing touches are any additions to water (rarely for me), clarifying agents, adjust the fermentation profile and then a review of everything. I compare the recipe to my notes, make sure the malts are in general balance for the style, and make any last minute adjustments.
I will mention that I have never entered any competitions, but was very pleased when I received a copy of Classic Styles this past Christmas and saw that most of my recipes were reasonably close to those in the book. I attribute this to the recipe review and profile research step.