Steeping is the use of specialty grains (caramel/crystal, toasted, roasted malts) which are pre-converted and only need to have the sugars solubilized and extracted from the grains. When you tell the program that you are steeping the grains, it uses a lower extraction efficiency number to determine how much sugar is realized from the grains. The default steeping efficiency in BeerSmith is set at 15% (you can find this at 'options' > 'advanced' > 'steeping efficiency').
Mashing involves use of a base malt which supplies enzymes to convert the starches in the grains to sugars. When you indicate that the grains are to be mashed, the program defaults to the mash efficiency which is calculated from your brew house efficiency you entered into your equipment profile. This is normally much higher value than straight steeping will extract.