We'll come down on the side of glass only because that's the way we went early on so thats what we mostly use. Like most we did start with a bucket and still use it. Now that we harvest and repitch yeast through our brew season, I'm starting to like the buckets again. We agree they hold odor and can be scratched etc. We also agree that they are permeable and oxygen can come through - slowly but through. So, when we are starting with a new Activator of Wyeast or sachet of dry yeast, we use the bucket for 7 days as the Primary because the yeast yield is low in the first batch and difficult to harvest the needed 300 to 500 ml for the next batch. We always secondary for a minimum of 14 days in glass - longer if the kegs are full and I have to wash bottles! ;D
Once we have repitched we typically collect 1000 ml or more from the bottom of the carboy. Since we only repitch 500 ml at most, we aren't as concerned about collecting every last ml of yeast that was grown. From then on we use the glass because we have more of them. Sooooo, We really see a downside to plastic if you keep it clean and don't keep the batch in it for extended periods of time. Common sense and care with cleaning go a long way
Preston