One uses irish moss (i.e., kettle finings=moss, whirlflock, SuperMoss, etc.) in the boil kettle in the last 10-15 minutes. It helps coagulate some proteins out of the wort.
So, the irish moss and all the unwanted proteins attached to it fall to the bottom of the boiler during cooling. Hopefully, one is leaving it behind or straining it out on the way to the fermenter.
If any moss made it to the fermenter, I personally have not read that it negatively affects yeast performance. There are different "finings" used to clarify beers in the final days of conditioning or in the keg. I believe it is possible to use too much of those finings and inadvertantly drop some of the proteins that boost body/mouthfeel, and also drop so much yeast that bottle priming might be tough. Those finings include gelatin and PolyClar.