Temperature Change Question

I ferment my beers in an old wine cooler that does a pretty good job at keeping the temps fairly constant.  But I guess I was naive to think that the thing was actually insulated well!  I keep it outside and we had a pretty drastic temperature drop this week and it affected the cooler's temp.  I checked it yesterday and it was reading 50F, down from 65F.  I kind of panicked and moved the ale, which is in it's secondary, inside where it's about 69F.  Would I have been better off to leave it in the cooler and let the temp gradually come back up with the outside temp?  And has the changing of temps now affected my beer's taste?