I worry about it. That's why I soak anything in vodka before it goes in (valilla beans, coffee beans, spices, etc).
For my pumpkin spice, I soaked two teaspoons in 1/2c vodka for two weeks. Then I used an eye-dropper to dose it into the finished beer. I feel pretty safe about that.
Interesting fact. if you soak pumpkin spice in vodka for two weeks....the spice will turn into a gooey/slimey mess. I only ended up with about 1/4 cup of spiced vodka after straining it. The rest was soaked up by the spice. I tried to filter it through some coffee filters, but not much came through. I ended up just pouring the vodka straight off the top.