I always break it. The nutrient pack is one reason I prefer Wyeast; it is "free" and gives an indication to the yeast vitality of that package.
If it is slow to expand, I figure the cell count is lower, so I either make the starter a little bigger, or double-down before brewing.