Travelers can only be affected by things from their home world, or other travelers. Thus they will die of old age, pathogens caught at home (or from cohorts), and violence at the hands of things from home (or cohorts).
The invulnerability holds onto to living cells, so hair and whatnot is still vulnerable to damage. Otherwise the non-home/non-traveler protection acts as if the traveler is simply a traveler-shaped bit of nothing; it can’t be pierced or affected in any way. Arrows just stop when they hit the barrier; fire stops, electricity stops, etc.
Muse: If nothing from a non-home world can affect a travelers living cells, then how do they eat/breathe/generate vitamin D?
Well the stone can infect things, but that only works on living things, so it would be a definite gatherer society. Unless they drag the stonebearer around on hunts and have him infect the prey right before it dies. … And isn’t that vaguely morbid. Of course if it’s a bloodborn pathogen you could also just have folks bleed on it, but that strikes me as a rather bad things to build into a culture. Plus, bonus points for the locals not actually knowing how the infection works. >.<;; Of course I can just say 'well, non-harmful things are fine' but that means something somewhere is deciding what is harmful and what isn't. Hmmm... Still, a handwave might be a better idea than trying to force it into a slightly more logical path. But I hate handwaving... *ponders* Of course I can have the stonebearer just touch all of the food first, but that would kill off the idea of multiple groups of travelers (the other groups would starve to death! :P), and I need that plot hook. So, hrm. Of course I can always have the other groups figure out the whole bloodborn thing... *ponders*