Shifting will be a lot smoother from Deore to XTR. But unless you're racing for time or are a serious enthusiast with $$ then top component sets are just way more expensive and probably not worth it.
I always wondered how much difference there could be. My HT has a Deore rear der. and Alivio shifters (2005 vintage), when I got it a few years ago it shifted so good I couldn't imagine it getting any better. It still shifts as good, but now I'm riding on a bike with older a Deore LX rear der. and Deore shifters (2002 vintage) and the difference is quite noticeable. I'd like to try out a bike with XT or XTR components to see just how good those are. That said the Deore Alivio combo is more than adequate for the riding I do, it's never caused me any problems.