The downside of knowing there is a God is the fact that I don't think I'd measure up very well on His yard stick of what makes a good and holy person. Besides the point of not having followed any religion and missing all the various trappings that might be necessary like baptisms or bar mitzvah's or whatever, I'm not sure that I'm all that good of a person in general. If God exists and you don't meet the standard that he is looking for, well, then I can't imagine that you have a lot to look forward to as far as an afterlife goes.
Many religious people nowadays believe that there is a loving and understanding God. That point of view has only become prevalent relatively recently as far as history is concerned. For hundreds, if not thousands, of years before this, God was often viewed as being vengeful or at least, sort of demanding and arbitrary. Why in the world would we believe that our view of God is more likely to be true than those of people closer to the start of history? If an event happened today wouldn't you believe that people 2 years in the future would have a better understanding of what occurred than people 400 years in the future? Why are we in a better position to judge if Jesus existed or the nature of God than those who were closer to the original source? That idea does disturb me a bit when I think about it. If God does exist, he might not be there just to pass out candy and puppies to the people of the world.
So I guess in the end it's a balance of sorts. It would be nice to know that there is a purpose to the universe, even if the purpose might not be a pleasant one. After all, what is there to hold on to if everything is simply a random happenstance? What would be the point of it all?