I have huge issues with passing the closed guard, and I work on it all the time. I am more comfortable on my back, and it's hard to keep weight on someone without putting myself off-balance. Physiology is against you, because you are locked in your opponent's legs. It helped me to think of passing as 2 parts: breaking the guard, and passing.
Focusing on the former, it's all about posture #1, and pressure #2: in that order. You HAVE to keep posture, whether you are planning to pass by pressuring the inside, or backing out and going around, so that your opponent can't successfully counter-attack. Also, your whole body is stronger when your spine is aligned, regardless of position (thank you, Ryan Hall, for emphasizing that point when I visited). Work on breaking grips around your neck (and collar if you have a gi on), keeping elbows inside the thighs, and staying based. Then you can break the guard open by standing, killing a leg, or whatever. Once you are postured, one of my coaches always screams, "PICK A SIDE!". Pick a leg to kill, or a place to trap the hips, and drive to that side. Force the opponent to make a decision. This is how you stop him from just recovering guard. You aren't trying to pass yet; you are trying to ruin your opponent's position and control. Passing comes easily from there.
Side question: What is your build? I am 6'0", 150 lbs. It's way easier if you are 5'5", 195 lbs. Build counts a lot, and guys shaped like a barrel will always have an easier time sitting in closed guard. Just watch Wilson Reis, Cacareco, or the Ribeiros on top. It's like the guard doesn't exist as an issue to them.