Your question is a very general one, so I'll have to give you a very general answer. If you are getting swept, it is because you have issues with your base. It is either not wide enough, or your center of gravity is too high or way off center.
This may be because the other guy is using great technique to make you lose your base, or it may be that you are just not starting with a good base or are losing your base on your own when you start to attempt to pass.
My suggestion: Next time you roll with this guy, take the entire time just trying not to get swept. Don't tell him that, just do it. The entire roll you can just play sweep defense. If you concentrate on this this, you'll see what it is the this guy does to try to sweep you and maybe you'll figure out what it is that you've been doing to play into his sweep attacks.