I personally hate points leagues. My favorite (by a fair amount) is roto, followed by H2H and then wayyyy behind that is points.
Roto is the most complicated, however the reason I mainly like it is because in being complicated, it forces you to be good in every catagory and have a well rounded team. It always pisses me off in H2H or points leagues when someone's team has 3 SBs all year and wins a league, or even more so, if they go the other way around and have thousands of steals getting them all their points.
Anyway, after that little rant, here is the difference between the leagues, which I believe was your original question.
In a roto league, you can get a certain amount of points per catagory (12 person league max pts in a cat is 12, 10 person league max is 10, etc.). Anyway, whoever has the most RBI (for example), would get 12 points, second most would get 11 pts, etc. So winning a catagory by 1 is the same as winning it by 100.
H2H is similar to that, except its on a weekly basis. So if you win steals for one week against ur opponant, you get 1 W, same with the other catagories.
2011 Live Free or DyNasty Champ