To define categories they are statistics. In general leagues are 5x5 (this means 5 hitting categories and 5 pitching categories), but you can choose to have as many categories as you want. In a 5x5 the categories are:
Hitting- R HR RBI SB AVG
Pitching- W SV K ERA WHIP
In Roto you try to have the best numbers in categories for the entire season. If you are first in a category in a 12 team league you get 12 pts, 2nd 11 pts, 3rd 10 pts...12th 1 pt. The team with the most total points for all the categories wins.
In H2H you try to beat your one opponent (one other team) for that week by winning more categories than he does. You face a different team each week and it only matters how you do against that team for that week. In H2H there are playoffs to determine the winner.
In Points you are awarded points for everything your starting players do. The team with the most total points at the end of the year wins. From my experience power hitters and great starting pitching win you points leagues, beacause a solo HR can be worth 6 points (1 pt for a Hit, 3 for the extra bases of the HR, 1 for the RBI and 1 for a Run scored). The amount of points awarded is also changable as you can change the categories in Roto or H2H leagues.
hope this helps