"Woe and sorrow" probably stem from the punishments Adam and Eve receive. Adam must toil the earth to get his food, and Eve is subjugated by man and must bear the pain of childbirth.
He doesn’t mention, however, the good side: free will, seeing the fruits of one's labor, and the need to make good use of the time one has on Earth while it lasts. Granted, that existence isn’t quite as great as the “happy state” of Paradise.
Here's an interesting question: If Paradise is a "state" (i.e. state of mind?), how can Earth be a "world" of woe and sorrow? Is Paradise/Earth a situation, or a site? When the couple is expelled from the garden, Milton seems to imply that Eden is a part of Earth, though it becomes severely altered after the fall.