Why are people in Harry Potter so afraid of Voldemort?

Okay I know what your think cuz he's a kick wizard yeah but the thing is he's one guy against the whole wizard world I mean cmon, okay lets say he has his whatchamacallems those dude's that look like clan members to back him up. Okay so now theres 20 of them, thats still 20 Vs like millions of wizards. Can't they like get their like an army and hunt his *** down or something?