Fudge would only be kicked out in The Half-Blood Prince after he is forced to step down as Minister For Magic.
Yes, Albus Dumbledore was kicked out. But I was looking for two people (one man, one woman) who resigned from the Wizengamot in protest at something the then-minister For Magic did.