You detect it with statistical analysis.  If Mallory is excessively unlucky, e.g. she's submitted 3x as many shares as necessary to find a block but none of her shares were blocks, you ban her.