In giving the nod to Jonathan Littell for his novel, “The Kindly Ones,” the jury cited “one incestuous scene that unfolds on the bed of a guillotine and another that invokes the myth of Cyclops, ‘whose single eye never blinks.’ These marred what the judges called an impressive work.”