How to make FxCop really ignore things you supress

I recently experienced a weird behavior with FxCop. The classical "everything works fine on my machine but not on the build server" situation. In this case I got a few FxCop warnings from the build server but could not replicate when I ran FxCop locally in Visual Studio. Turns out I had to turn on code analysis in the build (Project properties -> code analysis) since the FxCop suppress attributes used to suppress warnings on specific methods are conditional attributes. More on the background here.