Unofficial Fallout 4 Patch

This issue is caused by getting poisoned, then either using Med-X or equipping gear that gives you high poison resist. When the poison wears off, the calculations to remove it are not being done correctly and you're left with a permanent penalty.

There's also some indication this same issue can be caused by getting a head injury and using a stimpak that doesn't correct the entire perception drop, but this seems to be more erratic.

Unfortunately the only means to fix this seems to be to do the opposite of what caused it. Use poison resist gear, get into a fight with something that does poison damage (like a Stingwing or a Radscorpion), wait for it to poison you, and then quickly remove the poison resist gear before the 4 second duration of the poison expires. This will apparently get you back one penalty point for each time you do this. DO NOT use Med-X to execute this repair because it will outlast the poison and do nothing for you.

There is no way to prevent this from happening in the first place either. The only "solution" to this on any of the modding sites simply disables poison resistance which is obviously not a useful fix. We can't do anything about this other than have people pester Bethesda until THEY fix it.