Fame decreases while character does not perform any deeds which could raise fame for a while, unless they have a condition (like class) preventing it.