Performance optimizations harden code for specific situations. Once the code is running then optimization is considered. The code is profiled for very specific situations. Inefficiencies are identified and if these inefficiencies are addressed with true refactoring then the code base isn't necessarily hardened. However if the inefficiencies require rearchitecturing, either at the micro level or the macro, the code is hardened.
For instance, if the inefficiency is simply changing a for loop, either by removing nested loops or moving invariants outside of the loop then this is simple refactoring.
If the inefficiency has to be addressed with caching and object lifespan management then every new feature will experience additional cost of development in order to work in this more complex system. This type of optimization is done through architectural changes. Other optimizations that require rearchitecture include the introduction of threading, asynchronous calls, lazy loading, etc.
Sometimes new features will conflict so much with the optimized architecture that either the new feature is not added or the optimization code has to be unwound, untangled, and removed. From my experience trying to remove an optimization is too expensive. By their nature the optimization is highly coupled.