aarch64: gate CLIDR_EL1 override to 6.10 and newer kernels
Original CLIDR_EL1 overwrite was done in Firecracker in the
"b5ac3a6ee7bc0828596c386c9def1bf228f1234c"("fix(aarch64): override
fabricated CLIDR_EL1 to match host cache topology") commit to fix the
change in behavior in the kernel 6.3 from the commit "7af0c2534f4c"
("KVM: arm64: Normalize cache configuration") which started fabricating
CLIDR_EL1 register.
The issue here is that for secondary vcpus on aarch64 the PSCI boot up
process resets vcpu registers (including ID registers like CLIDR_EL1).
This in creates a situation where the vcpu0 has correct CLIDR_EL1 while
all secondary vcpus have KVM default CLIDR_EL1. This combination affects
guest kernel scheduling behavior significantly affecting the
performance.
6.10 kernel contains a fix for this issue
"e016333745c70c960e02b4a9b123c807669d2b22"("KVM: arm64: Only reset
vCPU-scoped feature ID regs once"). This commit (as the title suggests)
prevents resetting of ID registers for already initialized vcpus (all
vcpus are initialized by Firecracker before VM starts). This fixes the
issue of CLIDR_EL1.
All of this creates an unfortunate situation where [6.3..6.10) range of
kernels is affected and have a degraded performance. Even though this
range of kernels does not contain any officially supported kernels by
Firecracker, it would be nice to mediate the impact.
One fix for it is to make secondary vcpu threads to wait until they are
booted by the vcpu0 and only then enter the `KVM_RUN` syscall. This way
there is a window of time where CLIDR_EL1 can be set again to the
correct value. We decide to not use this option since it can potentially
introduce boot time regression and in general adds unwanted complexity.
This leaves the option of gating the CLIDR_EL1 overwrite to only kernels
past 6.10 release. This is not ideal either since cache topology in
affected range of kernels will encounter the original problem the
CLIDR_EL1 override was solving, but at least it does not introduce
performance regressions in the guest.
In the future we will investigate if removing CLIRD_EL1 overwrite and
any cache information from the DT is reasonable to alleviate this
problem once and for all.
Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk> E
Egor Lazarchuk committed
b9bcdd3098e682a92228d77346efa7948d4ebede
Parent: 924c6ad