Hello,
The issue seems to be related to X11, so I downgraded "plasma-workspace-x11.x86_64" to 6.0.3-1.fc40.
And I have no issue at all now, beside the update notification for plasma updates
More info about the bug :
https://bugs.kde.org/show_bug.cgi?id=488187#c7
Edit :
Actually I just need to downgrade to 6.0.5