How to switch keyslots during boot decryption/unlocking when having a LUKS TPM2-keyslot with PIN?

What I did

I used the ujust setup-luks-tpm-unlock command, which uses the script /usr/libexec/luks-enable-tpm2-autounlock, which in the end also just uses systemd-cryptenroll.

However I added a PIN for the TPM2 for added security. So the command likely is like this:

$ systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=7+14 --tpm2-with-pin=yes "$CRYPT_DISK"

I had a password for LUKS setup before, and I want to keep it, also as a recovery path or e.g. for having multiple people to have different unlock methods.

One can also see that both keyslots are still in there:

# systemd-cryptenroll ****
SLOT TYPE    
   0 password
   1 tpm2
# cryptsetup luksDump /dev/disk/by-uuid/****
LUKS header information
Version:        2
Epoch:          5
Metadata area:  16384 [bytes]
Keyslots area:  16744448 [bytes]
UUID:           ******
Label:          (no label)
Subsystem:      (no subsystem)
Flags:          (no flags)

Data segments:
  0: crypt
        offset: 16777216 [bytes]
        length: (whole device)
        cipher: aes-xts-plain64
        sector: 512 [bytes]

Keyslots:
  0: luks2
        Key:        512 bits
        Priority:   normal
        Cipher:     aes-xts-plain64
        Cipher key: 512 bits
        PBKDF:      argon2id
        Time cost:  7
        Memory:     1048576
        Threads:    4
        Salt:       ******
        AF stripes: 4000
        AF hash:    sha256
        Area offset:32768 [bytes]
        Area length:258048 [bytes]
        Digest ID:  0
  1: luks2
        Key:        512 bits
        Priority:   normal
        Cipher:     aes-xts-plain64
        Cipher key: 512 bits
        PBKDF:      pbkdf2
        Hash:       sha512
        Iterations: 1000
        Salt:       ******
        AF stripes: 4000
        AF hash:    sha512
        Area offset:290816 [bytes]
        Area length:258048 [bytes]
        Digest ID:  0
Tokens:
  0: systemd-tpm2
        tpm2-hash-pcrs:   7+14
        tpm2-pcr-bank:    sha256
        tpm2-pubkey:
                    (null)
        tpm2-pubkey-pcrs: 
        tpm2-primary-alg: ecc
        tpm2-pin:         true
        tpm2-pcrlock:     false
        tpm2-salt:        true
        tpm2-srk:         true
        tpm2-pcrlock-nv:  false
        tpm2-policy-hash:
                    ******
        tpm2-blob:  ******
        Keyslot:    1
Digests:
  0: pbkdf2
        Hash:       sha256
        Iterations: 122040
        Salt:     ******
        Digest:   ******

Problem

Now, when booting, one can enter the PIN and TPM-unlock works flawlessly. The problem is: There is no way to still use the passphrase that I can see - despite it being still available as a keyslot.

When booting and I am at the unlock screen, I can press Esc to see all log details. And there you can see it similar to “Requesting PIN for TPM:”.
If one enters the wrong PIN there, the requests just keeps showing up.

Does not this make having a recovery passphrase a moot point? If I cannot switch to it to use it for unlocking in practice.
Say I e.g. forget my TPM PIN. I cannot unlock and boot the device anymore?

Note I know I may be able to unlock it by booting some live system etc. and then maybe remove the tpm keyslot. But really, this is no useful point.

What I would have expected

Maybe to just take the entered “PIN”, which is also just like a “password” (aka it does not have to be numeric at all), and also check it against the passphrase keyslots.
If anyone unlocks that is fine, the user provided the correct one.

Alternatively though IMHO not that sleep: Somehow allow the user to switch authentication methods, at least when entering one does not work.

Another use case: multi-user

Also, to demonstrate a use case here why that may be useful beyond simple recovery is: multi-user setups. Linux is good for multi-users to use and at userland has good user segregation. However to decrypt an encrypted disk at startup, sharing a password would be bad. So either add a new password or TPM-unlock?
That said, people are reasonably weak at choosing passwords and in general, I thought, yeah using TPM2 surely can only increase security here, after all you can couple it with the PIN. The idea was: Unlocking works with the usual password and TPM2 with PIN.

Apparently, that does not work and just because of a “input/password request” problem. Technically, as far as I see, everything is set up for it. And that’s sad.
Or is there anything one can do here?

Docs

I checked multiple LUKS and systemd-cryptenroll docs, but found this special case (TPM with PIN) nowhere to be explained in detail.