@stefan I think you're tying yourself in a lot of knots to not accept that what people want, when they get replies to their followers-only posts, is to have those replies visible to their followers.
When you see a followers-only post, the default audience for your reply should be the OP's followers.
I'd like to make it clear that this is possible with ActivityPub, and it's the way other social networks handle this feature.