We need to decide which directory to create if neither of the 2
candidates exist. Previously, we had only one candidate, and we'd
create it without any regrets. I don't think I see how the current
situation is suddenly different. Moreover, I don't think I understand
how to do something smarter, without breaking gobs of code that
depends on user-emacs-directory be existing. Ideas welcome.
The problem I see (which is indeed the problem I ran into myself, someone who has used Emacs since the 80's) is that users will be confused about this because the behaviour is new, and from the user's point of view, Emacs simply isn't reading the $HOME/.emacs.d directory.
This confusion is caused by an unfamiliarity with new functionality. Therefore, we must conclude that this is a temporary problem. Once documentation, youtube videos, block posts, stackoveflow examples, etc has been updated, this will no longer be a problem. Just like the confusion between $HOME/.emacs and $HOME/.emacs.d/init.el is no longer an issue.
But for now, it is an issue, so the solution should be to put this information in from of the user's face. That means displaying a warning (including a y/n prompt) when the directory is created.
Perhaps this warning is only needed for one release cycle, and then users will be informed. But right now, it is my opinion that such warning is needed.