The problem is that there is no one solution that floats everyone's boat. Whatever you choose, someone sinks (i.e. has broken completion). The only way to float everyone's boat is for the completion function to check which function called it and then change its behavior based on prior knowledge of what each caller expects. Consider the following two functions:
(defun pick-a-fruit ()
(interactive)
(let* ((default "banana")
(prompt (format "Pick a fruit (default %s): " default))
(collection '("apple" "banana" "cherry"))
(selection (completing-read prompt collection nil t)))
(when (string= selection "")
(setq selection default))
(message "You selected %s" selection)))(defun pick-a-tool ()
(interactive)
(let* ((prompt "Pick a tool: ")
(collection '("hammer" "screwdriver" "drill"))
(selection (completing-read prompt collection nil t)))
(message "You selected %S" selection)))The fruit function expects completing-read to return an empty string to indicate that the default should be used, while the tool function expects completing-read to always return an element from the collection no matter what. Technically, the tool function is wrong, because completing-read is documented to return an empty string even when REQUIRE-MATCH is non-nil. But there are plenty of functions that use this pattern and expect it to work, because they don't know about this edge case, and it would be nice to be able to support them in a custom completion system without breaking the functions that use the former pattern. As it is, you have to choose one or the other to break, or else imeplement caller-dependent behavior to give everyone what they expect, which is messy.
As I said, though, treating DEF = nil as equivalent to DEF = "" might form the basis for an acceptable compromise. It will still break the pick-a-tool case, but it will be clearer why it's breaking, since the empty string will be presented explicitly as the first completion option. I'll test it out.