bug#23095: 24.5; python-shell-unbuffered causes problems on Windows

[Will Wykeham]

The relevant context is mostly below, but the important bits are:
Emacs 24.5
Microsoft Windows 8
Python 2.7.11

>From a user facing point of view, the issue I'm having is that 'out of
the box', the function python-shell-prompt-detect produces the warning
"Warning (emacs): Python shell prompts cannot be detected".

To reduce the test case down, this demonstrates the core problem:

(let ((process-environment (python-shell-calculate-process-environment))
       (code (concat "import sys\n"
                    "print(sys.ps1)")))
  (with-temp-buffer
    (let ((code-file (python-shell--save-temp-file code)))
      (call-process "python" code-file '(t nil) nil "-i")
      (delete-file code-file))
    (buffer-string)))

In an 'out of the box' emacs, with 'python-shell-unbuffered' set to t as
default, this evaluates to an empty string.

If python-shell-unbuffered is then set to nil, then the above evaluates to
the expected ">>>\n".

I haven't got to the bottom of why unbuffered is causing this
problem. My suspicion is that it relates to the fact that
PYTHONUNBUFFERED on Windows also causes python to process stdin/stdout
as binary (which also affects line ending processing).

Happy to provide any further information, testing etc. to help narrow
down the issue.

Regards,
Will

********************************************

In GNU Emacs 24.5.1 (i686-pc-mingw32)
 of 2015-04-11 on LEG570
Windowing system distributor `Microsoft Corp.', version 6.3.9600
Configured using:
 `configure --prefix=/c/usr --host=i686-pc-mingw32'

Important settings:
  value of $LANG: ENG
  locale-coding-system: cp1252

Major mode: Lisp Interaction

Minor modes in effect:
  which-function-mode: t
  diff-auto-refine-mode: t
  global-highlight-parentheses-mode: t
  highlight-parentheses-mode: t
  show-paren-mode: t
  dtrt-indent-mode: t
  iswitchb-mode: t
  shell-dirtrack-mode: t
  winner-mode: t
  global-ede-mode: t
  global-semantic-mru-bookmark-mode: t
  global-semanticdb-minor-mode: t
  global-semantic-decoration-mode: t
  global-semantic-idle-completions-mode: t
  global-semantic-idle-scheduler-mode: t
  global-semantic-idle-summary-mode: t
  global-semantic-highlight-func-mode: t
  global-semantic-stickyfunc-mode: t
  semantic-mode: t
  delete-selection-mode: t
  cua-mode: t
  global-auto-revert-mode: t
  desktop-save-mode: t
  savehist-mode: t
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
Type "q" in help window to restore its previous buffer.

Quit [2 times]
Mark set [2 times]
""
Mark set
Quit
Mark saved where search started
Type "q" in help window to restore its previous buffer.

""

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug conf-mode nxml-uchnm rng-xsd xsd-regexp
rng-cmpct nxml-mode-expansions html-mode-expansions sgml-mode rng-nxml
rng-valid rng-loc rng-uri rng-parse nxml-parse rng-match rng-dt rng-util
rng-pttrn nxml-ns nxml-mode nxml-outln nxml-rap nxml-util nxml-glyph
nxml-enc xmltok browse-url url-util cua-rect rect expand-region
text-mode-expansions cc-mode-expansions python-el-fgallina-expansions
er-basic-expansions expand-region-core expand-region-custom
semantic/tag-write semantic/edit debug misearch multi-isearch
semantic/db-typecache semantic/imenu semantic/sb ede/dired vc-hg xhg-dvc
xhg xhg-annotate xhg-mq xhg-log vc-svn eldoc which-func imenu
semantic/tag-file semantic/db-file data-debug cedet-files tla hl-line
smerge-mode dvc-state dvc-config dvc-diff dvc-fileinfo diff tla-gnus
dvc-gnus gnus-art mm-uu mml2015 mm-view mml-smime smime dig mailcap
gnus-sum nnoo gnus-group gnus-undo nnmail mail-source gnus-start
gnus-spec gnus-int gnus-range message rfc822 mml mml-sec mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
gnus-win gnus gnus-ems nnheader dvc-cmenu dvc-about dvc-version
dvc-revlist semantic/wisent/python semantic/decorate/include warnings
semantic/dep semantic/wisent/python-wy semantic/wisent
semantic/wisent/wisent rx superword subword python json ede/cpp-root
ede/emacs bzr-core tla-core pp tla-autoconf tla-defs dvc-log vc
vc-dispatcher add-log diff-mode ediff-merg ediff-wind ediff-diff
ediff-mult ediff-help ediff-init ediff-util ediff xdarcs-core xgit-core
xhg-core xmtn-minimal kalasm-mode qmake-mode bb-mode dts-mode php-mode
etags cc-langs generic generic-x cc-mode cc-fonts cc-guess cc-menus
cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs dvc-autoloads
dvc-unified dvc-tips dired-x dvc-core edebug dvc-lisp dvc-buffers
derived dvc-ui dvc-register dvc-utils dvc-emacs ewoc dvc-defs dvc-site
highlight-parentheses paren dtrt-indent iswitchb tramp tramp-compat
tramp-loaddefs trampver shell pcomplete format-spec icomplete+ icomplete
icicles icicles-mode easy-mmode dired icicles-cmd cus-edit cus-start
cus-load icicles-mcmd icicles-mac icicles-fn icicles-face icicles-var
icicles-opt ffap url-parse auth-source gnus-util password-cache url-vars
color-theme sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr
mail-utils reporter thingatpt hexrgb two-column cl-macs cl winner
windmove framemove findstr grep compile comint ansi-color ede/speedbar
ede/files ede ede/base ede/auto ede/source eieio-speedbar eieio-custom
wid-edit semantic/mru-bookmark semantic/db-mode semantic/decorate/mode
semantic/complete semantic/idle eieio-opt help-mode speedbar sb-image
dframe find-func semantic/ia semantic/analyze/refs semantic/db-find
semantic/db-ref semantic/senator semantic/decorate ring pulse
semantic/analyze semantic/sort semantic/scope semantic/analyze/fcn
semantic/db gv eieio-base semantic/ctxt semantic/format ezimage
semantic/tag-ls semantic/find semantic/util-modes semantic/util semantic
semantic/tag semantic/lex semantic/fw eieio byte-opt bytecomp
byte-compile cl-extra cconv eieio-core mode-local cedet edmacro kmacro
advice help-fns delsel cua-base saveplace autorevert filenotify desktop
frameset cl-loaddefs cl-lib savehist server info easymenu package
epg-config time-date tooltip electric uniquify ediff-hook vc-hooks
lisp-float-type mwheel dos-w32 ls-lisp w32-common-fns disp-table w32-win
w32-vars tool-bar dnd fontset image regexp-opt fringe tabulated-list
newcomment lisp-mode prog-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
frame cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese case-table epa-hook jka-cmpr-hook help
simple abbrev minibuffer nadvice loaddefs button faces cus-face macroexp
files text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
w32notify w32 multi-tty emacs)

Memory information:
((conses 8 649133 135398)
 (symbols 32 52813 0)
 (miscs 32 11322 2303)
 (strings 16 127824 18777)
 (string-bytes 1 3792793)
 (vectors 8 54846)
 (vector-slots 4 1586068 18114)
 (floats 8 526 1355)
 (intervals 28 5955 102)
 (buffers 508 40))