Non-blocking read on a subprocess.PIPE in python

Question

I'm using the subprocess module to start a subprocess and connect to its output stream (stdout). I want to be able to execute non-blocking reads on its stdout. Is there a way to make .readline non-blocking or to check if there is data on the stream before I invoke .readline? I'd like this to be portable or at least work under Windows and Linux.

here is how I do it for now (It's blocking on the .readline if no data is available):

p = subprocess.Popen('myprogram.exe', stdout = subprocess.PIPE)

output_str = p.stdout.readline()

Answer 1

To read a stream without blocking regardless of the operating system in Python you would do it by using Queue.get_nowait() function:

import sys

from subprocess import PIPE, Popen from threading

import Thread

try:

   from queue import Queue, Empty

except ImportError:

   from Queue import Queue, Empty # python 2.x

   ON_POSIX = 'posix' in sys.builtin_module_names

def enqueue_output(out, queue):

      for line in iter(out.readline, b''):

            queue.put(line)

            out.close()

p = Popen(['myprogram.exe'], stdout=PIPE, bufsize=1, close_fds=ON_POSIX)

q= Queue()

t = Thread(target=enqueue_output, args=(p.stdout, q))

t.daemon = True # thread dies with the program

t.start()

# ... do other things here

# read line without blocking

try:

    line = q.get_nowait() # or q.get(timeout=.1)

except Empty:

    print('no output yet')

else:

   # got line # ... do something with line