How to write script output to file and command-line?

I have a long-running Python script that I run from the command-line. The script writes progress messages and results to the standard output. I want to capture everything the script write to the standard output in a file, but also see it on the command line. Alternatively, I want the output to go to the file immediately, so I can use tail to view the progress. I have tried this:

python MyLongRunngingScript.py | tee log.txt

But it does not produce any output (just running the script produces output as expected). Can anyone propose a simple solution? I am using Mac OS X 10.6.4.

Edit I am using print for output in my script.


Asked by: Edgar690 | Posted: 30-11-2021






Answer 1

You are on the right path but the problem is python buffering the output.

Fortunately there is a way to tell it not to buffer output:

python -u MyLongRunngingScript.py | tee log.txt

Answered by: Nicole878 | Posted: 01-01-2022



Answer 2

The fact that you don't see anything is probably related to the fact that buffering is occurring. So you only get output every 4 Ko of text or so.

instead, try something like this :

class OutputSplitter(object):
    def __init__(self, real_output, *open_files):
        self.__stdout = real_output
        self.__fds = open_files
        self.encoding = real_output.encoding
    def write(self, string):
        self.__stdout.write(string) # don't catch exception on that one.
        self.__stdout.flush()
        for fd in self.__fds:
            try:
                fd.write(string)
                fd.flush()
            except IOError:
                pass # do what you want here. 
    def flush(self):
        pass # already flushed

Then decorate sys.stdout with that class with some code like that :

stdout_saved = sys.stdout
logfile = open("log.txt","a") # check exception on that one.
sys.stdout = OutputSplitter(stdout_saved, logfile)

That way, every output (print included) is flushed to the standard output and to the specified file. Might require tweaking because i haven't tested that implementation.

Of course, expect to see a (small most of the time) performance penalty when printing messages.

Answered by: Rafael466 | Posted: 01-01-2022



Answer 3

Another simple solution could also be

python script.py > output.log

Answered by: Kelsey272 | Posted: 01-01-2022



Answer 4

You could try doing sys.stdout.flush() occasionally in your script, and running with tee again. When stdout is redirected through to tee, it might get buffered for longer than if it's going straight to a terminal.

Answered by: Fenton593 | Posted: 01-01-2022



Similar questions

Using Python Fabric without the command-line tool (fab)

Altough Fabric documentations refers to a way of using the library for SSH access without requiring the fab command-line tool and/or tasks, I can't seem to manage a way to do it. I want to run this file (example.py) by only executing 'python example.py': env.hosts = [ "example.com" ] def ps(): run("ps") ps() Thanks.


python - Command-line input

I need to write a complete program that obtains three pieces of data and then process them. The three pieces of information are a Boolean value, a string, and an integer. The logic of the program is this: if the Boolean value is True, print out the string twice, once with double quotes and once without - otherwise print out twice the number. Which I can do just fine, but then I have to write the same program but this time ...


python - How to open a file that is input by the user in command-line?

So if the user is running my file (test.py) from the command line, and wants to open a file in that program, how do I open it? Say if the user writes 'test.py < file2.py' in command line, how do I then open file2.py in test.py? I know it's something to do with sys.argv.


python - Create a command-line in Ubuntu

In Ubuntu, I'd like to create a command-line called pycharm_help which will open firefox or another browser with the website https://docs.python.org/2.7/py-modindex.html. I know for doing this, I need to create a script with #!/usr/bin at the beginning. As I'm starting doing programming, I'd like that someone coul...


python - Adding command-line in cmd

Is there any way to create a command on Command Prompt like I want to create a command named createdirectory(I know there is an existing command for that but take this as an example) . when i execute the command "createdirectory" it will run a python file. I want it in such a way that i can run this command from anywhere any disk volume or any folder. If you know anything then please post your answer....


python - get user last input command-line input

is it possible to get the user's last input? I am working in a program that needs to check every char, that the user enters. my question how do I check if the user pressed the Up button, then get the last user input and display it like when you press the up button in a cmd, it gives you the last input you entered here's my simple test program


linux - How to make a python, command-line program autocomplete arbitrary things NOT interpreter

I am aware of how to setup autocompletion of python objects in the python interpreter (on unix). Google shows many hits for explanations on how to do this. Unfortunately, there are so many references to that it is difficult to find what I need to do, which is slightly different. I need to know how to enable, tab/auto completion of arbitrary items in a command-line program written in ...


java - OCSP command-line test tool?

Closed. This question does not meet Stack Overflow guid...


python - How to prompt for user input and read command-line arguments

Closed. This question needs to be more focused. It ...


eclipse - Integrating command-line generated python .coverage files with PyDev

My build environment is configured to compile, run and create coverage file at the command line (using Ned Batchelder coverage.py tool). I'm using Eclipse with PyDev as my editor, but for practical reasons, it's not possible/convenient for me to convert my whole build environment to Eclipse (and thus generate the coverage data directly from the IDE, as it's designed to do) PyDev seems to be using the same ...


c++ - How do I compile a Visual Studio project from the command-line?

I'm scripting the checkout, build, distribution, test, and commit cycle for a large C++ solution that is using Monotone, CMake, Visual Studio Express 2008, and custom tests. All of the other parts seem pretty straight-forward, but I don't see how to compile the Visu...


How to make a simple command-line chat in Python?

I study network programming and would like to write a simple command-line chat in Python. I'm wondering how make receving constant along with inputing available for sending at any time. As you see, this client can do only one job at a time: from socket import * HOST = 'localhost' PORT = 21567 BUFSIZE = 1024 ADDR = (HOST, PORT) tcpCliSock = socket(AF_INET, SOCK_STREAM) tcpCliSock.connect(A...


interpreter - Persistent Python Command-Line History

I'd like to be able to "up-arrow" to commands that I input in a previous Python interpreter. I have found the readline module which offers functions like: read_history_file, write_history_file, and set_startup_hook. I'm not quite savvy enough to put this into practice though, so could someone please help? My thoughts on the solution are: (1) Modify .login PYTHO...


data structures - Processing command-line arguments in prefix notation in Python

I'm trying to parse a command-line in Python which looks like the following: $ ./command -o option1 arg1 -o option2 arg2 arg3 In other words, the command takes an unlimited number of arguments, and each argument may optionally be preceded with an -o option, which relates specifically to that argument. I think this is called a "prefix notation". In the Bourne shell I wo...


Display constantly updating information in-place in command-line window using python?

I am essentially building a timer. I have a python script that monitors for an event and then prints out the seconds that have elapsed since that event. Instead of an ugly stream of numbers printed to the command line, I would like to display only the current elapsed time "in-place"-- so that only one number is visible at any given time. Is there a simple way to do this? If possible I'd like to u...


python - Linux command-line call not returning what it should from os.system?

I need to make some command line calls to linux and get the return from this, however doing it as below is just returning 0 when it should return a time value, like 00:08:19, I am testing the exact same call in regular command line and it returns the time value 00:08:19 so I am confused as to what I am doing wrong as I thought this was how to do it in python. import os...






Still can't find your answer? Check out these communities...



PySlackers | Full Stack Python | NHS Python | Pythonist Cafe | Hacker Earth | Discord Python



top