which inotify event signals the completion of a large file operation?

for large files or slow connections, copying files may take some time.

using pyinotify, i have been watching for the IN_CREATE event code. but this seems to occur at the start of a file transfer. i need to know when a file is completely copied - it aint much use if it's only half there.

when a file transfer is finished and completed, what inotify event is fired?

Asked by: Julia589 | Posted: 06-12-2021

Answer 1

IN_CLOSE probably means the write is complete. This isn't for sure since some applications are bad actors and open and close files constantly while working with them, but if you know the app you're dealing with (file transfer, etc.) and understand its' behaviour, you're probably fine. (Note, this doesn't mean the transfer completed successfully, obviously, it just means that the process that opened the file handle closed it).

IN_CLOSE catches both IN_CLOSE_WRITE and IN_CLOSE_NOWRITE, so make your own decisions about whether you want to just catch one of those. (You probably want them both - WRITE/NOWRITE have to do with file permissions and not whether any writes were actually made).

There is more documentation (although annoyingly, not this piece of information) in Documentation/filesystems/inotify.txt.

Answered by: Richard187 | Posted: 07-01-2022

Answer 2

For my case I wanted to execute a script after a file was fully uploaded. I was using WinSCP which writes large files with a .filepart extension till done.

I first started modifying my script to ignore files if they're themselves ending with .filepart or if there's another file existing in the same directory with the same name but .filepart extension, hence that means the upload is not fully completed yet.

But then I noticed at the end of the upload, when all the parts have been finished, I have a IN_MOVED_IN notification getting triggered which helped me run my script exactly when I wanted it.

If you want to know how your file uploader behaves, add this to the incrontab:

/your/directory/ IN_ALL_EVENTS echo "$$ $@ $# $% $&"

and then

tail -F /var/log/cron

and monitor all the events getting triggered to find out which one suits you best.

Good luck!

Answered by: Kristian881 | Posted: 07-01-2022

Answer 3

Why don't you add a dummy file at the end of the transfer? You can use the IN_CLOSE or IN_CREATE event code on the dummy. The important thing is that the dummy as to be transfered as the last file in the sequence.

I hope it'll help.

Answered by: Charlie935 | Posted: 07-01-2022

Similar questions

python - Remove progress bar after completion of an h2o operation

As a follow-up to How to suppress "Build Progress" bar when training an h2o model?, I would like to keep the progress bar, but only while it is not 100% yet, after which I want it to disappear. Note that clearing the output completely is not an answer: there may be valuable output there already that I want to keep...

python - AWS Batch Operation - completion reports, no message

I make batch operations in my lambda function on a huge number of csv files. I want to have the content of error/exception in my completion reports. I have only 5% of error files so lambda works fine, but it doesn't write errors in the report. When I test my lambda on a file that leads to errors, I see that "ResultMessage" is the same as error or exception. I tried adding a string with exception to report ...

Has anyone found a good set of python plugins for vim -- specifically module completion?

I'm looking for a suite of plugins that can help me finally switch over to vim full-time. Right now I'm using Komodo with some good success, but their vim bindings have enough little errors that I'm tired of it. What I do love in Komodo, though, is the code completion. So, here's what I'm looking for (ordered by importance). Code completion, meaning: the ability to code complete modules/functi...

How do I add tab completion to the Python shell?

When starting a django application using python manage.py shell, I get an InteractiveConsole shell - I can use tab completion, etc. Python 2.5.1 (r251:54863, Apr 15 2008, 22:57:26) [GCC 4.0.1 (Apple Inc. build 5465)] on darwin Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) When just starting a python interpreter using python

Python tell when an ftp transfer sits on completion

I have to download some files from an FTP server. Seems prosaic enough. However, the way this server behaves is if the file is very large, the connection will just hang when the download ostensibly completes. How can I handle this gracefully using ftplib in python? Sample python code: from ftplib import FTP ... ftp = FTP(host) ftp.login(login, passwd) files=ftp.nlst() ftp.set_debuglevel(2)...

python tab completion in windows

I'm writing a cross-platform shell like program in python and I'd like to add custom tab-completion actions. On Unix systems I can use the built-in readline module and use code like the following to specify a list of possible completions when I hit the TAB key: import readline readline.parse_and_bind( 'tab: complete' ) readline.set_completer( ... ) How can I do this on Windows? I'd like to...

python - IO completion port key confusion

I'm writing an IO completion port based server (source code here) using the Windows DLL API in Python using the ctypes module. But this is a pretty direct usage of the API and this question is directed at those who have a knowledge of IOCP, not Python. As I understand the documentation for...

Getting proper code completion for Python on Vim?

I've gotten omnicompletion with Pysmell to work before, but I can't seem to do it again. I tried following some steps online, but most, if not all, of them are to vague and assume too much that you know what you are doing to some extent. Can someone post a full, step-by-step tutorial on how to get code completion working properly, for complete Vim newbies (for dummies?)?

python - Trick code completion with PyDev like with PDT?

Is there a way to help PyDev code completion by telling it the type of a variable? With PDT, you can use PHPDoc-like syntax for such purpose: /* @var $my_var MyClass */ $my_var = myFunction(); // PDT is able to figure out that $my_var is a MyClass object. But till now, I cannot figure out how to do the same in python.

mercurial - Tab Completion in Python Command Line Interface - how to catch Tab events

I'm writing a little CLI in Python (as an extension to Mercurial) and would like to support tab-completion. Specifically, I would like catch tabs in the prompt and show a list of matching options (just like bash). Example: Enter section name: ext*TAB* extensions extras The problem is I'm not sure how to catch the Tab events. I'm using the ui.prompt() API of Merc...

Code completion in NetBeans' python plugin does not work properly

I am asking on StackOverflow because surely I am doing something completely silly and I hope S.O. might provide me with a quick answer. I've installed the latest stable Python-plugin for NetBeans. It works great, and I tested code completion with various packages such as sys, os and so on. It works beautifully. However, it does not seem to pick up the code completion for the code in my ow...

python - Weird behaviour with optparse and bash tab completion

I am building a script for users new to Linux, so please understand why I am asking this :) My script runs like this: python script.py -f filename.txt I am using the optparse module for this. However, I noticed the following when doing tab completion. The tab completion works when I do: python script.py <tab completion> # Tab completion wo...

Still can't find your answer? Check out these communities...

PySlackers | Full Stack Python | NHS Python | Pythonist Cafe | Hacker Earth | Discord Python