Are Python docstrings and comments stored in memory when a module is loaded?

Are Python docstrings and comments stored in memory when a module is loaded?

I've wondered if this is true, because I usually document my code well; may this affect memory usage?

Usually every Python object has a __doc__ method. Are those docstrings read from the file, or processed otherwise?

I've done searches here in the forums, Google and Mailing-Lists, but I haven't found any relevant information.

Do you know better?


Asked by: Aida769 | Posted: 30-11-2021






Answer 1

By default, docstrings are present in the .pyc bytecode file, and are loaded from them (comments are not). If you use python -OO (the -OO flag stands for "optimize intensely", as opposed to -O which stands for "optimize mildly), you get and use .pyo files instead of .pyc files, and those are optimized by omitting the docstrings (in addition to the optimizations done by -O, which remove assert statements). E.g., consider a file foo.py that has:

"""This is the documentation for my module foo."""

def bar(x):
  """This is the documentation for my function foo.bar."""
  return x + 1

you could have the following shell session...:

$ python -c'import foo; print foo.bar(22); print foo.__doc__'
23
This is the documentation for my module foo.
$ ls -l foo.pyc
-rw-r--r--  1 aleax  eng  327 Dec 30 16:17 foo.pyc
$ python -O -c'import foo; print foo.bar(22); print foo.__doc__'
23
This is the documentation for my module foo.
$ ls -l foo.pyo
-rw-r--r--  1 aleax  eng  327 Dec 30 16:17 foo.pyo
$ python -OO -c'import foo; print foo.bar(22); print foo.__doc__'
23
This is the documentation for my module foo.
$ ls -l foo.pyo
-rw-r--r--  1 aleax  eng  327 Dec 30 16:17 foo.pyo
$ rm foo.pyo
$ python -OO -c'import foo; print foo.bar(22); print foo.__doc__'
23
None
$ ls -l foo.pyo
-rw-r--r--  1 aleax  eng  204 Dec 30 16:17 foo.pyo

Note that, since we used -O first, the .pyo file was 327 bytes -- even after using -OO, because the .pyo file was still around and Python didn't rebuild/overwrite it, it just used the existing one. Removing the existing .pyo (or, equivalently, touch foo.py so that Python knows the .pyo is "out of date") means that Python rebuilds it (and, in this case, saves 123 bytes on disk, and a little bit more when the module's imported -- but all .__doc__ entries disappear and are replaced by None).

Answered by: William259 | Posted: 01-01-2022



Answer 2

Yes the docstrings are read from the file, but that shouldn't stop you writing them. Never ever compromise readability of code for performance until you have done a performance test and found that the thing you are worried about is in fact the bottleneck in your program that is causing a problem. I would think that it is extremely unlikely that a docstring will cause any measurable performance impact in any real world situation.

Answered by: Stuart373 | Posted: 01-01-2022



Answer 3

They are getting read from the file (when the file is compiled to pyc or when the pyc is loaded -- they must be available under object.__doc__) but no --> this will not significantly impact performance under any reasonable circumstances, or are you really writing multi-megabyte doc-strings?

Answered by: Marcus270 | Posted: 01-01-2022



Answer 4

Do Python docstrings and comments are stored in memory when module is loaded?

Docstrings are compiled into the .pyc file, and are loaded into memory. Comments are discarded during compilation and have no impact on anything except the insignificant extra time taken to ignore them during compilation (which happens once only after any change to a .py file, except for the main script which is re-compiled every time it is run).

Also note that these strings are preserved only if they are the first thing in the module, class definition, or function definition. You can include additional strings pretty much anywhere, but they will be discarded during compilation just as comments are.

Answered by: Kelvin461 | Posted: 01-01-2022



Answer 5

As other answers mentioned, comments are discarded in compilation process but docstrings are stored in .pyc file and are loaded into the memory.

In .pyc files, there are code objects that are serialized with marshal. Although it's not supposed to be readable but you can still find something. So why not just see that it is indeed in .pyc file?

import marshal

text = '''def fn():
    """ZZZZZZZZZZZZZZZZZZ"""
    # GGGGGGGGGGGGGGGGGGG'''

code_object = compile(text, '<string>', 'exec')
serialized = marshal.dumps(code_object)
print(serialized)
print(b"ZZZZZZZZZZZZZZZZZZ" in serialized)
print(b"GGGGGGGGGGGGGGGGGGG" in serialized)

output:

b'\xe3\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00@\x00\x00\x00s\x0c\x00\x00\x00d\x00d\x01\x84\x00Z\x00d\x02S\x00)\x03c\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00C\x00\x00\x00s\x04\x00\x00\x00d\x01S\x00)\x02Z\x12ZZZZZZZZZZZZZZZZZZN\xa9\x00r\x01\x00\x00\x00r\x01\x00\x00\x00r\x01\x00\x00\x00\xfa\x08<string>\xda\x02fn\x01\x00\x00\x00s\x02\x00\x00\x00\x00\x01r\x03\x00\x00\x00N)\x01r\x03\x00\x00\x00r\x01\x00\x00\x00r\x01\x00\x00\x00r\x01\x00\x00\x00r\x02\x00\x00\x00\xda\x08<module>\x01\x00\x00\x00\xf3\x00\x00\x00\x00'
True
False

where is it referenced in the function's code object? in .co_consts

new_code_object = marshal.loads(serialized)
print(new_code_object.co_consts[0].co_consts[0])

output:

ZZZZZZZZZZZZZZZZZZ

def fn():
    """ZZZZZZZZZZZZZZZZZZ"""
    # GGGGGGGGGGGGGGGGGGG

print(fn.__code__.co_consts[0] is fn.__doc__) # True

Answered by: John515 | Posted: 01-01-2022



Similar questions

python - Docstrings vs Comments

I'm a bit confused over the difference between docstrings and comments in python. In my class my teacher introduced something known as a 'design recipe', a set of steps that will supposedly help us students plot and organize our coding better in Python. From what I understand, the below is an example of the steps we follow - this so call design recipe (the stuff in the quotations): def term_work_ma...


python - Docstrings vs Comments

I'm a bit confused over the difference between docstrings and comments in python. In my class my teacher introduced something known as a 'design recipe', a set of steps that will supposedly help us students plot and organize our coding better in Python. From what I understand, the below is an example of the steps we follow - this so call design recipe (the stuff in the quotations): def term_work_ma...


python - How do I change pylint to look for doxygen comments instead of docstrings?

When I pylint my code, pylint warns about missing docstring for some function or member function. That is, it complains about missing '''function comment'''. My company uses doxygen's comment blocks that begin with ## and comes before the definition of the function. We don't use docstrings for commenting. Aside from turning off missing docstring warning, is there anyway to tell pylint, hey instead of loo...


python - Is it possible to let PyCharm auto break line when writing long docstrings and comments?

Writing long docstrings and long comments is annoying because of the 80-characters width limit. For example, I write something like: def fun(self): """Return some thing This function do some complex work and return something that need a long sentence to describe """ And then I found I need to insert something in the third line of docstring. After inserting, the wid...


python - Docstrings for data?

Is there a way to describe the module's data in a similar way that a docstring describes a module or a funcion? class MyClass(object): def my_function(): """This docstring works!""" return True my_list = [] """This docstring does not work!"""


syntax error - python docstrings

ok so I decided to learn python (perl, c, c++, java, objective-c, ruby and a bit of erlang and scala under my belt). and I keep on getting the following error when I try executing this: Tue Jul 21{stevenhirsch@steven-hirschs-macbook-pro-2}/projects/python:--&gt;./apache_logs.py File "./apache_logs.py", line 17 print __doc__ ^ SyntaxError: invalid syntax #!/usr/local/bin/python """...


Python decorator handling docstrings

I have a problem using docstrings with decorators. Given the following example: def decorator(f): def _decorator(): print 'decorator active' f() return _decorator @decorator def foo(): '''the magic foo function''' print 'this is function foo' help(foo) Now the help doesn't show me the docstring of foo as expected, it shows: Hel...


How to fold long docstrings in python source code in VIM?

Does anybody know of a method, or perhaps a plugin, that will automatically fold long docstrings in Python? I have docstrings in my code that span several pages, so it is troublesome to keep paging through them. The other tricky part is that there is embedded python testing code in the docstrings, so that might make parsing them difficult. Note that I only need to automatically fold the entire docstr...


python - Using shorter textwidth in comments and docstrings

From the mighty PEP 8: [P]lease limit all lines to a maximum of 79 characters. For flowing long blocks of text (docstrings or comments), limiting the length to 72 characters is recommended. When editing Python code in Vim, I set my textwidth to 79, and Vim au...


python - Writing docstrings - specifying functions arguments and returns

Suppose I have a function, say: &gt;&gt;&gt; def foo(a): return a+1 I want to write a documentation string for it. what is the convention in specifying in the docstring that it takes a and returns a+1?


python - Is there a way to keep docstrings separate from the functions they document?

I'm working on a module with many small functions but whose docstrings tend to be quite long. The docstrings make working on the module irritating as I have to constantly scroll over a long docstring to find a little bit of actual code. Is there a way to keep docstrings separate from the functions they document? I'd really like to be able to specify the docstrings at the end of the file away from the code or, even...


How do I change Emacs's font face for Python docstrings?

I'm just starting to learn Python and use Emacs as my editor. Currently, Emacs uses the same color for normal strings (single quotes) and docstrings (triple quotes). I want docstrings to be a different color, so I used the 'Options->Customize Emacs' menu option to change 'font-lock-doc-face' to a new color and saved the changes. However, Emacs continues to keep docstrings the same color as normal strings. Changing the ...


python - Put docstrings on special methods?

I'm trying to decide what information to put in the class docstring and what to put in the __init__ method docstring. Up until now I've been putting an overview of the class and how to work with it in the class docstring, while stuff directly related to initialization (argument details etc.) I put in the __init__ docstring. Today I started wondering if this was the right way of doing it, s...


static analysis - How to calculate lines of code, comments and docstrings for a Python module?

Is there a tool or snippet that produces the following output in some form: lines_of_code = 98 lines_of_comments = 24 lines_of_documentation = 11 NOTE 1: I will then try to feed this data to Jenkins to graph. NOTE 2: I am aware that CLOC co...






Still can't find your answer? Check out these communities...



PySlackers | Full Stack Python | NHS Python | Pythonist Cafe | Hacker Earth | Discord Python



top