What happens to overwritten variables in python?

Question

I am writing some python code to process huge amounts of data (almost 6 million pieces!). In the code, I'm using a huge for loop to process each set. In that loop, I'm using the same variables every loop and overwriting them. When I ran the program, I noticed that the longer I ran it, the slower it got. Furthermore, upon further experimenting, I discovered that the speed if you ran it for values 10,000 - 10,100 was the same as from 0 to 100. Thus I concluded that since I was not creating more variables and merely processing existing ones, every time I overwrote a variable, it must be being saved somewhere by python.

So: Am I right? must it be python saving my overwritten somewhere? Or am I wrong? Is something else happening?

we're going to need to see some code to help you with this one. — SingleNegationElimination, Jun 07 '13 at 19:27
Python does not 'remember' the old values; but it could well be that you are building up structures in memory you are not aware of, causing your system to swap. — Martijn Pieters, Jun 07 '13 at 19:27
Another possibility is that you have another inefficiency in your code: looping repeatedly over a growing list of values; this will get slower as your data structure grows. — Martijn Pieters, Jun 07 '13 at 19:28

score 2 · Answer 1 · answered Jun 07 '13 at 19:25

Python uses reference counting to keep track of variables. When all references to a variable are removed, that variable is garbage collected. However that garbage collection is done by python at it's own whim, not right away.

It could be that your code is going faster than python garbage collects, or that you have something wrong with your code. Since you didn't give any of your code there's no real way to know.

dr jimbob · Answer 2 · 2013-06-07T19:42:53.373

Python does not copy the original value of a variable before saving an overwritten variables.

Possibly you are seeing the effect of various caches with the program slowing down. Or if you are creating objects, an garbage collector is being called to delete the objects you created that are no longer referenced.

Do you have example code that shows this behavior you are seeing?

For example:

import hashlib
import random
import time
def test():
    t = []
    for i in xrange(20000):        
        if (i == 0) | (i==100)|(i==10000)|(i==10100):
            t.append(time.time())
        for j in range(1,10):
            a = hashlib.sha512(str(random.random()))
            b = hashlib.sha512(str(random.random()))
            c = hashlib.sha512(str(random.random()))
            d = hashlib.sha512(str(random.random()))
            e = hashlib.sha512(str(random.random()))
            f = hashlib.sha512(str(random.random()))
            g = hashlib.sha512(str(random.random()))
    print t[1]-t[0], t[3]-t[2]

Then running 10 times:

>>> for i in range(10):
        test()
0.0153768062592 0.0147190093994
0.0148379802704 0.0147860050201
0.0145788192749 0.0147390365601
0.0147459506989 0.0146520137787
0.0147008895874 0.0147621631622
0.0145609378815 0.0146908760071
0.0144789218903 0.014506816864
0.0146539211273 0.0145659446716
0.0145878791809 0.0146989822388
0.0146920681 0.0147240161896

Gives nearly identical times to within standard error (especially if I exclude the very first interval where it was slightly slower where it had to first initialize a,b,c,d,e,f,g).

What happens to overwritten variables in python?

2 Answers2

Linked