Jy­thon me­mo­ry leak/​Out Of Me­mo­ry pro­blem 5

No­te: This ar­ti­cle has be­en un­pu­blis­hed for qui­te so­me ti­me. It’s main parts da­te back to De­cem­ber 2007. The­re­fo­re, if so­me ver­si­on num­ber seem to be out­da­ted – I am re­fer­ring to the sta­te it had back tho­se days.

We ha­ve a Ja­va ap­p­li­ca­ti­on wi­th em­bed­ded Jy­thon scrip­t­ing en­gi­ne. The Jy­thon scripts do mass com­pu­ta­ti­ons on da­ta sets. So far, we had 3000-4000 da­ta sets in one chunk ma­xi­mum. Now, a new cust­o­m­er starts and will ha­ve 8000 and mo­re da­ta sets.

„No big deal,“ I thought. And star­ted the test run one day be­fo­re our cust­o­m­er will ha­ve the who­le thing run­ning on the pro­duc­tion sys­tem for the first ti­me. Com­pu­ta­ti­on starts: 1000… 2000… 3000… 4000… 5000… bang „Out of me­mo­ry. You should try to in­crea­se heap si­ze“. The ser­ver ap­p­li­ca­ti­on halts com­ple­te­ly and wi­thout any fur­ther warning.

I’m a bit sho­cked. The big pro­blems al­ways ari­se at pla­ce whe­re one would de­fi­ni­te­ly not ex­pect them. I start cir­cum­ven­ti­on at­tempts: Split­ting the run in­to smal­ler chunks – does not work. Re­initia­li­zing the Jy­thon en­vi­ron­ment pe­ri­o­di­cal­ly – ma­kes things worse. Re­pla­c­ing our ra­ther out­da­ted (but other­wi­se func­tio­nal) Jy­thon 2.1 wi­th the then-cur­rent Jy­thon 2.2.1 – does not mat­ter. I do not seem to ha­ve a chan­ce to cy­cle mo­re than about 5200 times through the script be­fo­re I catch an „Out of me­mo­ry“ si­tua­ti­on – or ha­ve to re­start the who­le ser­ver pro­cess.

Weird. What should I tell the cust­o­m­er? „Well, you can­not run your com­pu­ta­ti­ons in one step. Start wi­th the first half, then call us, we will re­start the ser­ver, then do the se­cond half.“ ??? Not a re­al­ly pro­fes­sio­nal way of do­ing things. Even mo­re sur­pri­sing, loo­king at the me­mo­ry si­tua­ti­on wi­th Runtime.freeMemory() and fri­en­ds shows that the­re is no shor­ta­ge of me­mo­ry at all. Ac­tual­ly, when the ap­p­li­ca­ti­on cras­hes, it has used not mo­re than 600 MB out of 2048 MB heap space and mo­re than 50 MB are mar­ked as „free“. This is not pre­cise­ly what I would sum­ma­ri­ze as „out of me­mo­ry“…

Fi­nal­ly, po­king Goo­gle on­ce mo­re brings the so­lu­ti­on. I find an ar­ti­cle about just a si­mi­lar pro­blem. For­t­u­n­a­te­ly, it has a so­lu­ti­on and even ex­plains what’s go­ing on: Jy­thon has an in­ter­nal map­ping of Py­XXX wrap­pers to Ja­va ob­jects. The de­fault con­fi­gu­ra­ti­on uses nor­mal re­fe­ren­ces which ma­kes the­se map­pings re­sis­tant to gar­ba­ge collec­tion. Due to me­cha­nis­ms I do not ful­ly un­der­stand, this leads to enor­mous grow­th of the map­ping set and fi­nal­ly an out-of-me­mo­ry-si­tua­ti­on wi­th the in­ter­nal re­sour­ce ma­nage­ment.

For­t­u­n­a­te­ly, the so­lu­ti­on is as sim­ple as put­ting a


so­mew­he­re in the code be­fo­re the Jy­thon sub­sys­tem is in­itia­li­zed. Then, the in­ter­nal ta­ble is built wi­th weak re­fe­ren­ces and sud­den­ly, ever­y­thing runs smooth­ly. The 8000 da­ta sets are no pro­blem any mo­re and I can de­li­ver the ap­p­li­ca­ti­on as ex­pec­ted. Lu­cky me.

The­re is on­ly one ques­ti­on re­mai­ning: What kind of pa­ra­psy­cho­lo­gi­cal abili­ties are de­ve­l­o­pers ex­pec­ted to ha­ve to find su­ch a so­lu­ti­on wi­thout ha­ving the luck to find an ar­ti­cle de­scri­bing this. And: Why the heck does Jy­thon not use weak re­fe­ren­ces as de­fault? I could not find any pro­blems or even speed pe­nal­ties.

Schreibe einen Kommentar

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert.

5 Gedanken zu “Jy­thon me­mo­ry leak/​Out Of Me­mo­ry pro­blem

  • spacekadety

    Thank you ver­ry mut­ch for this hint, we are ha­ving the sa­me is­sue, be­en loo­king for this so­lu­ti­on for a whi­le ;-). Hap­py Co­ding ever­yo­ne

  • Denis Haskin

    In­te­res­ting. We’re run­ning in­to what so­unds li­ke a very si­mi­lar pro­blem, and I tried ap­p­ly­ing your fix, which didn’t seem to ha­ve any ef­fect (and Jim’s com­ment im­plies that it wouldn’t ha­ve had any ef­fect, any­way).

    We ha­ve to crea­te & store many (~2 mil­lion Py­thon ob­jects) be­fo­re we can pro­cess them, and we seem to be get­ting 10x ConcurrentHashMap$HashEntry ob­jects — I can’t say for su­re sin­ce I’m run­ning on OSX and I’m ha­ving trou­ble get­ting hprof to show me a de­eper stack tra­ce for its tra­ces. We’re pro­bab­ly go­ing to re­fac­tor so­mew­hat and store the ob­ject so­le­ly in Java–we can then post-pro­cess in smal­ler bat­ches via the em­bed­ded Py­thon script.