Jython memory leak/Out Of Memory problem 5


Note: This arti­cle has been unpu­blished for qui­te some time. It’s main parts date back to Decem­ber 2007. The­re­fo­re, if some ver­si­on num­ber seem to be out­da­ted – I am refer­ring to the sta­te it had back tho­se days.

We have a Java app­li­ca­ti­on with embed­ded Jython script­ing engi­ne. The Jython scripts do mass com­pu­ta­ti­ons on data sets. So far, we had 3000 – 4000 data sets in one chunk maxi­mum. Now, a new custo­mer starts and will have 8000 and more data sets.

„No big deal,” I thought. And star­ted the test run one day befo­re our custo­mer will have the who­le thing run­ning on the pro­duc­tion sys­tem for the first time. Com­pu­ta­ti­on starts: 1000… 2000… 3000… 4000… 5000… bang „Out of memo­ry. You should try to increa­se heap size”. The ser­ver app­li­ca­ti­on halts com­ple­te­ly and without any fur­t­her warning.

I’m a bit sho­cked. The big pro­blems always ari­se at place whe­re one would defi­ni­te­ly not expect them. I start cir­cum­ven­ti­on attempts: Split­ting the run into smal­ler chunks – does not work. Reinitia­li­zing the Jython envi­ron­ment peri­o­di­cal­ly – makes things worse. Repla­cing our rather out­da­ted (but other­wi­se func­tio­nal) Jython 2.1 with the then-cur­rent Jython 2.2.1 – does not mat­ter. I do not seem to have a chan­ce to cycle more than about 5200 times through the script befo­re I catch an „Out of memo­ry” situa­ti­on – or have to restart the who­le ser­ver pro­cess.

Weird. What should I tell the custo­mer? „Well, you can­not run your com­pu­ta­ti­ons in one step. Start with the first half, then call us, we will restart the ser­ver, then do the second half.” ??? Not a real­ly pro­fes­sio­nal way of doing things. Even more sur­pri­sing, loo­king at the memo­ry situa­ti­on with Runtime.freeMemory() and fri­ends shows that the­re is no shor­ta­ge of memo­ry at all. Actual­ly, when the app­li­ca­ti­on cras­hes, it has used not more than 600 MB out of 2048 MB heap space and more than 50 MB are mar­ked as „free”. This is not pre­cise­ly what I would sum­ma­ri­ze as „out of memo­ry”…

Final­ly, poking Goog­le once more brings the solu­ti­on. I find an arti­cle about just a simi­lar pro­blem. For­tu­n­a­te­ly, it has a solu­ti­on and even exp­lains what’s going on: Jython has an inter­nal map­ping of PyXXX wrap­pers to Java objects. The default con­fi­gu­ra­ti­on uses nor­mal refe­ren­ces which makes the­se map­pings resistant to gar­ba­ge collec­tion. Due to mecha­nisms I do not ful­ly under­stand, this leads to enor­mous growth of the map­ping set and final­ly an out-of-memo­ry-situa­ti­on with the inter­nal resour­ce manage­ment.

For­tu­n­a­te­ly, the solu­ti­on is as simp­le as put­ting a

System.setProperty(„python.options.internalTablesImpl”,„weak”);

some­whe­re in the code befo­re the Jython sub­sys­tem is initia­li­zed. Then, the inter­nal table is built with weak refe­ren­ces and sud­den­ly, ever­y­thing runs smooth­ly. The 8000 data sets are no pro­blem any more and I can deli­ver the app­li­ca­ti­on as expec­ted. Lucky me.

The­re is only one ques­ti­on remai­ning: What kind of para­psy­cho­lo­gi­cal abi­li­ties are deve­l­o­pers expec­ted to have to find such a solu­ti­on without having the luck to find an arti­cle describing this. And: Why the heck does Jython not use weak refe­ren­ces as default? I could not find any pro­blems or even speed pen­al­ties.


Schreibe einen Kommentar

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert.

5 Gedanken zu “Jython memory leak/Out Of Memory problem

  • spacekadety

    Thank you ver­ry mutch for this hint, we are having the same issue, been loo­king for this solu­ti­on for a while ;-). Hap­py Coding ever­yo­ne

  • Denis Haskin

    Inte­res­ting. We’re run­ning into what sounds like a very simi­lar pro­blem, and I tried app­ly­ing your fix, which didn’t seem to have any effect (and Jim’s com­ment implies that it wouldn’t have had any effect, any­way).

    We have to crea­te & store many (~2 mil­li­on Python objects) befo­re we can pro­cess them, and we seem to be get­ting 10x ConcurrentHashMap$HashEntry objects – I can’t say for sure sin­ce I’m run­ning on OSX and I’m having trou­ble get­ting hprof to show me a deeper stack trace for its traces. We’re pro­bab­ly going to refac­tor some­what and store the object sole­ly in Java – we can then post-pro­cess in smal­ler bat­ches via the embed­ded Python script.