Jython memory leak/​Out Of Memory problem

Note: This arti­cle has been unpublis­hed for quite some time. It’s main parts date back to Decem­ber 2007. The­re­fore, if some ver­sion num­ber seem to be out­da­ted — I am refer­ring to the state it had back those days.

We have a Java app­li­ca­tion with embed­ded Jython script­ing engine. The Jython scripts do mass com­pu­ta­ti­ons on data sets. So far, we had 3000 – 4000 data sets in one chunk maxi­mum. Now, a new cust­o­mer starts and will have 8000 and more data sets.

„No big deal,” I thought. And star­ted the test run one day before our cust­o­mer will have the whole thing run­ning on the pro­duc­tion sys­tem for the first time. Com­pu­ta­tion starts: 1000… 2000… 3000… 4000… 5000… bang „Out of memory. You should try to increase heap size”. The ser­ver app­li­ca­tion halts com­ple­tely and wit­hout any fur­ther warning.

I’m a bit sho­cked. The big pro­blems always arise at place where one would defi­ni­tely not expect them. I start cir­cum­ven­tion attempts: Split­ting the run into smal­ler chunks — does not work. Reinitia­li­zing the Jython environ­ment perio­di­cally — makes things worse. Repla­c­ing our rather out­da­ted (but other­wise func­tio­nal) Jython 2.1 with the then-​current Jython 2.2.1 — does not mat­ter. I do not seem to have a chance to cycle more than about 5200 times through the script before I catch an „Out of memory” situa­tion — or have to restart the whole ser­ver process.

Weird. What should I tell the cust­o­mer? „Well, you can­not run your com­pu­ta­ti­ons in one step. Start with the first half, then call us, we will restart the ser­ver, then do the second half.” ??? Not a really pro­fes­sio­nal way of doing things. Even more sur­pri­sing, loo­king at the memory situa­tion with Runtime.freeMemory() and fri­ends shows that there is no shor­tage of memory at all. Actually, when the app­li­ca­tion cras­hes, it has used not more than 600 MB out of 2048 MB heap space and more than 50 MB are mar­ked as „free”. This is not pre­ci­sely what I would sum­ma­rize as „out of memory”…

Finally, poking Google once more brings the solu­tion. I find an arti­cle about just a simi­lar pro­blem. For­t­u­na­tely, it has a solu­tion and even explains what’s going on: Jython has an inter­nal map­ping of PyXXX wrap­pers to Java objects. The default con­fi­gu­ra­tion uses nor­mal refe­ren­ces which makes these map­pings resis­tant to gar­bage collec­tion. Due to mecha­nisms I do not fully under­stand, this leads to enor­mous growth of the map­ping set and finally an out-​of-​memory-​situation with the inter­nal resource management.

For­t­u­na­tely, the solu­tion is as sim­ple as put­ting a


some­where in the code before the Jython sub­sys­tem is initia­li­zed. Then, the inter­nal table is built with weak refe­ren­ces and sud­denly, ever­y­thing runs smoothly. The 8000 data sets are no pro­blem any more and I can deli­ver the app­li­ca­tion as expec­ted. Lucky me.

There is only one ques­tion remai­ning: What kind of para­psy­cho­lo­gi­cal abi­li­ties are deve­l­o­pers expec­ted to have to find such a solu­tion wit­hout having the luck to find an arti­cle descri­bing this. And: Why the heck does Jython not use weak refe­ren­ces as default? I could not find any pro­blems or even speed penalties.

5 Antworten to “Jython memory leak/​Out Of Memory problem”

  • Thank you verry mutch for this hint, we are having the same issue, been loo­king for this solu­tion for a while ;-). Happy Coding everyone

  • F?rat KÜÇÜK

    The same pro­blem occu­red. I am sear­ching for a solution.

  • Inte­res­ting. We’re run­ning into what sounds like a very simi­lar pro­blem, and I tried app­ly­ing your fix, which didn’t seem to have any effect (and Jim’s com­ment implies that it wouldn’t have had any effect, anyway).

    We have to create & store many (~2 mil­lion Python objects) before we can pro­cess them, and we seem to be get­ting 10x ConcurrentHashMap$HashEntry objects — I can’t say for sure since I’m run­ning on OSX and I’m having trou­ble get­ting hprof to show me a deeper stack trace for its tra­ces. We’re pro­bably going to refac­tor some­what and store the object solely in Java – we can then post-​process in smal­ler bat­ches via the embed­ded Python script.

  • Inter­nal­Ta­bles is com­ple­tely gone in 2.5, so you should not be run­ning into pro­blems with uncollec­ta­ble objects now.

  • THX, memo­ri­zed for future use *smile*

Hinterlasse eine Nachricht

Mit dem Absenden des Kommentars willigen Sie ein, dass der angegebene Name, Ihre E-Mail-Adresse und die IP-Adresse Ihres Zugangs im Zusammenhang mit Ihrem Kommentar gespeichert werden. E-Mail- und IP-Adresse werden nicht veröffentlicht oder weitergegeben. Siehe Datenschutzhinweise.