Note: This artic­le has been unpu­blished for quite some time. It’s main parts date back to Decem­ber 2007. The­r­e­fo­re, if some ver­si­on num­ber seem to be out­da­ted – I am refer­ring to the sta­te it had back tho­se days.

We have a Java appli­ca­ti­on with embedded Jython scrip­ting engi­ne. The Jython scripts do mass com­pu­ta­ti­ons on data sets. So far, we had 3000 – 4000 data sets in one chunk maxi­mum. Now, a new cus­to­mer starts and will have 8000 and more data sets.

„No big deal,” I thought. And star­ted the test run one day befo­re our cus­to­mer will have the who­le thing run­ning on the pro­duc­tion sys­tem for the first time. Com­pu­ta­ti­on starts: 1000… 2000… 3000… 4000… 5000… bang „Out of memo­ry. You should try to increase heap size”. The ser­ver appli­ca­ti­on halts com­ple­te­ly and wit­hout any fur­ther warning.

I’m a bit sho­cked. The big pro­blems always ari­se at place whe­re one would defi­ni­te­ly not expect them. I start cir­cum­ven­ti­on attempts: Split­ting the run into smal­ler chunks – does not work. Reinitia­li­zing the Jython envi­ron­ment peri­odi­cal­ly – makes things worse. Repla­cing our rather out­da­ted (but other­wi­se func­tion­al) Jython 2.1 with the then-cur­rent Jython 2.2.1 – does not mat­ter. I do not seem to have a chan­ce to cycle more than about 5200 times through the script befo­re I catch an „Out of memo­ry” situa­ti­on – or have to restart the who­le ser­ver process.

Weird. What should I tell the cus­to­mer? „Well, you can­not run your com­pu­ta­ti­ons in one step. Start with the first half, then call us, we will restart the ser­ver, then do the second half.” ??? Not a real­ly pro­fes­sio­nal way of doing things. Even more sur­pri­sing, loo­king at the memo­ry situa­ti­on with Runtime.freeMemory() and fri­ends shows that the­re is no shorta­ge of memo­ry at all. Actual­ly, when the appli­ca­ti­on cra­s­hes, it has used not more than 600 MB out of 2048 MB heap space and more than 50 MB are mark­ed as „free”. This is not pre­cis­e­ly what I would sum­ma­ri­ze as „out of memory”…

Final­ly, poking Goog­le once more brings the solu­ti­on. I find an artic­le about just a simi­lar pro­blem. For­t­u­na­te­ly, it has a solu­ti­on and even explains what’s going on: Jython has an inter­nal map­ping of PyXXX wrap­pers to Java objects. The default con­fi­gu­ra­ti­on uses nor­mal refe­ren­ces which makes the­se map­pings resistant to gar­ba­ge coll­ec­tion. Due to mecha­nisms I do not ful­ly under­stand, this leads to enorm­ous growth of the map­ping set and final­ly an out-of-memo­ry-situa­ti­on with the inter­nal resour­ce management.

For­t­u­na­te­ly, the solu­ti­on is as simp­le as put­ting a

System.setProperty(„python.options.internalTablesImpl”,„weak”);

some­whe­re in the code befo­re the Jython sub­sys­tem is initia­li­zed. Then, the inter­nal table is built with weak refe­ren­ces and sud­den­ly, ever­y­thing runs smooth­ly. The 8000 data sets are no pro­blem any more and I can deli­ver the appli­ca­ti­on as expec­ted. Lucky me.

The­re is only one ques­ti­on remai­ning: What kind of para­psy­cho­lo­gi­cal abili­ties are deve­lo­pers expec­ted to have to find such a solu­ti­on wit­hout having the luck to find an artic­le describ­ing this. And: Why the heck does Jython not use weak refe­ren­ces as default? I could not find any pro­blems or even speed penalties.