Uploaded image for project: 'Fedora Repository Project'
  1. Fedora Repository Project
  2. FCREPO-1812

OOM when migrating many objects



    • Roadmap Theme:
    • Sprint:
      Sprint 2015 - 12, Sprint 2015 - 13, Sprint 2015 - 14


      We have started running migration tests using migration-utils, and are receiving an out of memory (OOM) error after migrating a few thousand objects, with heap sizes tested between 1-5 GB. The OOM is observed in the migration-utils JVM.

      For example, for one test we specified a max heap of 2 GB. After 25 min the JVM started running full GCs; after 40 min the JVM was spending most of its time in full GC, and an OOM was thrown at 50 min after 1533 Fedora 3 objects were migrated. The gc and migration-utils logs from this run are attached.

      Similar results were experienced with max heaps specified in the range of 1-5 GB, see table below. It appears that the heap usage grows linearly with the number of objects migrated to the max available, and some objects in the heap are never garbage collected.

      On one of these runs I took a snapshot of the heap and looked at it through jvisualvm. A fair amount of memory is consumed by char arrays and Strings, whose values appear to be triples. Perhaps these triples are not garbage collected?

      Max heap # objects Time to OOM
      1 GB 1262 25 mins
      2 GB 1533 50 mins
      4 GB 3145 1 hr 41 mins
      5 GB 4032 2 hr 8 mins

      migration-utils-4.3.1-SNAPSHOT, running on the same CentOS server as the destination Fedora.
      Source data is from Fedora 3.8.1 legacy-fs format.
      Target is Fedora 4.3, running with 5 GB max heap.

      Eventually we would like to use this utility to migrate two different Fedora repos, each with approx. 3.5M objects.



          Issue Links



              • Assignee:
                mdurbin Michael Durbin
                dshalvi Doron Shalvi
                Andrew Woods
              • Votes:
                0 Vote for this issue
                4 Start watching this issue


                • Created: