Uploaded image for project: 'Fedora Repository Project'
  1. Fedora Repository Project
  2. FCREPO-1302

Ingest degradation bulk load

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Closed
    • Priority: Critical
    • Resolution: Won't Fix
    • Affects Version/s: Fedora 4.0.0
    • Fix Version/s: Fedora 4.3.0
    • Component/s: f4-core
    • Labels:
    • Roadmap Theme:
      Scalability/Performance
    • Sprint:
      Sprint 2015 - 1, Sprint 2015 - 2, Sprint 2015 - 3, Sprint 2015 - 4, Sprint 2015 - 7

      Description

      It has been observed that when ingesting 50,000 resources with the attached mods.rdf file, ingest performance drops between the creation of 30,000 and 40,000 resources.
      The attached mods.rdf includes 13 blanknodes.

      The following script can be used to reproduce the issue:
      ===========
      curl -XPUT localhost:8080/fcrepo/rest/test/collection0

      time for x in

      {0..50000}

      ; do if [ $(( $x % 500 )) == 0 ]; then d=`date`; echo "$d — $x" ; fi; curl -s -XPOST --data-binary @mods.ttl -H"Content-Type: text/turtle"
      localhost:8080/fcrepo/rest/test/collection0 > /dev/null; done
      ===========

      Tests should be performed that create resources that do and do not include RDF bodies.
      Additionally, tests should be performed in repository hierarchies that are initially empty and in those that already contain thousands of resources.

      Once a scenario is found to exhibit performance degradation, repeat the test while profiling the application with YourKit:
      https://wiki.duraspace.org/display/FF/Profiling+Fedora

        Attachments

        1. dams.xml
          19 kB
        2. mods.ttl
          3 kB

          Issue Links

            Activity

              People

              • Assignee:
                unknown Unknown
                Reporter:
                awoods Andrew Woods
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: