Uploaded image for project: 'Islandora'
  1. Islandora
  2. ISLANDORA-1736

Large multi-object updates overwhelm MySQL and break Solr index in multisites

    Details

      Description

      This problem cropped up when using the Find/Replace module, but according to discoverygarden seems like it would occur with any bulk update to the repository.

      After a minor update to the MODS datastreams on several hundred objects at once, Solr metadata disappeared from many objects in Browse mode - both within the set of modified objects and outside it.

      discoverygarden looked into the problem and gave the following diagnosis:

      We've narrowed the problem down to the MySQL database connections. Fedoragsearch (application that handles updating Solr) was set to multithreaded. We believe the multithreading was overwhelming the database during the full re-index and so the process was failing part way through. We've disabled the multithreading for now (and completed a re-index successfully) and may revisit the issue in the future if the amount of resources on your system changes.

      ... (later) ...

      The underlying issue is mysql can't handle bulk updates when all the Drupal filters connections are enabled in Fedora. The connections are required for Fedora to authenticate against Drupal; there is a connection for each multisite and Fedora tests each connections for every request it receives.

      We were hoping that disabling the Fedoragsearch multithreading would help, but it clearly still fails under mass updates.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              brandonw Brandon Weigel
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: