Uploaded image for project: 'DSpace'
  1. DSpace
  2. DS-1059

Statistics utilities should be filters

    XMLWordPrintable

    Details

    • Type: New Feature
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Fixed
    • Fix Version/s: 3.0
    • Component/s: Solr
    • Labels:
      None
    • Attachments:
      3
    • Comments:
      1
    • Documentation Status:
      Complete or Committed

      Description

      Log files on large, busy sites may be enormous. This can make it difficult to find enough storage when (re)loading statistical cases from logs. Very large files also are very likely to be compressed by the sysadmin to save storage, which means they have to be decompressed before feeding them to the utilities, requiring even more temporary storage.

      One should be able to operate these utilities in a pipeline so that intermediate storage is eliminated. That is, they should be able to operate as filters: read standard input, write standard output.

      Attached patches (as they come) will add this optional behavior. If i or -o switches are not given, or are given the value "" or "", this means respectively standard input or standard output. Any other values will work as before: the value names a file.

      One can then do things like 'bunzip2 < logs/gigundo.log | bin/dspace stats-log-converter | bin/dspace stats-log-importer' without any additional storage required beyond what Solr will use.

        Attachments

          Activity

            People

            • Assignee:
              mwood Mark H. Wood
              Reporter:
              mwood Mark H. Wood
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: