Uploaded image for project: 'Fedora Repository Project'
  1. Fedora Repository Project
  2. FCREPO-1192

REST API can not upload files greater than 2^31-1 in size

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: Fedora 3.5
    • Fix Version/s: Fedora 3.7
    • Component/s: legacy - Fedora
    • Labels:
      None
    • Environment:
      RHEL 6.x. Tomcat 7.x Fedora 3.5.

      Description

      I have a version of fedora 3.5 running that errors with

      javax.ws.rs.WebApplicationException: org.fcrepo.server.errors.StreamWriteException: Error writing temp stream
            at org.fcrepo.server.rest.BaseRestResource.handleException(BaseRestResource.java:172)
            at org.fcrepo.server.rest.DatastreamResource.addOrUpdateDatastream(DatastreamResource.java:583)


      when I write a file 2GB in size (1024*1024*2048) via the API:
      curl -H "Content-type:application/octet-stream" -X POST -u "name:pass" --upload-file 3g.data \
      http://host:8080/fedora/objects/scholarsphere:5425k9692/datastreams/blah?controlGroup=M

      But when I write a 2GB-1MB file (1024*1024*2047) via the API, it works without issue. Multipart uploads also do not work.

      Ben Armintor has been working with me on this and first identified https://jira.duraspace.org/browse/FCREPO-704 as a potential starting place. He has since followed up with:

        I've got a test verifying a bug in FCRepo 3.6 (and earlier) for content posted in the request body- I'll push a fix today (I think). There are two places in the REST library in which signed ints are being used got content-length in the kind of upload you have:
      https://github.com/fcrepo/fcrepo/blob/master/fcrepo-server/src/main/java/org/fcrepo/server/rest/RestUtil.java#L74
      (where it's implicit in the j2ee interface, so we just have to ignore it and parse the header)
      and at https://github.com/fcrepo/fcrepo/blob/master/fcrepo-server/src/main/java/org/fcrepo/server/rest/RestUtil.java#L63
      (our fault- that was clearly a place we missed when an earlier bug in this line was fixed 3 years ago).

        Attachments

        1. trace_curl.txt
          7 kB
          Jeff Minelli
        2. trace_fedora.txt
          8 kB
          Jeff Minelli

          Activity

            People

            • Assignee:
              barmintor Benjamin Armintor
              Reporter:
              jgm106@psu.edu Jeff Minelli
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: