DSpace
  1. DSpace
  2. DS-768

All XMLUI Error Pages respond with 200 OK, instead of 404 Not Found

    Details

    • Type: Bug Bug
    • Status: Closed Closed (View Workflow)
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.0, 1.6.1, 1.6.2, 1.7.0, 1.8.0
    • Fix Version/s: 1.8.0
    • Component/s: XMLUI
    • Labels:
      None
    • Attachments:
      6
    • Comments:
      58

      Description

      In DSpace 1.7.0 RC1, the XMLUI "Page Not Found" page responds with a 200 OK, rather than the necessary 404 Not Found error.

      For example:
      http://demo.dspace.org/xmlui/NOTAVALIDPATH

      I believe this used to function properly in 1.6.x, but it seems to be broken now.

      Obviously, we want this to return a 404 Not Found, in order to ensure that search engines do not index "Page Not Found" responses.

      Furthermore, it has been discovered that all other XMLUI Error pages (Cocoon Errors or Invalid Continuation errors) also respond with 200 OK. In addition, it seems this affects all 1.6.x versions of DSpace, as the same responses can be found on http://dspace.mit.edu (currently running 1.6.0 with patches), and http://researchspace.auckland.ac.nz (currently running 1.6.2 with patches).
      1. [DS-768]_add_ProcessingException_to_transformers
        6 kB
        Kim Shepherd
      2. [DS-768]_dspace-cocoon-servlet-service-impl-1_0_2.patch
        13 kB
        Kim Shepherd
      3. add_ResourceNotFoundException_to_xmlui_transformers.patch
        5 kB
        Kim Shepherd
      1. Screen shot 2011-08-17 at 2.20.40 PM.png
        74 kB
      2. Screenshot-Resource not found-Mozilla Firefox.png
        119 kB

        Issue Links

          Activity

          Hide
          Tim Donohue added a comment -
          This problem also seems to extend to all error screens in 1.7.0 XMLUI. Currently the "Invalid Continuation" error screen (e.g. see DS-613), and general Cocoon Error screens both also return 200 OK instead of 404 Not Found.

          I'm not sure what could have changed, as the error handling at the bottom of the main sitemap.xmap, specifically says to return a 404 response (see the 'status-code' value below):

          http://scm.dspace.org/svn/repo/dspace/trunk/dspace-xmlui/dspace-xmlui-webapp/src/main/webapp/sitemap.xmap

                                         <map:when test="invalid-continuation">
                                                  <map:generate type="exception"/>
                                                  <map:transform src="exception2html.xslt">
                                                          <map:parameter name="contextPath" value="{request:contextPath}"/>
                                                          <map:parameter name="pageTitle" value="Invalid Continuation"/>
                                                  </map:transform>
                                                  <map:serialize type="xhtml" status-code="404"/>
                                          </map:when>

          If anyone has an ideas on this, I'd appreciate suggestions. I'll keep digging around a bit to see what I can figure out.
          Show
          Tim Donohue added a comment - This problem also seems to extend to all error screens in 1.7.0 XMLUI. Currently the "Invalid Continuation" error screen (e.g. see DS-613 ), and general Cocoon Error screens both also return 200 OK instead of 404 Not Found. I'm not sure what could have changed, as the error handling at the bottom of the main sitemap.xmap, specifically says to return a 404 response (see the 'status-code' value below): http://scm.dspace.org/svn/repo/dspace/trunk/dspace-xmlui/dspace-xmlui-webapp/src/main/webapp/sitemap.xmap                                <map:when test="invalid-continuation">                                         <map:generate type="exception"/>                                         <map:transform src="exception2html.xslt">                                                 <map:parameter name="contextPath" value="{request:contextPath}"/>                                                 <map:parameter name="pageTitle" value="Invalid Continuation"/>                                         </map:transform>                                         <map:serialize type="xhtml" status-code="404"/>                                 </map:when> If anyone has an ideas on this, I'd appreciate suggestions. I'll keep digging around a bit to see what I can figure out.
          Hide
          Tim Donohue added a comment -
          Changed title and description above to detail that this invalid 200 OK response seems to affect all 1.6.x versions of DSpace, and is returned for *all* errors reported by XMLUI.
          Show
          Tim Donohue added a comment - Changed title and description above to detail that this invalid 200 OK response seems to affect all 1.6.x versions of DSpace, and is returned for *all* errors reported by XMLUI.
          Hide
          Tim Donohue added a comment -
          We seem to be experiencing this Cocoon Bug: https://issues.apache.org/jira/browse/COCOON-2218

          If I switch all 404 response codes (in main /src/main/webapps/sitemap.xmap) to instead return 500 (or any other response code), then everything works as planned. Only the 404 responses seem to be affected by this problem.

          This Cocoon bug seems to have been resolved in the current Cocoon 2.2 Trunk code, but was never officially released in a stable version of Cocoon.
          Show
          Tim Donohue added a comment - We seem to be experiencing this Cocoon Bug: https://issues.apache.org/jira/browse/COCOON-2218 If I switch all 404 response codes (in main /src/main/webapps/sitemap.xmap) to instead return 500 (or any other response code), then everything works as planned. Only the 404 responses seem to be affected by this problem. This Cocoon bug seems to have been resolved in the current Cocoon 2.2 Trunk code, but was never officially released in a stable version of Cocoon.
          Hide
          Kim Shepherd added a comment -
          Update from Cocoon 2218, in answer to Mark D's question about the revision of the patch in cocoon trunk:


          "Robin Wyles added a comment - 24/Mar/11 11:16
          Mark, I'm not entirely sure, but looking at the history I think it could've been revision 685544. Can't easily test this at the moment though..."
          Show
          Kim Shepherd added a comment - Update from Cocoon 2218, in answer to Mark D's question about the revision of the patch in cocoon trunk: "Robin Wyles added a comment - 24/Mar/11 11:16 Mark, I'm not entirely sure, but looking at the history I think it could've been revision 685544. Can't easily test this at the moment though..."
          Hide
          Tim Donohue added a comment -
          This issue was discussed in the DSpace Developers Mtg on April 6, 2011:

          [20:17] <tdonohue> Next up, is our much beloved DS-768.
          [20:17] <kompewter> [ https://jira.duraspace.org/browse/DS-768 ] - [#DS-768] All XMLUI Error Pages respond with 200 OK, instead of 404 Not Found - DuraSpace JIRA
          [20:17] <tdonohue> Ok. Ds-767 - assign to kshepherd
          [20:17] <richardrodgers> agreed, sounds like nice to have if not too expensive
          [20:17] <tdonohue> anyone do further research/investigation into Ds-768?
          [20:18] <kshepherd> mdiggory asked a cocoon guy about the revision the patch was committed
          [20:18] <kshepherd> so we could try and track it down
          [20:18] <kshepherd> i checked out the cocoon 2.2 source to try and track it down via commit comments, but had no luck
          [20:18] <tdonohue> +1 to tracking down this cocoon patch, if we can.
          [20:18] <kshepherd> there's a reply in COCOON-2218 to mark's questoin
          [20:18] <kshepherd> "Mark, I'm not entirely sure, but looking at the history I think it could've been revision 685544. Can't easily test this at the moment though..."
          [20:19] <kshepherd> so that gives us a possible lead
          [20:19] <kshepherd> i'll add to our comments
          [20:19] <tdonohue> sounds good. volunteers to test this lead out? (or is mdiggory planning on doing so?)
          [20:20] <kshepherd> well i know mdiggory and i have it on our radars
          [20:20] * tdonohue remembers mdiggory is out this week. so, he actually isn't here today.
          [20:20] <kshepherd> not sure if it needs assigning until we're closer to a fix..?
          [20:21] <tdonohue> ok. Ds-768: kshepherd will add updates. He and mdiggory have this on their radars. One or both will bring it back to our attention as help is necessary, etc
          Show
          Tim Donohue added a comment - This issue was discussed in the DSpace Developers Mtg on April 6, 2011: [20:17] <tdonohue> Next up, is our much beloved DS-768 . [20:17] <kompewter> [ https://jira.duraspace.org/browse/DS-768 ] - [# DS-768 ] All XMLUI Error Pages respond with 200 OK, instead of 404 Not Found - DuraSpace JIRA [20:17] <tdonohue> Ok. Ds-767 - assign to kshepherd [20:17] <richardrodgers> agreed, sounds like nice to have if not too expensive [20:17] <tdonohue> anyone do further research/investigation into Ds-768? [20:18] <kshepherd> mdiggory asked a cocoon guy about the revision the patch was committed [20:18] <kshepherd> so we could try and track it down [20:18] <kshepherd> i checked out the cocoon 2.2 source to try and track it down via commit comments, but had no luck [20:18] <tdonohue> +1 to tracking down this cocoon patch, if we can. [20:18] <kshepherd> there's a reply in COCOON-2218 to mark's questoin [20:18] <kshepherd> "Mark, I'm not entirely sure, but looking at the history I think it could've been revision 685544. Can't easily test this at the moment though..." [20:19] <kshepherd> so that gives us a possible lead [20:19] <kshepherd> i'll add to our comments [20:19] <tdonohue> sounds good. volunteers to test this lead out? (or is mdiggory planning on doing so?) [20:20] <kshepherd> well i know mdiggory and i have it on our radars [20:20] * tdonohue remembers mdiggory is out this week. so, he actually isn't here today. [20:20] <kshepherd> not sure if it needs assigning until we're closer to a fix..? [20:21] <tdonohue> ok. Ds-768: kshepherd will add updates. He and mdiggory have this on their radars. One or both will bring it back to our attention as help is necessary, etc
          Hide
          Kim Shepherd added a comment -
          I've attached an example of a fixed cocoon-servlet-service-impl (based on 1.0.0).
          This is returning proper 404, unlike the old 1.0.0 and dspace's 1.0.1 customisation.

          I'm 99% sure that it incorporates the fixes for DS-253 / COCOON-2217, as well as DS-768 / COCOON-2218, but I'd appreciate more eyes on it... the source for dspace-cocoon-servlet-service-impl-1.0.1 isn't available anywhere that I can find, but the patches listed in DS-253 certainly imply that I've applied the same fixes to 1.02.

          So I think this will fix up our response codes in XMLUI once and for all :)

          Would really appreciate testers, so I can be sure I'm not reintroducing old bugs.
          A quick and dirty test is just to drop this jar into [ your deployed xmlui ]/WEB-INF/lib and remove the old 1.0.1 version, clear tomcat cache, and restart.
          Show
          Kim Shepherd added a comment - I've attached an example of a fixed cocoon-servlet-service-impl (based on 1.0.0). This is returning proper 404, unlike the old 1.0.0 and dspace's 1.0.1 customisation. I'm 99% sure that it incorporates the fixes for DS-253 / COCOON-2217, as well as DS-768 / COCOON-2218, but I'd appreciate more eyes on it... the source for dspace-cocoon-servlet-service-impl-1.0.1 isn't available anywhere that I can find, but the patches listed in DS-253 certainly imply that I've applied the same fixes to 1.02. So I think this will fix up our response codes in XMLUI once and for all :) Would really appreciate testers, so I can be sure I'm not reintroducing old bugs. A quick and dirty test is just to drop this jar into [ your deployed xmlui ]/WEB-INF/lib and remove the old 1.0.1 version, clear tomcat cache, and restart.
          Hide
          Kim Shepherd added a comment -
          Attaching patch which can be applied to the cocoon-servlet-service-impl-1.0.0 source to create dspace-cocoon-servlet-service-impl-1.0.2

          Original cocoon source can be checked out here:
          https://svn.apache.org/repos/asf/cocoon/tags/cocoon-servlet-service/cocoon-servlet-service-impl/cocoon-servlet-service-impl-1.0.0
          Show
          Kim Shepherd added a comment - Attaching patch which can be applied to the cocoon-servlet-service-impl-1.0.0 source to create dspace-cocoon-servlet-service-impl-1.0.2 Original cocoon source can be checked out here: https://svn.apache.org/repos/asf/cocoon/tags/cocoon-servlet-service/cocoon-servlet-service-impl/cocoon-servlet-service-impl-1.0.0
          Hide
          Andrea Schweer added a comment -
          I tested this on 1.6.2 by dropping Kim's jar file into webapps/xmlui/WEB-INF/lib as suggested. I can confirm that this fixed the problem in my very limited test case:

          standard 1.6.2:

          $ wget http://localhost/xmlui/NOTVALID
          --2011-07-15 11:41:49-- http://localhost/xmlui/NOTVALID
          Resolving localhost... ::1, 127.0.0.1
          Connecting to localhost|::1|:80... failed: Connection refused.
          Connecting to localhost|127.0.0.1|:80... connected.
          HTTP request sent, awaiting response... 200 OK

          with Kim's jar file:

          $ wget http://localhost/xmlui/NOTVALIDTOO--2011-07-15 11:43:50-- http://localhost/xmlui/NOTVALIDTOO
          Resolving localhost... ::1, 127.0.0.1
          Connecting to localhost|::1|:80... failed: Connection refused.
          Connecting to localhost|127.0.0.1|:80... connected.
          HTTP request sent, awaiting response... 404 Not Found
          Show
          Andrea Schweer added a comment - I tested this on 1.6.2 by dropping Kim's jar file into webapps/xmlui/WEB-INF/lib as suggested. I can confirm that this fixed the problem in my very limited test case: standard 1.6.2: $ wget http://localhost/xmlui/NOTVALID --2011-07-15 11:41:49-- http://localhost/xmlui/NOTVALID Resolving localhost... ::1, 127.0.0.1 Connecting to localhost|::1|:80... failed: Connection refused. Connecting to localhost|127.0.0.1|:80... connected. HTTP request sent, awaiting response... 200 OK with Kim's jar file: $ wget http://localhost/xmlui/NOTVALIDTOO--2011-07-15 11:43:50-- http://localhost/xmlui/NOTVALIDTOO Resolving localhost... ::1, 127.0.0.1 Connecting to localhost|::1|:80... failed: Connection refused. Connecting to localhost|127.0.0.1|:80... connected. HTTP request sent, awaiting response... 404 Not Found
          Hide
          Tim Donohue added a comment -
          Also verified that this fix works for me (on Trunk code), by using FireFox 5 > Web Developer > Web Console.

          Currently, Trunk code responds with 200 for invalid paths:
          [11:25:38.147] GET http://localhost:8080/xmlui/NOTAPATH [HTTP/1.1 200 OK 94ms]

          However, if I just drop in Kim's JAR file and restart Tomcat, it responds 404 (as it should):
          [11:43:16.288] GET http://localhost:8080/xmlui/NOTAPATH [HTTP/1.1 404 Not Found 2953ms]

          Again, this was just a simple test, but it definitely does fix the HTTP Response issues. So, I'm satisfied and would like to see this committed to Trunk for 1.8 & 1.8 Testathon (assuming no other concerns exist from others)
          Show
          Tim Donohue added a comment - Also verified that this fix works for me (on Trunk code), by using FireFox 5 > Web Developer > Web Console. Currently, Trunk code responds with 200 for invalid paths: [11:25:38.147] GET http://localhost:8080/xmlui/NOTAPATH [HTTP/1.1 200 OK 94ms] However, if I just drop in Kim's JAR file and restart Tomcat, it responds 404 (as it should): [11:43:16.288] GET http://localhost:8080/xmlui/NOTAPATH [HTTP/1.1 404 Not Found 2953ms] Again, this was just a simple test, but it definitely does fix the HTTP Response issues. So, I'm satisfied and would like to see this committed to Trunk for 1.8 & 1.8 Testathon (assuming no other concerns exist from others)
          Hide
          Hardy Pottinger added a comment -
          In attempting to verify the new jar file, I dropped it in to the XMLUI lib folder, deleted the old dspace-cocoon jar, cleared the cache and restarted Tomcat. I used wget to test, following Andrea's lead, and, also got a 404 result when visiting http://localhost:8080/xmlui/NOTVALID... unfortunately, in a spirit of celebration, I tried the test again (just hit up arrow and return on the keyboard), and got a 200 result. o.O Pasting the commands I entered, and the output, below. But, it appears that cocoon is caching the 404 pages and then returning them as valid pages (with a header of 200 instead of 404). Which is very likely a problem with cocoon (and may be a problem we've had for a while, and just not known about). I've already notified Kim, as Jira was down when I found this issue this morning. He's looking into possible workarounds (he's already tried turning off caching, and the error header returns as 404, even on repeated tests). I'm noting the problem here, in case anyone else wants to chime in, and to give everyone a heads up that we're not out of the woods as far as cocoon caching issues are concerned.

          Pasting commands + output now:

          At first, things look good:

          [pottingerhj@lso-test2 ~]$ wget http://localhost:8080/xmlui/NOTVALID
          --2011-08-16 06:27:47-- http://localhost:8080/xmlui/NOTVALID
          Resolving localhost... 127.0.0.1
          Connecting to localhost|127.0.0.1|:8080... connected.
          HTTP request sent, awaiting response... 404 Not Found
          2011-08-16 06:27:47 ERROR 404: Not Found.

          But, then, trying to repeat the test (can't help, it's just an up-arrow +
          return away)... Oddness...

          [pottingerhj@lso-test2 ~]$ wget http://localhost:8080/xmlui/NOTVALID
          --2011-08-16 06:28:15-- http://localhost:8080/xmlui/NOTVALID
          Resolving localhost... 127.0.0.1
          Connecting to localhost|127.0.0.1|:8080... connected.
          HTTP request sent, awaiting response... 200 OK
          Length: 7134 (7.0K) [text/html]
          Saving to: `NOTVALID.2

          For grins, I tried to see if this was repeatable. It looks it:

          [pottingerhj@lso-test2 ~]$ wget http://localhost:8080/xmlui/REALLYNOTVALID
          --2011-08-16 06:39:51-- http://localhost:8080/xmlui/REALLYNOTVALID
          Resolving localhost... 127.0.0.1
          Connecting to localhost|127.0.0.1|:8080... connected.
          HTTP request sent, awaiting response... 404 Not Found
          2011-08-16 06:39:51 ERROR 404: Not Found.

          [pottingerhj@lso-test2 ~]$ wget http://localhost:8080/xmlui/REALLYNOTVALID
          --2011-08-16 06:39:56-- http://localhost:8080/xmlui/REALLYNOTVALID
          Resolving localhost... 127.0.0.1
          Connecting to localhost|127.0.0.1|:8080... connected.
          HTTP request sent, awaiting response... 200 OK
          Length: 7134 (7.0K) [text/html]
          Saving to: `REALLYNOTVALID'

          100%[======================================================================
          ===========================================================================
          =================================================>] 7,134 --.-K/s
          in 0s

          2011-08-16 06:39:56 (548 MB/s) - `REALLYNOTVALID' saved [7134/7134]

          [pottingerhj@lso-test2 ~]$ wget
          http://localhost:8080/xmlui/REALLYREALLYNOTVALID
          --2011-08-16 06:40:10-- http://localhost:8080/xmlui/REALLYREALLYNOTVALID
          Resolving localhost... 127.0.0.1
          Connecting to localhost|127.0.0.1|:8080... connected.
          HTTP request sent, awaiting response... 404 Not Found
          2011-08-16 06:40:11 ERROR 404: Not Found.

          [pottingerhj@lso-test2 ~]$ wget
          http://localhost:8080/xmlui/REALLYREALLYNOTVALID
          --2011-08-16 06:40:14-- http://localhost:8080/xmlui/REALLYREALLYNOTVALID
          Resolving localhost... 127.0.0.1
          Connecting to localhost|127.0.0.1|:8080... connected.
          HTTP request sent, awaiting response... 200 OK
          Length: 7134 (7.0K) [text/html]
          Saving to: `REALLYREALLYNOTVALID'

          100%[======================================================================
          ===========================================================================
          =================================================>] 7,134 --.-K/s
          in 0s

          2011-08-16 06:40:14 (534 MB/s) - `REALLYREALLYNOTVALID' saved [7134/7134]
          Show
          Hardy Pottinger added a comment - In attempting to verify the new jar file, I dropped it in to the XMLUI lib folder, deleted the old dspace-cocoon jar, cleared the cache and restarted Tomcat. I used wget to test, following Andrea's lead, and, also got a 404 result when visiting http://localhost:8080/xmlui/NOTVALID ... unfortunately, in a spirit of celebration, I tried the test again (just hit up arrow and return on the keyboard), and got a 200 result. o.O Pasting the commands I entered, and the output, below. But, it appears that cocoon is caching the 404 pages and then returning them as valid pages (with a header of 200 instead of 404). Which is very likely a problem with cocoon (and may be a problem we've had for a while, and just not known about). I've already notified Kim, as Jira was down when I found this issue this morning. He's looking into possible workarounds (he's already tried turning off caching, and the error header returns as 404, even on repeated tests). I'm noting the problem here, in case anyone else wants to chime in, and to give everyone a heads up that we're not out of the woods as far as cocoon caching issues are concerned. Pasting commands + output now: At first, things look good: [ pottingerhj@lso-test2 ~]$ wget http://localhost:8080/xmlui/NOTVALID --2011-08-16 06:27:47-- http://localhost:8080/xmlui/NOTVALID Resolving localhost... 127.0.0.1 Connecting to localhost|127.0.0.1|:8080... connected. HTTP request sent, awaiting response... 404 Not Found 2011-08-16 06:27:47 ERROR 404: Not Found. But, then, trying to repeat the test (can't help, it's just an up-arrow + return away)... Oddness... [ pottingerhj@lso-test2 ~]$ wget http://localhost:8080/xmlui/NOTVALID --2011-08-16 06:28:15-- http://localhost:8080/xmlui/NOTVALID Resolving localhost... 127.0.0.1 Connecting to localhost|127.0.0.1|:8080... connected. HTTP request sent, awaiting response... 200 OK Length: 7134 (7.0K) [text/html] Saving to: `NOTVALID.2 For grins, I tried to see if this was repeatable. It looks it: [ pottingerhj@lso-test2 ~]$ wget http://localhost:8080/xmlui/REALLYNOTVALID --2011-08-16 06:39:51-- http://localhost:8080/xmlui/REALLYNOTVALID Resolving localhost... 127.0.0.1 Connecting to localhost|127.0.0.1|:8080... connected. HTTP request sent, awaiting response... 404 Not Found 2011-08-16 06:39:51 ERROR 404: Not Found. [ pottingerhj@lso-test2 ~]$ wget http://localhost:8080/xmlui/REALLYNOTVALID --2011-08-16 06:39:56-- http://localhost:8080/xmlui/REALLYNOTVALID Resolving localhost... 127.0.0.1 Connecting to localhost|127.0.0.1|:8080... connected. HTTP request sent, awaiting response... 200 OK Length: 7134 (7.0K) [text/html] Saving to: `REALLYNOTVALID' 100%[====================================================================== =========================================================================== =================================================>] 7,134 --.-K/s in 0s 2011-08-16 06:39:56 (548 MB/s) - `REALLYNOTVALID' saved [7134/7134] [ pottingerhj@lso-test2 ~]$ wget http://localhost:8080/xmlui/REALLYREALLYNOTVALID --2011-08-16 06:40:10-- http://localhost:8080/xmlui/REALLYREALLYNOTVALID Resolving localhost... 127.0.0.1 Connecting to localhost|127.0.0.1|:8080... connected. HTTP request sent, awaiting response... 404 Not Found 2011-08-16 06:40:11 ERROR 404: Not Found. [ pottingerhj@lso-test2 ~]$ wget http://localhost:8080/xmlui/REALLYREALLYNOTVALID --2011-08-16 06:40:14-- http://localhost:8080/xmlui/REALLYREALLYNOTVALID Resolving localhost... 127.0.0.1 Connecting to localhost|127.0.0.1|:8080... connected. HTTP request sent, awaiting response... 200 OK Length: 7134 (7.0K) [text/html] Saving to: `REALLYREALLYNOTVALID' 100%[====================================================================== =========================================================================== =================================================>] 7,134 --.-K/s in 0s 2011-08-16 06:40:14 (534 MB/s) - `REALLYREALLYNOTVALID' saved [7134/7134]
          Hide
          Hardy Pottinger added a comment -
          I neglected to mention, but in case it helps, we're running DSpace 1.7.2 in dev and production, only tested this on our development server.
          Show
          Hardy Pottinger added a comment - I neglected to mention, but in case it helps, we're running DSpace 1.7.2 in dev and production, only tested this on our development server.
          Hide
          Tim Donohue added a comment -
          Ugh. I just verified this is the same thing that happens on Trunk code. :(

          With dspace-cocoon-servlet-service-impl-1.0.1.jar (Old JAR), I get:

          [14:45:13.278] GET http://localhost:8080/xmlui18/NOTVALID [HTTP/1.1 200 OK 126ms]

          If I then replace it with dspace-cocoon-servlet-service-impl-1.0.2.jar (Kim's new JAR), visiting the *SAME* bad URL still gives a 200:

          [14:47:33.111] GET http://localhost:8080/xmlui18/NOTVALID [HTTP/1.1 200 OK 78ms]

          But, if I visit a *NEW* bad URL, I get the proper 404:

          [14:47:47.049] GET http://localhost:8080/xmlui18/NOTAVALIDURL [HTTP/1.1 404 Not Found 156ms]

          Unfortunately, refreshing the page & visiting that same bad URL a second time, also gives a 200:

          [14:47:53.771] GET http://localhost:8080/xmlui18/NOTAVALIDURL [HTTP/1.1 200 OK 53ms]

          If I clear the Cocoon cache (from XMLUI Control Panel), each of the above bad URLs will again throw a 404 on the first access, and then a 200 thereafter. So, it's definitely a Cocoon caching issue. Though, maybe we can figure out a way to tell Cocoon not to cache these pages?

          Good find, Hardy.
          Show
          Tim Donohue added a comment - Ugh. I just verified this is the same thing that happens on Trunk code. :( With dspace-cocoon-servlet-service-impl-1.0.1.jar (Old JAR), I get: [14:45:13.278] GET http://localhost:8080/xmlui18/NOTVALID [HTTP/1.1 200 OK 126ms] If I then replace it with dspace-cocoon-servlet-service-impl-1.0.2.jar (Kim's new JAR), visiting the *SAME* bad URL still gives a 200: [14:47:33.111] GET http://localhost:8080/xmlui18/NOTVALID [HTTP/1.1 200 OK 78ms] But, if I visit a *NEW* bad URL, I get the proper 404: [14:47:47.049] GET http://localhost:8080/xmlui18/NOTAVALIDURL [HTTP/1.1 404 Not Found 156ms] Unfortunately, refreshing the page & visiting that same bad URL a second time, also gives a 200: [14:47:53.771] GET http://localhost:8080/xmlui18/NOTAVALIDURL [HTTP/1.1 200 OK 53ms] If I clear the Cocoon cache (from XMLUI Control Panel), each of the above bad URLs will again throw a 404 on the first access, and then a 200 thereafter. So, it's definitely a Cocoon caching issue. Though, maybe we can figure out a way to tell Cocoon not to cache these pages? Good find, Hardy.
          Hide
          Hardy Pottinger added a comment -
          I think caching 404 pages is actually not a bad idea, helps with performance... Cocoon should just return the correct status header for a cached 404 page.
          Show
          Hardy Pottinger added a comment - I think caching 404 pages is actually not a bad idea, helps with performance... Cocoon should just return the correct status header for a cached 404 page.
          Hide
          Hardy Pottinger added a comment -
          It's possible that COCOON-1354 (https://issues.apache.org/jira/browse/COCOON-1354) might be at work here, since the sitemap for xmlui does not use a number for the status, but appears to rely on a variable/static, so, perhaps another patch to dspace-cocoon is in order, or else a tweak to the sitemap to use 404?
          Show
          Hardy Pottinger added a comment - It's possible that COCOON-1354 ( https://issues.apache.org/jira/browse/COCOON-1354 ) might be at work here, since the sitemap for xmlui does not use a number for the status, but appears to rely on a variable/static, so, perhaps another patch to dspace-cocoon is in order, or else a tweak to the sitemap to use 404?
          Hide
          Hardy Pottinger added a comment -
          whoops, sorry, was looking at some example code, the sitemap for xmlui passes an actual number, <map:serialize type="xhtml" status-code="404"/> back to the drawing board...
          Show
          Hardy Pottinger added a comment - whoops, sorry, was looking at some example code, the sitemap for xmlui passes an actual number, <map:serialize type="xhtml" status-code="404"/> back to the drawing board...
          Hide
          Tim Donohue added a comment -
          Hmmm.. Actually, I think there's more at play here.

          At least on my local machine, it seems like Cocoon is not obeying the passed in 'status-code' values in the sitemap.xmap. I just changed *all* of the error "status-code" values to 500 in my /xmlui/sitemap.xmap, cleared both Cocoon & Tomcat's cache, restarted Tomcat, and then tried again.

          I still get the same behavior -- I still see a 404 error followed by a 200 (if I refresh the page). That's a bit odd, as now it should be throwing 500 errors instead.

          Maybe I'm just going crazy :)
          Show
          Tim Donohue added a comment - Hmmm.. Actually, I think there's more at play here. At least on my local machine, it seems like Cocoon is not obeying the passed in 'status-code' values in the sitemap.xmap. I just changed *all* of the error "status-code" values to 500 in my /xmlui/sitemap.xmap, cleared both Cocoon & Tomcat's cache, restarted Tomcat, and then tried again. I still get the same behavior -- I still see a 404 error followed by a 200 (if I refresh the page). That's a bit odd, as now it should be throwing 500 errors instead. Maybe I'm just going crazy :)
          Hide
          Kim Shepherd added a comment - - edited
          Good catch Hardy, and thanks for testing, Tim.
          I've verified the problem myself, and if I disable caching in Cocoon (set the default pipe to "noncaching"), the problem disappears.

          I'll keep playing with sitemap and caching, and I'll see what I come up with... any help is appreciated :)
          Show
          Kim Shepherd added a comment - - edited Good catch Hardy, and thanks for testing, Tim. I've verified the problem myself, and if I disable caching in Cocoon (set the default pipe to "noncaching"), the problem disappears. I'll keep playing with sitemap and caching, and I'll see what I come up with... any help is appreciated :)
          Hide
          Kim Shepherd added a comment -
          I've been testing out what Tim tried as well, and in cases where we actually throw a Cocoon exception (eg. try a /browse?type=invalidbrowseindex), we can alter the status code thrown in sitemap.xmap. For example, I know an exception is thrown on invalid browse index (it's *supposed* to be a RESOURCE_NOT_FOUND, but some stuff left over form DS-754 means it's throwing a NPE instead), so I changed the default exception handling to throw a random status code (eg. 777), and it does.

          Speculation: Part of the problem could be that when a resource is missing or a path is invalid, we're not throwing the right kind of Cocoon exception, and so it's not matching <map:handle-errors>...

          Here's an example of what PageNotFoundTransformer does:

          HttpServletResponse response = (HttpServletResponse)objectModel.get(HttpEnvironment.HTTP_RESPONSE_OBJECT);
          response.setStatus(HttpServletResponse.SC_NOT_FOUND);

          Rather than the "typical" choice of throwing a Cocoon ResourceNotFoundException.
          Show
          Kim Shepherd added a comment - I've been testing out what Tim tried as well, and in cases where we actually throw a Cocoon exception (eg. try a /browse?type=invalidbrowseindex), we can alter the status code thrown in sitemap.xmap. For example, I know an exception is thrown on invalid browse index (it's *supposed* to be a RESOURCE_NOT_FOUND, but some stuff left over form DS-754 means it's throwing a NPE instead), so I changed the default exception handling to throw a random status code (eg. 777), and it does. Speculation: Part of the problem could be that when a resource is missing or a path is invalid, we're not throwing the right kind of Cocoon exception, and so it's not matching <map:handle-errors>... Here's an example of what PageNotFoundTransformer does: HttpServletResponse response = (HttpServletResponse)objectModel.get(HttpEnvironment.HTTP_RESPONSE_OBJECT); response.setStatus(HttpServletResponse.SC_NOT_FOUND); Rather than the "typical" choice of throwing a Cocoon ResourceNotFoundException.
          Hide
          Kim Shepherd added a comment -
          Further to this, I can confirm that throwing a ResourceNotFoundException results in a good status code for all requests, but, of course, a generic Tomcat error page rather than our nice themed 404. I'll see how easy it is to wire that in via sitemap, and if it's fairly trivial, perhaps we should rethink PageNotFoundTransformer, etc.
          Show
          Kim Shepherd added a comment - Further to this, I can confirm that throwing a ResourceNotFoundException results in a good status code for all requests, but, of course, a generic Tomcat error page rather than our nice themed 404. I'll see how easy it is to wire that in via sitemap, and if it's fairly trivial, perhaps we should rethink PageNotFoundTransformer, etc.
          Hide
          Kim Shepherd added a comment -
          More progress.. I'm now displaying 404s using the exception2html.xslt page, not a generic Tomcat 404. Screenshot attached.
          So hopefully the only thing left to do is discuss the merits of throwing ResourceNotFound vs. manually setting http status in transformers, and how to theme our exceptions a bit better.

          (though personally, I'd rather have an ugly error page throwing a true 404 status code than a nice-looking "Page not found" page with a 200 OK or 304 NOT MODIFIED in its headers)
           
          Show
          Kim Shepherd added a comment - More progress.. I'm now displaying 404s using the exception2html.xslt page, not a generic Tomcat 404. Screenshot attached. So hopefully the only thing left to do is discuss the merits of throwing ResourceNotFound vs. manually setting http status in transformers, and how to theme our exceptions a bit better. (though personally, I'd rather have an ugly error page throwing a true 404 status code than a nice-looking "Page not found" page with a 200 OK or 304 NOT MODIFIED in its headers)  
          Hide
          Hardy Pottinger added a comment -
          I thought you said it was ugly? All I see is awesome. :-)
          Show
          Hardy Pottinger added a comment - I thought you said it was ugly? All I see is awesome. :-)
          Hide
          Kim Shepherd added a comment -
          Attached quick'n'dirty patch against trunk to show what I've been trying. You may still need to drop in my modified dspace-cocoon-servlet-service-impl as well.
          Show
          Kim Shepherd added a comment - Attached quick'n'dirty patch against trunk to show what I've been trying. You may still need to drop in my modified dspace-cocoon-servlet-service-impl as well.
          Hide
          Tim Donohue added a comment -
          Kim, so this will need delaying to post-1.8.0 (I noticed you updated the 'fix version' listed above)?

          Is there any other help we can provide in the coming weeks that would potentially allow us to get this resolved in 1.8.0? I know this is obviously a bug that affects a lot of folks, so I wanted to see if there's anything I can do to help out. If you feel this is just "too big of a change" (or potentially could cause a snowball of necessary changes), then obviously we may just need to wait until post 1.8.0.

          Just thought I'd ask if there's any way that someone else can help speed this along.
           
          Show
          Tim Donohue added a comment - Kim, so this will need delaying to post-1.8.0 (I noticed you updated the 'fix version' listed above)? Is there any other help we can provide in the coming weeks that would potentially allow us to get this resolved in 1.8.0? I know this is obviously a bug that affects a lot of folks, so I wanted to see if there's anything I can do to help out. If you feel this is just "too big of a change" (or potentially could cause a snowball of necessary changes), then obviously we may just need to wait until post 1.8.0. Just thought I'd ask if there's any way that someone else can help speed this along.  
          Hide
          Kim Shepherd added a comment -
          I don't feel we have quite enough testing / feedback under our belts... I know there were other bugs that the previous dspace-cocoon patch fixed, and I can't be 100% sure I've captured all those same fixes in my new patch. I'm also a wee bit nervous about the mvn release to get the new dspace-servlet artifact into our maven repo.... but I'm keeping a close eye on this so if I get some more thumbs up from testers in the next few days/week, I'll go ahead and try a release :)
          Show
          Kim Shepherd added a comment - I don't feel we have quite enough testing / feedback under our belts... I know there were other bugs that the previous dspace-cocoon patch fixed, and I can't be 100% sure I've captured all those same fixes in my new patch. I'm also a wee bit nervous about the mvn release to get the new dspace-servlet artifact into our maven repo.... but I'm keeping a close eye on this so if I get some more thumbs up from testers in the next few days/week, I'll go ahead and try a release :)
          Hide
          Hardy Pottinger added a comment -
          I intend to test the new "non-fancy-error-pages" patch tomorrow morning, it's high on my list of things to do. Would be awesome to have this running for the testathon.
          Show
          Hardy Pottinger added a comment - I intend to test the new "non-fancy-error-pages" patch tomorrow morning, it's high on my list of things to do. Would be awesome to have this running for the testathon.
          Hide
          Mark Diggory added a comment -
          tdonohue: Ok, I specifically wanted to ask to see if anyone has an ideas/time to spend on DS-768. From my understanding, kshepherd has gotten decently far, but he has rescheduled it for post-1.8.0 cause he needs feedback & help
          [8:06pm] tdonohue: https://jira.duraspace.org/browse/DS-768
          [8:06pm] keithgilbertson joined the chat room.
          [8:06pm] tdonohue: it'd be good to finally get this fixed, if anyone can spend some time in the coming weeks. It's annoying that we cannot get XMLUI to throw a proper 404
          [8:07pm] hpottinger: I'm happy to run that patch on my testathon instance, someone talk me out of it
          [8:08pm] aschweer: hpottinger: I think it sounds like a good idea
          [8:08pm] tdonohue: go for it!
          [8:08pm] • mhwood tries unsuccessfully to talk him out of it
          [8:08pm] aschweer: to me it sounds like kshepherd is worried that his fix breaks other things (because of cocoon patches), and I guess we won't find out unless we run it
          [8:09pm] tdonohue: yea, I think that's right aschweer. We mainly need more testing / testers / eyes on what kshepherd has proposed
          [8:10pm] mdiggory: Perhaps I should take a look at this, I did recently rip apart cocoon servlet service once again
          [8:10pm] tdonohue: ok. sounds like we can test it out on hpottinger's test instance and see how it goes.
          [8:10pm] tdonohue: mdiggory: it'd be great if you could
          [8:10pm] hpottinger: will try to have it patched and running by 10pm tonight
          [8:11pm] tdonohue: mainly, I want to avoid having yet another release where our XMLUI just always says "200 OK"
          [8:11pm] mdiggory: We did plan to drop the code here
          [8:11pm] mdiggory: http://scm.dspace.org/svn/repo/modules/dspace-cocoon-servlet-service-impl/
          [8:12pm] robint: Looks like its still just attached to the Jira issue
          [8:13pm] mdiggory: I beleive what we should do is dump the cocoon source under modules, commit it, patch it, release a SNAPSHOT, have testathon instances test building with the snapshot in place
          [8:14pm] sandsfish joined the chat room.
          [8:15pm] mdiggory: We might be better served to throw org/apache/cocoon/ProcessingException.html rather than ResourceNotFoundException, it would allow us to trap other conditions later as well.
          [8:15pm] mdiggory: Direct Known Subclasses:
          [8:15pm] mdiggory: ConnectionResetException, FormsException, InvalidContinuationException, ResourceNotFoundException
          [8:15pm] tdonohue: i'm fine with that direction, as long as we get some more eyes on what kshepherd proposes first. I just don't want to post it to demo.dspace.org until we have some idea that it should 'work'
          [8:17pm] tdonohue: mdiggory, do you want to bring your ideas to DS-768 comments, or to dspace-devel? I think this sounds reasonable, just don't want to take up all the meeting on just this one issue (we've got many more that are still open as well)
          [8:17pm] hpottinger: looks like this is the patch I need to try: https://jira.duraspace.org/secure/attachment/11922/add_ResourceNotFoundException_to_xmlui_transformers.patch but willing to try something else, as long as I have a patch to apply
          [8:17pm] mdiggory: continue on, I'll comment in the ticket and start progress.
          Show
          Mark Diggory added a comment - tdonohue: Ok, I specifically wanted to ask to see if anyone has an ideas/time to spend on DS-768 . From my understanding, kshepherd has gotten decently far, but he has rescheduled it for post-1.8.0 cause he needs feedback & help [8:06pm] tdonohue: https://jira.duraspace.org/browse/DS-768 [8:06pm] keithgilbertson joined the chat room. [8:06pm] tdonohue: it'd be good to finally get this fixed, if anyone can spend some time in the coming weeks. It's annoying that we cannot get XMLUI to throw a proper 404 [8:07pm] hpottinger: I'm happy to run that patch on my testathon instance, someone talk me out of it [8:08pm] aschweer: hpottinger: I think it sounds like a good idea [8:08pm] tdonohue: go for it! [8:08pm] • mhwood tries unsuccessfully to talk him out of it [8:08pm] aschweer: to me it sounds like kshepherd is worried that his fix breaks other things (because of cocoon patches), and I guess we won't find out unless we run it [8:09pm] tdonohue: yea, I think that's right aschweer. We mainly need more testing / testers / eyes on what kshepherd has proposed [8:10pm] mdiggory: Perhaps I should take a look at this, I did recently rip apart cocoon servlet service once again [8:10pm] tdonohue: ok. sounds like we can test it out on hpottinger's test instance and see how it goes. [8:10pm] tdonohue: mdiggory: it'd be great if you could [8:10pm] hpottinger: will try to have it patched and running by 10pm tonight [8:11pm] tdonohue: mainly, I want to avoid having yet another release where our XMLUI just always says "200 OK" [8:11pm] mdiggory: We did plan to drop the code here [8:11pm] mdiggory: http://scm.dspace.org/svn/repo/modules/dspace-cocoon-servlet-service-impl/ [8:12pm] robint: Looks like its still just attached to the Jira issue [8:13pm] mdiggory: I beleive what we should do is dump the cocoon source under modules, commit it, patch it, release a SNAPSHOT, have testathon instances test building with the snapshot in place [8:14pm] sandsfish joined the chat room. [8:15pm] mdiggory: We might be better served to throw org/apache/cocoon/ProcessingException.html rather than ResourceNotFoundException, it would allow us to trap other conditions later as well. [8:15pm] mdiggory: Direct Known Subclasses: [8:15pm] mdiggory: ConnectionResetException, FormsException, InvalidContinuationException, ResourceNotFoundException [8:15pm] tdonohue: i'm fine with that direction, as long as we get some more eyes on what kshepherd proposes first. I just don't want to post it to demo.dspace.org until we have some idea that it should 'work' [8:17pm] tdonohue: mdiggory, do you want to bring your ideas to DS-768 comments, or to dspace-devel? I think this sounds reasonable, just don't want to take up all the meeting on just this one issue (we've got many more that are still open as well) [8:17pm] hpottinger: looks like this is the patch I need to try: https://jira.duraspace.org/secure/attachment/11922/add_ResourceNotFoundException_to_xmlui_transformers.patch but willing to try something else, as long as I have a patch to apply [8:17pm] mdiggory: continue on, I'll comment in the ticket and start progress.
          Hide
          Mark Diggory added a comment -
          This is looking good, we need to get the fixes under version control and I will initiate this effort.
          Show
          Mark Diggory added a comment - This is looking good, we need to get the fixes under version control and I will initiate this effort.
          Hide
          Kim Shepherd added a comment -
          I've committed the source for my modified dspace-cocoon-servlet-service-impl to: http://scm.dspace.org/svn/repo/modules/dspace-cocoon-servlet-service-impl/trunk/
          Mark D and I will work on releasing this as a maven artifact (v1.01 probably?) tomorrow
          The patch that needs testing is https://jira.duraspace.org/secure/attachment/11922/add_ResourceNotFoundException_to_xmlui_transformers.patch, and this is something that would need to be committed to DSpace trunk.
          Show
          Kim Shepherd added a comment - I've committed the source for my modified dspace-cocoon-servlet-service-impl to: http://scm.dspace.org/svn/repo/modules/dspace-cocoon-servlet-service-impl/trunk/ Mark D and I will work on releasing this as a maven artifact (v1.01 probably?) tomorrow The patch that needs testing is https://jira.duraspace.org/secure/attachment/11922/add_ResourceNotFoundException_to_xmlui_transformers.patch, and this is something that would need to be committed to DSpace trunk.
          Hide
          Tim Donohue added a comment -
          This sounds as though this bug fix is "on schedule" again for 1.8.0. So, I'm marking it as "fix for" 1.8.0 again.
          Show
          Tim Donohue added a comment - This sounds as though this bug fix is "on schedule" again for 1.8.0. So, I'm marking it as "fix for" 1.8.0 again.
          Hide
          Kim Shepherd added a comment -
          Updated patch attached, to add ProcessingException to transformers and throw ResourceNotFoundException from PageNotFoundTransformer.
          POM has not been touched yet since we need to release Cocoon Servlet Service Impl 1.0.2.
          (and I think it's worthwhile testing without it, since the two solutions are effectively separate).

          The string passed with the Exception is English by default, as we can't pass Messages via Exceptions, but we can still internationalise the message in the XMLUI template for Exceptions. Also, we should probably alter these templates to exclude stack traces from simple errors like 404s.
          Show
          Kim Shepherd added a comment - Updated patch attached, to add ProcessingException to transformers and throw ResourceNotFoundException from PageNotFoundTransformer. POM has not been touched yet since we need to release Cocoon Servlet Service Impl 1.0.2. (and I think it's worthwhile testing without it, since the two solutions are effectively separate). The string passed with the Exception is English by default, as we can't pass Messages via Exceptions, but we can still internationalise the message in the XMLUI template for Exceptions. Also, we should probably alter these templates to exclude stack traces from simple errors like 404s.
          Hide
          Kim Shepherd added a comment -
          Also, I'm trying to commit, but can't auth to DSpace SCM for some reason, will keep trying.
          Show
          Kim Shepherd added a comment - Also, I'm trying to commit, but can't auth to DSpace SCM for some reason, will keep trying.
          Hide
          Mark Diggory added a comment -
          Hi guys, I tried it last week and am trying again as well.
          Show
          Mark Diggory added a comment - Hi guys, I tried it last week and am trying again as well.
          Hide
          Mark Diggory added a comment -
          Ok, sorry for all the commit noise, but I've completed quite a bit of surgery on the pom and go it to release to sonatype finally without complaints. Please wait for it to show up and test it out on your end.

          Mark
          Show
          Mark Diggory added a comment - Ok, sorry for all the commit noise, but I've completed quite a bit of surgery on the pom and go it to release to sonatype finally without complaints. Please wait for it to show up and test it out on your end. Mark
          Hide
          Kim Shepherd added a comment -
          Thanks Mark.
          I've committed the XMLUI patch to trunk now. No changes to pom yet.
          Show
          Kim Shepherd added a comment - Thanks Mark. I've committed the XMLUI patch to trunk now. No changes to pom yet.
          Hide
          Robin Taylor added a comment -
          Hi Kim,

          I just checked out the latest trunk and updated the dependency for dspace-cocoon-servlet-service-impl to be version 1.0.2. Unfortunately when I tested it I still get a status code of 200, see attached screenshot.

          Cheers, Robin.
          Show
          Robin Taylor added a comment - Hi Kim, I just checked out the latest trunk and updated the dependency for dspace-cocoon-servlet-service-impl to be version 1.0.2. Unfortunately when I tested it I still get a status code of 200, see attached screenshot. Cheers, Robin.
          Hide
          Robin Taylor added a comment -
          Attached 'Resource not found' with status code 200 example
          Show
          Robin Taylor added a comment - Attached 'Resource not found' with status code 200 example
          Hide
          Kim Shepherd added a comment -
          OK, I'll re-test this morning and see what needs fixing. (Just heading into work now)
          Show
          Kim Shepherd added a comment - OK, I'll re-test this morning and see what needs fixing. (Just heading into work now)
          Hide
          Robin Taylor added a comment -
          Retested but with a bit more care and attention ('mvn clean package' , deleted tomcat work directory, deleted browser cache), still gives a 200 ok.
          Show
          Robin Taylor added a comment - Retested but with a bit more care and attention ('mvn clean package' , deleted tomcat work directory, deleted browser cache), still gives a 200 ok.
          Hide
          Kim Shepherd added a comment -
          Stuart's just tested too, but with the 1.0.1 Cocoon jar, and has reproduced Robin's results.
          I've still got the 1.0.2 jar I attached here, and I'm still getting 404s every time. I'll get Stuart to update his poms and rebuild, and then I'll try the same, and I'll also compare our "released" 1.0.2 jar with the jar attached here to make sure we're testing the same thing!
          Show
          Kim Shepherd added a comment - Stuart's just tested too, but with the 1.0.1 Cocoon jar, and has reproduced Robin's results. I've still got the 1.0.2 jar I attached here, and I'm still getting 404s every time. I'll get Stuart to update his poms and rebuild, and then I'll try the same, and I'll also compare our "released" 1.0.2 jar with the jar attached here to make sure we're testing the same thing!
          Hide
          Stuart Lewis added a comment -
          I have reproduced this: trunk + pom version upgrade to 1.0.2 still returns 200 OK. Manually patching with the jar attached to this issue gives me 404s as expected. The checksum of the jar attached here is different to that pulled in automatically by maven.
          Show
          Stuart Lewis added a comment - I have reproduced this: trunk + pom version upgrade to 1.0.2 still returns 200 OK. Manually patching with the jar attached to this issue gives me 404s as expected. The checksum of the jar attached here is different to that pulled in automatically by maven.
          Hide
          Robin Taylor added a comment -
          Can we confirm that the changes to the jar are in the module trunk and 1.0.2 branch ?

          Cheers, Robin.
          Show
          Robin Taylor added a comment - Can we confirm that the changes to the jar are in the module trunk and 1.0.2 branch ? Cheers, Robin.
          Hide
          Mark Diggory added a comment -
          1.) we need the right changes to be in the SVN
          2.) We are going to need to run a 1.0.3 release
          3.) please package, install and test 1.0.3-SNAPSHOT off the trunk in your local build and we will try the release again after that.

          Mark
          Show
          Mark Diggory added a comment - 1.) we need the right changes to be in the SVN 2.) We are going to need to run a 1.0.3 release 3.) please package, install and test 1.0.3-SNAPSHOT off the trunk in your local build and we will try the release again after that. Mark
          Hide
          Kim Shepherd added a comment - - edited
          I can confirm that the released 1.0.2 jar is not the same jar as the one I've attached to this issue.
          Investigating and trying a 1.0.3 release this weekend.
          Show
          Kim Shepherd added a comment - - edited I can confirm that the released 1.0.2 jar is not the same jar as the one I've attached to this issue. Investigating and trying a 1.0.3 release this weekend.
          Hide
          Kim Shepherd added a comment -
          OK, I can confirm that the source I committed to svn (modules/dspace-cocoon-servlet-service-impl) is the right stuff -- I just manually built it into a .jar, renamed to 1.0.2 and dropped it in after clearing cache etc., and it works. It's also the same filesize as the jar attached to this job, which is a good sign :) If anyone feels like doing a checkout on https://scm.dspace.org/svn/repo/modules/dspace-cocoon-servlet-service-impl/trunk/ and building it (it'll be named 1.0.3 in this case, i think.. don't worry about the version number it gets given), removing your existing dspace-servlet-service-impl, and dropping this into xmlui/WEB-INF/lib and testing, that'd be great, but i'm very confident that this is the right jar, and that my source is building the right jar.

          So we just have to sort out the maven/sonatype release stuff. Not sure on our sonatype procedure for modules but I'd like to try it straight from my computer here if that'd help.. Mark, is there any danger?
          Show
          Kim Shepherd added a comment - OK, I can confirm that the source I committed to svn (modules/dspace-cocoon-servlet-service-impl) is the right stuff -- I just manually built it into a .jar, renamed to 1.0.2 and dropped it in after clearing cache etc., and it works. It's also the same filesize as the jar attached to this job, which is a good sign :) If anyone feels like doing a checkout on https://scm.dspace.org/svn/repo/modules/dspace-cocoon-servlet-service-impl/trunk/ and building it (it'll be named 1.0.3 in this case, i think.. don't worry about the version number it gets given), removing your existing dspace-servlet-service-impl, and dropping this into xmlui/WEB-INF/lib and testing, that'd be great, but i'm very confident that this is the right jar, and that my source is building the right jar. So we just have to sort out the maven/sonatype release stuff. Not sure on our sonatype procedure for modules but I'd like to try it straight from my computer here if that'd help.. Mark, is there any danger?
          Hide
          Mark Diggory added a comment -
          Do you have the sonatype account and a gpg signature setup yet? I can't remember if we set you up with this stuff.
          Show
          Mark Diggory added a comment - Do you have the sonatype account and a gpg signature setup yet? I can't remember if we set you up with this stuff.
          Hide
          Kim Shepherd added a comment -
          I've created an account on Sonayype. Username is "kshepherd". Requesting authorisation as per https://wiki.duraspace.org/display/DSPACE/Release+Procedure#ReleaseProcedure-Prerequisites.

          (although I seem to already be able to login to oss.sonatype.org... maybe not with the right privileges?)

          In the meantime, I'll generate a GPG keypair and publish to the appropriate places.
          Show
          Kim Shepherd added a comment - I've created an account on Sonayype. Username is "kshepherd". Requesting authorisation as per https://wiki.duraspace.org/display/DSPACE/Release+Procedure#ReleaseProcedure-Prerequisites . (although I seem to already be able to login to oss.sonatype.org... maybe not with the right privileges?) In the meantime, I'll generate a GPG keypair and publish to the appropriate places.
          Hide
          Robin Taylor added a comment -
          Hi Kim,

          When anyone does a release an email is sent by Sonatype and you don't appear to be on the distribution list, so I guess that you have an account but no request has ever been made to add you as an admin for dspace. Unfortunately Sonatype's Jira is not responding at the moment but I'll raise a request to have you added as soon as it returns. If you prefer that I do the release please don't hesitate to ask.

          Cheers, Robin.
          Show
          Robin Taylor added a comment - Hi Kim, When anyone does a release an email is sent by Sonatype and you don't appear to be on the distribution list, so I guess that you have an account but no request has ever been made to add you as an admin for dspace. Unfortunately Sonatype's Jira is not responding at the moment but I'll raise a request to have you added as soon as it returns. If you prefer that I do the release please don't hesitate to ask. Cheers, Robin.
          Hide
          Robin Taylor added a comment -
          Looks like Mark was ahead of me and had already requested that you be granted authorisation, should now be in place.

          Cheers.
          Show
          Robin Taylor added a comment - Looks like Mark was ahead of me and had already requested that you be granted authorisation, should now be in place. Cheers.
          Hide
          Tim Donohue added a comment - - edited
          Hi All,

          I did a bit of testing this morning. I actually cannot get the Trunk code at https://scm.dspace.org/svn/repo/modules/dspace-cocoon-servlet-service-impl/trunk/ working.

          Here's the behavior I'm seeing using Tomcat 7:

          1) From Trunk code I get a 200 OK at all times if I do the following:
                   * Build (mvn clean install) https://scm.dspace.org/svn/repo/modules/dspace-cocoon-servlet-service-impl/trunk/ -> builds a 1.0.3-SNAPSHOT version
                   * Change dependency in [dspace-src]/dspace-xmlui/dspace-xmlui-wing/pom.xml to point at version 1.0.3-SNAPSHOT
                   * Rebuild DSpace (mvn clean package) - You should see the 1.0.3-SNAPSHOT version get pulled into XMLUI's WEB-INF/lib/
                   * Clear Tomcat cache & startup & test
                   * No matter what invalid path I enter, I seem to always get a 200 OK.

          2) I also get a 200 OK at all times if I use the 1.0.2 version released to Maven Central.

          3) However, if I manually drop in the 1.0.2 JAR that Kim attached above, then I get the "404 Not Found" response as expected.

          All of the above makes me suspect that Kim's changes are not actually implemented at: https://scm.dspace.org/svn/repo/modules/dspace-cocoon-servlet-service-impl/trunk/ I suspect something must be missing in that code (but I'm not familiar enough with it to know what is missing). However, I do notice that the code changes in the '[DS-768]_dspace-cocoon-servlet-service-impl-1_0_2.patch' attached above do not seem to be implemented in that trunk code (I had mentioned this to Kim yesterday, but he thought that patch may be out-of-date).

          In any case, I'm not sure a 1.0.3 release will change anything quite yet. I believe the trunk code itself is not working right (though this could use more eyes on it to make sure the behavior I'm seeing is consistent with others)
          Show
          Tim Donohue added a comment - - edited Hi All, I did a bit of testing this morning. I actually cannot get the Trunk code at https://scm.dspace.org/svn/repo/modules/dspace-cocoon-servlet-service-impl/trunk/ working. Here's the behavior I'm seeing using Tomcat 7: 1) From Trunk code I get a 200 OK at all times if I do the following:          * Build (mvn clean install) https://scm.dspace.org/svn/repo/modules/dspace-cocoon-servlet-service-impl/trunk/ -> builds a 1.0.3-SNAPSHOT version          * Change dependency in [dspace-src]/dspace-xmlui/dspace-xmlui-wing/pom.xml to point at version 1.0.3-SNAPSHOT          * Rebuild DSpace (mvn clean package) - You should see the 1.0.3-SNAPSHOT version get pulled into XMLUI's WEB-INF/lib/          * Clear Tomcat cache & startup & test          * No matter what invalid path I enter, I seem to always get a 200 OK. 2) I also get a 200 OK at all times if I use the 1.0.2 version released to Maven Central. 3) However, if I manually drop in the 1.0.2 JAR that Kim attached above, then I get the "404 Not Found" response as expected. All of the above makes me suspect that Kim's changes are not actually implemented at: https://scm.dspace.org/svn/repo/modules/dspace-cocoon-servlet-service-impl/trunk/ I suspect something must be missing in that code (but I'm not familiar enough with it to know what is missing). However, I do notice that the code changes in the '[ DS-768 ]_dspace-cocoon-servlet-service-impl-1_0_2.patch' attached above do not seem to be implemented in that trunk code (I had mentioned this to Kim yesterday, but he thought that patch may be out-of-date). In any case, I'm not sure a 1.0.3 release will change anything quite yet. I believe the trunk code itself is not working right (though this could use more eyes on it to make sure the behavior I'm seeing is consistent with others)
          Hide
          Kim Shepherd added a comment -
          Thanks for the help with testing, Tim... will definitely make sure the code in trunk is working for everybody before cutting another release. I might try a total re-import into SCM this morning.
          Sorry to everyone (especially Robin!) that this fix is so drawn out and confusing... it's a combination of weird things like not having the source for the 1.0.1 jar we currently ship, too many working copies / open projects on my computers, inconsistent testing results, etc... I have good code, and I have a good jar, so I just need to make sure SCM has the good code, now, and that sonatype gets the good jar ;)
          Show
          Kim Shepherd added a comment - Thanks for the help with testing, Tim... will definitely make sure the code in trunk is working for everybody before cutting another release. I might try a total re-import into SCM this morning. Sorry to everyone (especially Robin!) that this fix is so drawn out and confusing... it's a combination of weird things like not having the source for the 1.0.1 jar we currently ship, too many working copies / open projects on my computers, inconsistent testing results, etc... I have good code, and I have a good jar, so I just need to make sure SCM has the good code, now, and that sonatype gets the good jar ;)
          Hide
          Tim Donohue added a comment -
          Hi Kim,

          A little more info...it looks like in our SCM we just need to apply the Cocoon r685544 patch that is supposed to fix *both* DS-253 / COCOON-2217, as well as DS-768 / COCOON-2218.

          So, I think the issue is that amongst all our hurrying to get this "fixed", that full patch (to mainly the HttpServletResponseBufferWrapper) has not been applied to our SCM.

          Here's the specific changes that are not in our SCM:
          * Full list of r685544 changes: http://svn.apache.org/viewvc?view=revision&revision=685544
          * Patch for HttpServletResponseBufferWrapper.java : http://svn.apache.org/viewvc/cocoon/trunk/subprojects/cocoon-servlet-service/cocoon-servlet-service-impl/src/main/java/org/apache/cocoon/servletservice/HttpServletResponseBufferingWrapper.java?r1=685544&r2=685543&pathrev=685544&view=patch
          * Patch for HttpServletResponseBufferWrapperTestCase.java : http://svn.apache.org/viewvc/cocoon/trunk/subprojects/cocoon-servlet-service/cocoon-servlet-service-impl/src/test/java/org/apache/cocoon/servletservice/HttpServletResponseBufferingWrapperTestCase.java?r1=685544&r2=685543&pathrev=685544&view=patch

          Assuming those patches apply cleanly to our SCM, I think they should resolve our issues (or at least that seems to be the case, based on the comments on both COCOON-2217 and COCOON-2218)
          Show
          Tim Donohue added a comment - Hi Kim, A little more info...it looks like in our SCM we just need to apply the Cocoon r685544 patch that is supposed to fix *both* DS-253 / COCOON-2217, as well as DS-768 / COCOON-2218. So, I think the issue is that amongst all our hurrying to get this "fixed", that full patch (to mainly the HttpServletResponseBufferWrapper) has not been applied to our SCM. Here's the specific changes that are not in our SCM: * Full list of r685544 changes: http://svn.apache.org/viewvc?view=revision&revision=685544 * Patch for HttpServletResponseBufferWrapper.java : http://svn.apache.org/viewvc/cocoon/trunk/subprojects/cocoon-servlet-service/cocoon-servlet-service-impl/src/main/java/org/apache/cocoon/servletservice/HttpServletResponseBufferingWrapper.java?r1=685544&r2=685543&pathrev=685544&view=patch * Patch for HttpServletResponseBufferWrapperTestCase.java : http://svn.apache.org/viewvc/cocoon/trunk/subprojects/cocoon-servlet-service/cocoon-servlet-service-impl/src/test/java/org/apache/cocoon/servletservice/HttpServletResponseBufferingWrapperTestCase.java?r1=685544&r2=685543&pathrev=685544&view=patch Assuming those patches apply cleanly to our SCM, I think they should resolve our issues (or at least that seems to be the case, based on the comments on both COCOON-2217 and COCOON-2218)
          Hide
          Kim Shepherd added a comment -
          You're right about the Cocoon revision, but since that was the revision I used to create my patch against in the first place, I'm pretty sure my working copy already should have already had this patch. But it's probably about time I stopped talking about it and just figured out where I went wrong with my previous commits/patches/releases. (and why all my local copies/tests work?!)
          Show
          Kim Shepherd added a comment - You're right about the Cocoon revision, but since that was the revision I used to create my patch against in the first place, I'm pretty sure my working copy already should have already had this patch. But it's probably about time I stopped talking about it and just figured out where I went wrong with my previous commits/patches/releases. (and why all my local copies/tests work?!)
          Hide
          Tim Donohue added a comment -
          Your working copy may have that patch, but I've verified that SCM does not. For instance, notice that the HttpServletResponseBufferWrapper in our SCM does *not* define the new 'ForwardingOrLimitingServletOutputStream' inner class (instead it uses the older 'LimitingServletOutputStream' inner class).

          http://scm.dspace.org/svn/repo/modules/dspace-cocoon-servlet-service-impl/trunk/src/main/java/org/apache/cocoon/servletservice/HttpServletResponseBufferingWrapper.java

          That's an example of something that changed in r685544 which hasn't made it into our DSpace SCM.

          I've also noticed that there have been even more Cocoon fixes to the HttpServletResponseBufferWrapper since that 'r685544', which we also may want to apply in our version -- they all seem to be minor bug fixes to the larger changes that were made in 'r685544'

          http://svn.apache.org/viewvc/cocoon/trunk/subprojects/cocoon-servlet-service/cocoon-servlet-service-impl/src/main/java/org/apache/cocoon/servletservice/HttpServletResponseBufferingWrapper.java


           
          Show
          Tim Donohue added a comment - Your working copy may have that patch, but I've verified that SCM does not. For instance, notice that the HttpServletResponseBufferWrapper in our SCM does *not* define the new 'ForwardingOrLimitingServletOutputStream' inner class (instead it uses the older 'LimitingServletOutputStream' inner class). http://scm.dspace.org/svn/repo/modules/dspace-cocoon-servlet-service-impl/trunk/src/main/java/org/apache/cocoon/servletservice/HttpServletResponseBufferingWrapper.java That's an example of something that changed in r685544 which hasn't made it into our DSpace SCM. I've also noticed that there have been even more Cocoon fixes to the HttpServletResponseBufferWrapper since that 'r685544', which we also may want to apply in our version -- they all seem to be minor bug fixes to the larger changes that were made in 'r685544' http://svn.apache.org/viewvc/cocoon/trunk/subprojects/cocoon-servlet-service/cocoon-servlet-service-impl/src/main/java/org/apache/cocoon/servletservice/HttpServletResponseBufferingWrapper.java  
          Hide
          Tim Donohue added a comment -
          To slightly restate what I just "implied" above with my last statement. I think if we pull down the latest versions (HEAD) of HttpServletResponseBufferingWrapper.java & HttpServletResponseBufferingWrapperTestCase.java & commit both to our DSpace SCM, then that will resolve our issues.

          http://svn.apache.org/viewvc/cocoon/trunk/subprojects/cocoon-servlet-service/cocoon-servlet-service-impl/src/main/java/org/apache/cocoon/servletservice/HttpServletResponseBufferingWrapper.java

          http://svn.apache.org/viewvc/cocoon/trunk/subprojects/cocoon-servlet-service/cocoon-servlet-service-impl/src/test/java/org/apache/cocoon/servletservice/HttpServletResponseBufferingWrapperTestCase.java

          (famous last words) ;)
          Show
          Tim Donohue added a comment - To slightly restate what I just "implied" above with my last statement. I think if we pull down the latest versions (HEAD) of HttpServletResponseBufferingWrapper.java & HttpServletResponseBufferingWrapperTestCase.java & commit both to our DSpace SCM, then that will resolve our issues. http://svn.apache.org/viewvc/cocoon/trunk/subprojects/cocoon-servlet-service/cocoon-servlet-service-impl/src/main/java/org/apache/cocoon/servletservice/HttpServletResponseBufferingWrapper.java http://svn.apache.org/viewvc/cocoon/trunk/subprojects/cocoon-servlet-service/cocoon-servlet-service-impl/src/test/java/org/apache/cocoon/servletservice/HttpServletResponseBufferingWrapperTestCase.java (famous last words) ;)
          Hide
          Kim Shepherd added a comment -
          Good call Tim, I've tested that out and it all seems fine here, so I've committed the changes back to source. Nearly the end of the day here so after dinner I'll have a go at a sonatype release.
          Show
          Kim Shepherd added a comment - Good call Tim, I've tested that out and it all seems fine here, so I've committed the changes back to source. Nearly the end of the day here so after dinner I'll have a go at a sonatype release.
          Hide
          Tim Donohue added a comment -
          I re-tested since Kim re-patched the above classes (see r6800 in DSpace SCM). Everything now seems to be working perfectly. I get a 404 when accessing any invalid path (and continue to get a 404 if I keep accessing that same path over and over).

          I think the only things left to do are:
          * Release a 1.0.3 version of dspace-cocoon-servlet-service-impl to Maven Central
          * Once propagated to Maven Central, change the dependency in [Trunk]/dspace-xmlui/dspace-xmlui-wing/pom.xml to point at version 1.0.3 of dspace-cocoon-servlet-service-impl
          Show
          Tim Donohue added a comment - I re-tested since Kim re-patched the above classes (see r6800 in DSpace SCM). Everything now seems to be working perfectly. I get a 404 when accessing any invalid path (and continue to get a 404 if I keep accessing that same path over and over). I think the only things left to do are: * Release a 1.0.3 version of dspace-cocoon-servlet-service-impl to Maven Central * Once propagated to Maven Central, change the dependency in [Trunk]/dspace-xmlui/dspace-xmlui-wing/pom.xml to point at version 1.0.3 of dspace-cocoon-servlet-service-impl
          Hide
          Kim Shepherd added a comment -
          Version 1.0.3 of dspace-cocoon-servlet-service-impl released to sonatype
          Show
          Kim Shepherd added a comment - Version 1.0.3 of dspace-cocoon-servlet-service-impl released to sonatype
          Hide
          Robin Taylor added a comment -
          dspace-xmlui-wing/pom.xml updated to pick up version 1.0.3 of dspace-cocoon-servlet-service-impl. Committed to trunk revision 6811.

          Cheers, Robin.
          Show
          Robin Taylor added a comment - dspace-xmlui-wing/pom.xml updated to pick up version 1.0.3 of dspace-cocoon-servlet-service-impl. Committed to trunk revision 6811. Cheers, Robin.

            People

            • Assignee:
              Kim Shepherd
              Reporter:
              Tim Donohue
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: