Details

    • Type: Improvement
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0
    • Component/s: None
    • Labels:
      None
    • Attachments:
      0
    • Comments:
      8
    • Documentation Status:
      Needed

      Description

      https://wiki.duraspace.org/display/DSPACE/Maven+Project+Consolidation

      This project would reorganize and consolidate some of the maven project hierarchy to reduce the number of maven projects neccessary to operate DSpace.

      Benefits: Reduce the number of maven projects in DSpace "Core" from 39 to 19.

      GitHub Branch for this is located here: https://github.com/DSpace/DSpace/tree/maven-project-consolidation

      This project would consolidate:

      dspace-xmlui
      dspace-xmlui-wing
      src/main/java
      dspace-xmlui-api
      src/main/java
      src/main/resources
      dspace-xmlui-webapp
      src/main/resources
      src/main/webapp

      and it would reduce it down to

      dspace-xmlui
      src/main/java
      src/main/resources
      src/main/webapp

      We would utilize the current model found in dspace-swordv2 (http://scm.dspace.org/svn/repo/dspace/trunk/dspace/modules/swordv2)

      dspace-swordv2
      src/main/java
      src/main/resources
      src/main/webapp

        Attachments

          Activity

          Hide
          tdonohue Tim Donohue added a comment -

          Hi Mark,

          Taking a look at this work. First off, I like the idea a lot. A few immediate concerns/questions though:

          1) A bit nit-picky, but I'm not a fan of the name 'dspace-common-api'. This looks to be the idea of the "business logic api". I'd rather this be named something like "dspace-ui-logic-api" or "dspace-business-api" or "dspace-common-ui-api" (or something else). Essentially, the name "-common-api" to me implies it is used everywhere in DSpace. But, obviously it's not "common" across everything in DSpace. It's really UI specific.

          2) I'm wondering what the logic is behind keeping "dspace-discovery-solr" as a standalone top-level project? Why does the "dspace-sword-client" project get swallowed into the "dspace-common-api" while the "dspace-discovery-solr" does not? Both of these projects seem similar, in that they are XMLUI specific currently, but in the future may be expandable to other UIs.

          3) Just curious about the new [dspace-src]/src/main/assembly/ folder structure. This isn't described anywhere in your docs. Is this where the 'dspace-assembly-plugin' will be moved to? http://scm.dspace.org/svn/repo/tools/maven/dspace-assembly-plugin/

          Beyond that, I do really like the consoldation of folders. All the source code for the web applications (xmlui, jspui, oai, sword, lni) seems a lot "cleaner".

          Show
          tdonohue Tim Donohue added a comment - Hi Mark, Taking a look at this work. First off, I like the idea a lot. A few immediate concerns/questions though: 1) A bit nit-picky, but I'm not a fan of the name 'dspace-common-api'. This looks to be the idea of the "business logic api". I'd rather this be named something like "dspace-ui-logic-api" or "dspace-business-api" or "dspace-common-ui-api" (or something else). Essentially, the name "-common-api" to me implies it is used everywhere in DSpace. But, obviously it's not "common" across everything in DSpace. It's really UI specific. 2) I'm wondering what the logic is behind keeping "dspace-discovery-solr" as a standalone top-level project? Why does the "dspace-sword-client" project get swallowed into the "dspace-common-api" while the "dspace-discovery-solr" does not? Both of these projects seem similar, in that they are XMLUI specific currently, but in the future may be expandable to other UIs. 3) Just curious about the new [dspace-src] /src/main/assembly/ folder structure. This isn't described anywhere in your docs. Is this where the 'dspace-assembly-plugin' will be moved to? http://scm.dspace.org/svn/repo/tools/maven/dspace-assembly-plugin/ Beyond that, I do really like the consoldation of folders. All the source code for the web applications (xmlui, jspui, oai, sword, lni) seems a lot "cleaner".
          Hide
          mdiggory Mark Diggory added a comment -

          Tim,

          I'm happy for any feedback. Jump in where-ever you want.

          1.) dspace-common-api is not necessarily UI centric, so I'd not use UI in its name. It is all the secondary addons that were outside scope of dspace-api and needed across all applications. I see common as vague too. I wish I had an opportunity to rename dspace-api, it was really poorly named. I'll leave the naming open to the community to decide though...

          2.) dspace-discovery-solr was left behind because it is an example of the plugability of Discovery to be implemented in Solr. for instance, there would be a dspace-discovery-elasticsearch or a dspace-discovery-legacy that would encapsulate the dependencies in each case, IE we don't want to build in elastic search unless the user wants to include that addon. This said, we do need to assure theres a good default for discovery, I'd be up for consolidating if we can accept solr as the default.

          3.) I believe this is in trunk already...
          http://scm.dspace.org/svn/repo/dspace/trunk/src/main/...
          its part of MarkWs work on the Test environment. But we could place the assemblies in here if it would make things easier to support in the long run.

          Show
          mdiggory Mark Diggory added a comment - Tim, I'm happy for any feedback. Jump in where-ever you want. 1.) dspace-common-api is not necessarily UI centric, so I'd not use UI in its name. It is all the secondary addons that were outside scope of dspace-api and needed across all applications. I see common as vague too. I wish I had an opportunity to rename dspace-api, it was really poorly named. I'll leave the naming open to the community to decide though... 2.) dspace-discovery-solr was left behind because it is an example of the plugability of Discovery to be implemented in Solr. for instance, there would be a dspace-discovery-elasticsearch or a dspace-discovery-legacy that would encapsulate the dependencies in each case, IE we don't want to build in elastic search unless the user wants to include that addon. This said, we do need to assure theres a good default for discovery, I'd be up for consolidating if we can accept solr as the default. 3.) I believe this is in trunk already... http://scm.dspace.org/svn/repo/dspace/trunk/src/main/ ... its part of MarkWs work on the Test environment. But we could place the assemblies in here if it would make things easier to support in the long run.
          Hide
          mdiggory Mark Diggory added a comment -

          Taking a look at this work. First off, I like the idea a lot. A few immediate concerns/questions though:

          > 1) A bit nit-picky, but I'm not a fan of the name 'dspace-common-api'. This looks to be the idea of
          > the "business logic api". I'd rather this be named something like "dspace-ui-logic-api" or "dspace-business-api"
          > or "dspace-common-ui-api" (or something else). Essentially, the name "-common-api" to me implies
          > it is used everywhere in DSpace. But, obviously it's not "common" across everything in DSpace.
          > It's really UI specific.

          Ok, in the meantime I dropped the project name. We still have

          https://github.com/DSpace/DSpace/tree/maven-project-consolidation/dspace-stats
          https://github.com/DSpace/DSpace/tree/maven-project-consolidation/dspace-discovery-solr

          The sword-client was consolidated entirely into xmlui and org.purl.sword was left in dspace-sword

          > 2) I'm wondering what the logic is behind keeping "dspace-discovery-solr" as a standalone top-level project?
          > Why does the "dspace-sword-client" project get swallowed into the "dspace-common-api" while the
          > "dspace-discovery-solr" does not? Both of these projects seem similar, in that they are XMLUI
          > specific currently, but in the future may be expandable to other UIs.

          Because its a plugin for discovery, elastic-search will be another in the near future. (We should review this with Peter and Kevin on Wednesday)

          Show
          mdiggory Mark Diggory added a comment - Taking a look at this work. First off, I like the idea a lot. A few immediate concerns/questions though: > 1) A bit nit-picky, but I'm not a fan of the name 'dspace-common-api'. This looks to be the idea of > the "business logic api". I'd rather this be named something like "dspace-ui-logic-api" or "dspace-business-api" > or "dspace-common-ui-api" (or something else). Essentially, the name "-common-api" to me implies > it is used everywhere in DSpace. But, obviously it's not "common" across everything in DSpace. > It's really UI specific. Ok, in the meantime I dropped the project name. We still have https://github.com/DSpace/DSpace/tree/maven-project-consolidation/dspace-stats https://github.com/DSpace/DSpace/tree/maven-project-consolidation/dspace-discovery-solr The sword-client was consolidated entirely into xmlui and org.purl.sword was left in dspace-sword > 2) I'm wondering what the logic is behind keeping "dspace-discovery-solr" as a standalone top-level project? > Why does the "dspace-sword-client" project get swallowed into the "dspace-common-api" while the > "dspace-discovery-solr" does not? Both of these projects seem similar, in that they are XMLUI > specific currently, but in the future may be expandable to other UIs. Because its a plugin for discovery, elastic-search will be another in the near future. (We should review this with Peter and Kevin on Wednesday)
          Hide
          tdonohue Tim Donohue added a comment -

          Hi Mark,

          Finally getting back to this ticket.

          Again, I do like nearly all the ideas you have here. I like the direction this is going in.

          I guess the only outstanding question I have is around the treatment of 'dspace-stats' and 'dspace-discovery-solr' (which seem to be similar in a lot of ways, but are treated differently, at least in terms of naming).

          As you mentioned, there is hope that in the future there will be an option to use either Solr or Elastic-Search for 'dspace-stats' and 'dspace-discovery'. This makes sense, but it sounds like you are then suggesting that once Elastic-Search is an option we'd have a folder structure like:

          [src]/dspace-stats-solr/
          [src]/dspace-stats-elastic/
          [src]/dspace-discovery-solr/
          [src]/dspace-discovery-elastic/

          My question is, should we just let those all end up sitting at the "root" level? Or do we want to rethink this into a structure more like:
          [src]/dspace-stats/dspace-stats-solr/
          [src]/dspace-stats/dspace-stats-elastic/
          [src]/dspace-discovery/dspace-discovery-solr/
          [src]/dspace-discovery/dspace-discovery-elastic/

          Or, potentially for the time being, just remove all mentions of Solr or Elastic search (and as necessary in the future, revisit this should we ever release DSpace Stats & Discovery with multiple options).
          [src]/dspace-stats/src/main/java/
          [src]/dspace-discovery/src/main/java/

          Beyond that, I think this is looking good. I just want us to be consistent with how we are treating "-discovery" and "-stats". We should either suffix them both with "-solr" or suffix neither with "-solr" (and potentially revisit it later on, if we decide to release an Elastic-Search option)

          Show
          tdonohue Tim Donohue added a comment - Hi Mark, Finally getting back to this ticket. Again, I do like nearly all the ideas you have here. I like the direction this is going in. I guess the only outstanding question I have is around the treatment of 'dspace-stats' and 'dspace-discovery-solr' (which seem to be similar in a lot of ways, but are treated differently, at least in terms of naming). As you mentioned, there is hope that in the future there will be an option to use either Solr or Elastic-Search for 'dspace-stats' and 'dspace-discovery'. This makes sense, but it sounds like you are then suggesting that once Elastic-Search is an option we'd have a folder structure like: [src] /dspace-stats-solr/ [src] /dspace-stats-elastic/ [src] /dspace-discovery-solr/ [src] /dspace-discovery-elastic/ My question is, should we just let those all end up sitting at the "root" level? Or do we want to rethink this into a structure more like: [src] /dspace-stats/dspace-stats-solr/ [src] /dspace-stats/dspace-stats-elastic/ [src] /dspace-discovery/dspace-discovery-solr/ [src] /dspace-discovery/dspace-discovery-elastic/ Or, potentially for the time being, just remove all mentions of Solr or Elastic search (and as necessary in the future, revisit this should we ever release DSpace Stats & Discovery with multiple options). [src] /dspace-stats/src/main/java/ [src] /dspace-discovery/src/main/java/ Beyond that, I think this is looking good. I just want us to be consistent with how we are treating "-discovery" and "-stats". We should either suffix them both with "-solr" or suffix neither with "-solr" (and potentially revisit it later on, if we decide to release an Elastic-Search option)
          Hide
          mdiggory Mark Diggory added a comment -

          I think that we need some feedback from Peter Dietz per his work on elastic stats because this will clarify if dspace-stats can be generalized and separate providers established. Much of the stats code relies on solr query syntax, which for me, is a concern in accomplishing that... it should probably be a latter topic.

          Remember I pulled the "Discovery Provider" code into dspace-api. And its the responsibility of the above packages to provide an implementation of that provider.

          Your questions suggests to me the following choice to make.

          1.) Place the default Providers (dspace-statistics and dspace-discovery-solr) into dspace-api

          2.) Keep the Default Providers in an external projects.

          Note, all this is very good for working out a "path of incubation" for new addons to DSpace becoming "core". but in regards to "modularity" I'm not sure yet it is necessary to keep the default implementations separate.

          Show
          mdiggory Mark Diggory added a comment - I think that we need some feedback from Peter Dietz per his work on elastic stats because this will clarify if dspace-stats can be generalized and separate providers established. Much of the stats code relies on solr query syntax, which for me, is a concern in accomplishing that... it should probably be a latter topic. Remember I pulled the "Discovery Provider" code into dspace-api. And its the responsibility of the above packages to provide an implementation of that provider. Your questions suggests to me the following choice to make. 1.) Place the default Providers (dspace-statistics and dspace-discovery-solr) into dspace-api 2.) Keep the Default Providers in an external projects. Note, all this is very good for working out a "path of incubation" for new addons to DSpace becoming "core". but in regards to "modularity" I'm not sure yet it is necessary to keep the default implementations separate.
          Hide
          tdonohue Tim Donohue added a comment -

          I may be missing something here then. My comments above are based on the fact that I see Stats & Discovery as very "similar" modules (in the same way that all the webapps are similar in structure). They both currently have Solr implementations & in the future may or may not support other options (like Elastic Search).

          The oddity (to me), is that despite those similarities, as far as I can tell, the Stats code is all in [src]/dspace-stats/ whereas the Discovery code is split into [src]/dspace-api (Provider) and [src]/dspace-discovery-solr/ (Implementation).

          I guess my assumption was that we should be treating these both similar (just like we've restructured all the webapps in a similar fashion). The point of this whole reorg is obviously to simplify our Maven structure (in order to ease in new/novice developers learning how to work with DSpace).

          In my mind this makes we wonder whether all the Discovery stuff shouldn't just be in [src]/dspace-discovery (both Provider & implemenation). But, maybe there's a reason that this won't work well?

          The key point here is that – if I'm confused about why Discovery & Stats seem to have taken different paths in terms of "where the code lives", than I wonder what newer developers will run into.

          Show
          tdonohue Tim Donohue added a comment - I may be missing something here then. My comments above are based on the fact that I see Stats & Discovery as very "similar" modules (in the same way that all the webapps are similar in structure). They both currently have Solr implementations & in the future may or may not support other options (like Elastic Search). The oddity (to me), is that despite those similarities, as far as I can tell, the Stats code is all in [src] /dspace-stats/ whereas the Discovery code is split into [src] /dspace-api (Provider) and [src] /dspace-discovery-solr/ (Implementation). I guess my assumption was that we should be treating these both similar (just like we've restructured all the webapps in a similar fashion). The point of this whole reorg is obviously to simplify our Maven structure (in order to ease in new/novice developers learning how to work with DSpace). In my mind this makes we wonder whether all the Discovery stuff shouldn't just be in [src] /dspace-discovery (both Provider & implemenation). But, maybe there's a reason that this won't work well? The key point here is that – if I'm confused about why Discovery & Stats seem to have taken different paths in terms of "where the code lives", than I wonder what newer developers will run into.
          Hide
          mdiggory Mark Diggory added a comment -

          I don't think your far off from my perspective. My consolidation goal is based on libraries that are common across all apps canbe consolidated into dspace-api. And it is better to "extract" the providers of services (Search, Browse, Discovery, Stats, MediaFilters, Consumers, DAO, etc) out of and make them dependent on dspace-api

          dspace-common was going to be the project that assembled all these dependencies back together so that you did not need to add them all to each and every webapp.

          (Note at one time, I was trying to make the actual dspace assembly project this common maven project, but its difficult to combine assembly and jar projects into one unit).

          The reason dspace-stats still is outside is just the dspace-stats is not as as cleanly separated into service and provider as dspace-discovery is. It needs to be separated apart into stats service and stats solr provider to keep the solr parts out of dspace-api (if that is even an issue).


          Even if you and I agree is our long term goal to break apart dspace-api into content/core and separate apps, the rest of the committers group isn't fully buying into it, there is not general trend emerging to separate these out (unless theres some big project over in RichardR's space I'm not aware of, but thats another story). This is why I started folding everything together again,

          For instance, if we want to replace DSQuery and Browse with Discovery, I think it important that we get the code organized into one project and do it there. This either means that we would alter XMLUI and JSPUI code to dormally remove Search/Browse in favor of Discovery, or we will reimplement DSQuery and Browse as services that reuse the Solr/Discovery Provider.

          Show
          mdiggory Mark Diggory added a comment - I don't think your far off from my perspective. My consolidation goal is based on libraries that are common across all apps canbe consolidated into dspace-api. And it is better to "extract" the providers of services (Search, Browse, Discovery, Stats, MediaFilters, Consumers, DAO, etc) out of and make them dependent on dspace-api dspace-common was going to be the project that assembled all these dependencies back together so that you did not need to add them all to each and every webapp. (Note at one time, I was trying to make the actual dspace assembly project this common maven project, but its difficult to combine assembly and jar projects into one unit). The reason dspace-stats still is outside is just the dspace-stats is not as as cleanly separated into service and provider as dspace-discovery is. It needs to be separated apart into stats service and stats solr provider to keep the solr parts out of dspace-api (if that is even an issue). Even if you and I agree is our long term goal to break apart dspace-api into content/core and separate apps, the rest of the committers group isn't fully buying into it, there is not general trend emerging to separate these out (unless theres some big project over in RichardR's space I'm not aware of, but thats another story). This is why I started folding everything together again, For instance, if we want to replace DSQuery and Browse with Discovery, I think it important that we get the code organized into one project and do it there. This either means that we would alter XMLUI and JSPUI code to dormally remove Search/Browse in favor of Discovery, or we will reimplement DSQuery and Browse as services that reuse the Solr/Discovery Provider.
          Hide
          tdonohue Tim Donohue added a comment -

          This work was completed in Pull #90:
          https://github.com/DSpace/DSpace/pull/90

          Show
          tdonohue Tim Donohue added a comment - This work was completed in Pull #90: https://github.com/DSpace/DSpace/pull/90

            People

            • Assignee:
              mdiggory Mark Diggory
              Reporter:
              mdiggory Mark Diggory
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved: