Repository navigation
Pros and Cons for more modularisation #1031
Description
Refactoring the RDFlib package into a core and a series of satellite repositories has been proposed for a long time (see Issue #391) and it was considered a possible goal of RDFlib's 6.0.0 release.
In order to proceed, we need at least some pros and cons of such as move, as well as a refactoring suggestions (what stays in core, what moves out) and this Issue is to document that.
Please add comments to this starting with PRO: or CON: and your suggestions for what should be in or out of core, with reference to the current repo structure.
Reacted by Andreas Kuckartz
Activity
tgbugs commentedon May 1, 2020
CON: More packages means more work for maintainers. In my experience time cost is essentially constant per package per release, so if you split a single package into 10 you have 10x the work that you have to dedicate to managing releases. This is the general boilerplate overhead.
CON: Could increase the number of dependencies and complicate the process needed to get the right set of optional dependencies for some desired functionality.
Can we get more modularization internally within the rdflib repository without adding the maintenance overhead?
What release cycles currently suffer because of the current system?
Are there clear examples where some code could be used by another project? If the code is only used by rdflib then it seems like there is only a cost.
I'll add some pros at some point.
niklasl commentedon May 1, 2020
PRO: If a feature depends on a third-party library, that burdens anyone who only needs to use the core features. As it stands, I doubt we have much of that though. If e.g. RDFa parsing depends on html5lib though, that's a case. One can conceive of future cases though, like efficient stream parsers or asynchronous handling of networking.
(I do think it's better for RDFLib to err on the side of pragmatic though, i.e. a bit heavier on features and dependencies is reasonably good enough. Still, third-party dependency issues have historically and will possibly continue to crop up occasionally.)
rduivenvoorde commentedon Oct 11, 2020
@tgbugs as an example rdflib being used by a UI-based plugin for QGIS (a GIS desktop application): https://github.com/sparqlunicorn/sparqlunicornGoesGIS
Being tied to urllib2 is in this case a CON, as QGIS is Qt based and it is better to use Qt based classes then for network requests as then settings/debug etc of QGIS can be used (see: sparqlunicorn/sparqlunicornGoesGIS#18 (comment)).
As a visuable example: the 2nd WorldWar bunker in The Netherlands queried via the plugin/rdflib and shown in QGIS:
So speaking as QGIS community member: it would be nice to at least make the networking requests modular (maybe using some abstraction layer like:
https://github.com/planetfederal/lib-qgis-commons/blob/master/qgiscommons2/network/networkaccessmanager.py
I'm not sure if this is feasible or not...
nicholascar commentedon Oct 11, 2020
@rduivenvoorde can you please try running your QGIC plugin with RDFlib
master, as in built from the master branch of this repo? Since we've removed support for Python 2, I don't think we are tied to urllib2 anywhere and we've also just removed the dependency on therequestsmodule too, so it and it would be good to know if these changes positively affect your library.ashleysommer commentedon Oct 12, 2020
The dependency on
urllib2is not directly fromRDFLib(if you look at the code, you'll see no usages for urllib2 and urllib2 is not in our list of install requirements).@nicholascar is right, it comes from the use of the
requestslibrary. We have recently removed the dependency onrequestsin favor of the newer built-inurllib.requestPython3 module, so if you test onmasteryou will see thaturllib2is no longer brought in.situx commentedon Oct 29, 2020
Hello, as the main developer of the QGIS plugin in question I might shed some light on the details of the aforementioned request.
So I assume SPARQLWrapper uses urllib to do the HTTP request to the SPARQL endpoint.
Is this wrong or is there indeed some way I could use QgsNetworkAccessManager instead?
nicholascar commentedon Oct 30, 2020
Hi Timo,
Yes it does. Both SPARQLWrapper & RDFlib, if doing web requests, use urllib without the option to delegate.
Not directly within SPARQLWrapper but: SPARQLWrapper was written a very long time ago before modern HTTP Python libraries like requests or httpx. I would suggest that to delegate network functions, you keep using RDFlib but just write your SPARQL requests into a requests or httpx-based script and take advantage of their better networking capabilities. There aren't so many benefits that you get from SPARQLWrapper over a few requests/httpx lines of code. I haven't looked into this myself buy presumably QgsNetworkAccessManager can be used by things like requests/httpx.
situx commentedon Oct 30, 2020
Hi Nicolas,
I think it is probably not so easy. I believe QgsNetworkAccessManager will use Qt-builtin network handling.
The issue is rather not to use requests or httpx, because we would like to take advantage of internal QGIS proxy settings which are stored as a QSettings object.
My current workaround for this is to check for QGIS Settings and then apply them in urllib2 for SPARQLWrapper, but it appears that this is not sufficient. I probably have to write some new code using Rdflib and QgsNetworkManager skipping SPARQLWrapper to finally solve this issue.
Probably needs some more investigation.
nicholascar commentedon Oct 30, 2020
This is what I meant! Skip SPARQLWrapper altogether and write your own code on top of RDFlib. If you use requests/httpx for the network calls - and they really should be able to use any proxy settings - you won’t be writing much more code than SPARQLWrapper anyway!
(I do think that eventually we will need to update SPARQLWrapper to use all the modern internet connection options)