Knora-api-js-lib PR Changing Public API

The is a PR that changes the public API of knora-api-js-lib: https://github.com/dasch-swiss/knora-api-js-lib/pull/169.

@subotic @t.schweizer After Tobias wrote this proposal about ForbiddenResource yesterday, I have discovered just now that the PR#1615 may have a greater impact on knora-ui and knora-app than I (we) thought.

André and I implemented a pagination system to display search results and display the number of results to the user. My question is how deep this knora-api PR and knora-api-js-lib changes will impact the frontend apps and how will we deal with some specific cases like:

when a user has restricted permissions, which number of results and search results should we display? e.g. a user has access to 25 results on 75 in total (due to your user permissions)
The user sees only the number and results he has access to? or Can the user see there are 75 total results but he can access only 25 of them?

How should we handle this kind of situation in the future? Where should I write my questions and comments about an open PR (user story related to a knora-api user story or comment the knora-api user story)? because we need to evaluate the potential breaking changes and changes to fix or implement in knora-ui and/or knora-app…

1 Like

e.g. a user has access to 25 results on 75 in total

This was always the case. It isn’t changed by removing ForbiddenResource.

Yes, it has always been like this.

@flavie.laurens Let’s start from the beginning:

ForbiddenResource was intended to solve two problems:

  1. In Gravsearch, the count query can’t check permissions. So it could be that you do a Gravsearch count query, and it tells you that there are 75 resources that match your query criteria. But when you ask for those resources, Knora sees that you only have permission to view 25 of them. What should Knora do about this?

  2. The contents of the repository can change all the time. So you could do a Gravsearch count query, and it could tell you that there are 75 resources that match your query criteria. Two seconds later, someone adds more resources to the repository, so now there are 80 that match your criteria. Or they delete some resources, so now there are only 10 that match your criteria. Meanwhile, you ask for a page of search results (matching resources), then another page, then another page. How can Knora know whether you have reached the last page? We don’t want to do the count query for every page, because this would be slow.

So the idea of ForbiddenResource was this: Each page of search results has a fixed size (https://docs.knora.org/paradox/03-apis/api-v2/query-language.html#supported-sparql-syntax), e.g. 25. When Knora gets a page of results, it filters them for permissions. Any resource that you don’t have permission to see is replaced by ForbiddenResource (https://docs.knora.org/paradox/03-apis/api-v2/query-language.html#permission-checking). This means that if a page contains exactly 25 results (perhaps including some ForbiddenResource), the client should realise that there could be at least one more page of results available. If a page contains fewer than 25 results, it must be the last page.

Therefore, I assumed that the client code (knora-api-js-lib and/or knora-ui) must already (a) filter out ForbiddenResource so as not to show it to the user, and (b) have some kind of logic like this for processing pages of Gravsearch results:

if (page.count == 25) {
   mayHaveMoreResults = true;
}

But there are some problems with this approach:

  1. Only Gravsearch uses ForbiddenResource. The rest of Knora API v2 has other ways of indicating that you don’t have permission to see something. This is inconsistent.

  2. The client has to know the page size, but there is no way to ask Knora about this.

  3. What if a resource contains a nested (dependent) resource that the user doesn’t have permission to see?

  4. What about values that the user doesn’t have permission to see?

Therefore, on 2 March I posted this topic on DaSCH Discuss to talk about different possible ways of making this all consistent:

I asked, “Does anyone have an opinion about this?” I figured that anyone who uses Knora API v2 might have an opinion. The only person who replied was @loic.jaouen.

After some discussion, on 4 March I suggested getting rid of ForbiddenResource:

I said I thought this would be the simplest approach. Nobody objected, so I did Knora PR 1615.

As far as I can see, the effect on client code should be very small:

  • You no longer need to filter out ForbiddenResource.
  • To find out whether there might be another page of results, you no longer need to know the page size. You just need to look for the boolean knora-api:mayHaveMoreResults.

It seemed to me that this would just be a few lines of code to change.

I thought that this was one of the reasons why we set up DaSCH Discuss. Everyone said that they wanted communication as early as possible about potential Knora API changes. It seems to me that in this case, I did exactly that: I called for advice on possible changes, three weeks before choosing an approach and doing the implementation. I don’t think I could possibly have made the situation any clearer.

As far as I understand, Scrum sprint planning meetings are not intended for these kinds of discussions; they’re just about picking user stories for the next sprint. But how do we get the user stories? I think we need to keep using DaSCH Discuss (or something like it) to have these discussions online, in writing. But that only works if people read and participate in discussions.

1 Like

To find out whether there might be another page of results, you no longer need to know the page size. You just need to look for the boolean knora-api:mayHaveMoreResults .

I love your description of the idea behind having ForbiddenResource, the problems it caused, and the solution.
As far as I see, this change will not break BEOL, since we do not do anything with ForbiddenResources there and we do not count the number of results per page. By default in the config file, we have set the page size=25.
Currently, BEOL loads search results with the help of offset:

      if (this.numberOfAllResults > 0) {
        // offset is 0-based
        // if numberOfAllResults equals the pagingLimit=25, the max. offset is 0
        this.maxOffset = Math.floor((this.numberOfAllResults - 1) / this._appInitService.getSettings().pagingLimit);
    } else {
        this.maxOffset = 0;
    }

and loads more results if offset<maxOffset.

I wonder if the existing applications like BEOL should be refactored/updated to check for mayHaveMoreResults.

Thanks! I guess BEOL hasn’t needed to worry about permissions at all so far, because everything is publicly viewable, and the data never changes. This means that in BEOL, you can rely on the result of the count query. But that won’t work in the general case.

I suppose that someday it will be possible to add/remove data in BEOL via the user interface. So it would probably be best to change BEOL to check for mayHaveMoreResults instead of relying on the count query.

True, I will add a user story for it so that we can plan the update of BEOL for a sprint.

I think we should be fine because the information comes from the count query (https://github.com/dasch-swiss/knora-api/pull/1615#issuecomment-597172324)

As mentioned above (both by me and Ben), BEOL is fine for now working only with count query since all its resources are public. This would be wrong as soon as resources are added to it that have other permissions. Therefore, over time BEOL should be refactored to use mayHaveMoreResults.

I think if we had resources that weren’t public users could get even no results for some pages (before they would have gotten just forbidden resources) but I think the calculation of the max offset would still be valid, simply because getting no results for a page doesn’t mean that there are not more results to be fetched. Maybe some day Knora should try to fill in resources when hits are removed but that would mean that the whole prequery process would have to be redesigned (mainly permissions would have to be evaluated by the triplestore, I think Ben is looking into this).

Therefore, over time BEOL should be refactored to use mayHaveMoreResults .

Eventually BEOL has to use a newer version of the existing Knora-ui that uses knora-api-js-lib or switch to the new knora-ui. The new flag will be handled in knora-api-js-lib.

Yes and this would also mean that the whole permissions model would have to be changed, so existing projects would need to be adapted. Maybe not so bad if the project is finished, because as Lukas said, the data in the DaSCH is eventually supposed to be public.

Yes and this would also mean that the whole permissions model would have to be changed, so existing projects would need to be adapted.

That would be a hell of a work!

And even if a project is finished, it doesn’t mean that all data should automatically be made public : some copyrights and consent agreements remain valid even after the end of a research project and access to this data should remain restricted.

Thanks for pointing that out. I wonder if we could find some compromise or hybrid system, which would keep the existing permissions model and still do some filtering in SPARQL. Definitely this needs more thought.

I shortly talked to Lukas about this a while back. Mine opinion is that we need to either simplify the permission model or we have to extend it. The current implementation is neither here nor there. At the moment I don’t have a concrete idea on how to proceed.

If we want to do it in a non-breaking way, then it will be difficult to simplify it.

I remember that some of our user requirements contributed significantly to add complexity of the original permission model. Since some of this user requirements were further shared by others, I am not sure that simplifying the mode is a good option… Because then, we wouldn’t meet the needs of our users.

1 Like