In a recent news item, the important Dutch legal knowledge platform Recht.nl announced that they have launched a new case law tracker (‘Jurisprudentie Tracker’).
Until recently, the platform focused on providing legal news and reporting developments of note across legal domains, gathering and mapping relevant legal scholarship in those domains, and functioned as a go-to place for legal vacancies across sectors. Recht.nl claims to have 20,000 subscribers and 4.5 million pageviews per year. Many can access the service for free with an institutional subscription, for instance through a university licence. A personal subscription currently costs €125 per year. Jurisprudentie Tracker is available to all subscribers, providing a quick and easy overview of published case law in the Netherlands. The new tool promises a more convenient way of working with case law and finding relevant legal information. The claimed functionality of the tool has the potential to have significant implications for the legal profession. Therefore, it is important to get an explicit understanding of how the tool works and how it can affect the way that that one interacts with published case law.
The Typology of Legal Technologies
To glean such an understanding we refer to the Typology of the Legal Technologies developed by the COHUBICOL team. The Typology consists of a curated set of legal technologies that were “handpicked to demonstrate the potential impact on legal effect of different types of ‘legal tech’”. Presented as an online tool, it allows the interactive comparison of a select set of applications, scientific papers, and datasets that we consider paradigmatic of the ongoing trend towards automating parts of the law and legal work.
Ultimately, the Typology is intended to be thought of as both a method and a mindset. It aims to provide an approach to gain a deeper understanding of the implications of a transition of law and legal practice as we know it (which relies on text as its underlying technology) towards the use of systems that rely on code and data. To properly appreciate the effects of such a shift for legal protection and the Rule of Law, whether positive or negative, we need to (be able to) investigate such systems critically. In the context of the Typology, we analyse these effects in terms of their ‘potential legal impact’, meaning how the technology might affect how legal states of affairs are (i) recognised by law, (ii) created by law, and (iii) disputes about them adjudicated by law.
This is done in the hope that the questions we have posed and the answers we have formulated can serve as a roadmap to an informed, constructive and open debate on the use of these systems in law, now and in the future.
Claimed rationale and essential features
First, we address the problem that Recht.nl’s new case law tracker is attempting to solve. Dutch judges decide more than a million cases each year, ca. 55,000 of which are published on the official online platform of the Dutch judiciary rechtspraak.nl (approximately a thousand cases each week). To monitor this constant stream of information, Recht.nl’s case law tracker has been designed as a “knowledge tool” with which the user can quickly and more easily take note of all the relevant court decisions published in their domain of interest. The Tracker was therefore intended to be a one-stop dynamic database for all Dutch court decisions, with its claimed rationale and benefits being to increase the manageability of the constant stream of published court decisions.
The claimed essential features of the Tracker are as follows:
- Categorisation of all cases published on Rechtspraak.nl into 18 legal domains and into a further 180 more granular specialist categories, which is a much more detailed categorisation than rechtspraak.nl provides.
- Monitoring and flagging the cases that are being discussed in 200+ open public sources and in more than 125 legal journals. With this feature, Recht.nl claims to be “a legal news resource that covers 180 topics that will make subscriptions to most case law journals redundant”.
- Ranking the most discussed cases, per legal domain and per sub-category within that domain, of the last 12 months. According to Recht.nl, this means you are “up to speed within the blink of an eye with the need-to-know case law within your area of expertise”.
To understand the implications of the tool, we need to address how the system achieves the claimed functionality, and whether those claims can be substantiated.
Substantiation of claims and potential issues
Not much is known publicly about how the system identifies the category to which any given case belongs, how it flags which the cases are being discussed, or which parameters are used to rank the judgements. The main information available for evaluation are promotional leaflets in Dutch and English, and the tracker itself. However, this still gives us a clue towards the potential impact of the design choices and limitations of the tracker:
- Jurisprudentie Tracker states that artificial intelligence is used to classify the case law into 180 categories. It claims to “use Natural Language Processing (NLP) software, which we have ‘trained’ to match texts with specific legal categories”. Such systems can perform well, but are rarely perfect. Additionally, Recht.nl does not report how accurate the categorisation of the cases is. If having a collection of all case law from a specific category is essential, then machine learning generally cannot proved a perfect solution.
- A case appears in the ‘trending tab’ because of a recent mention in the public sources and journals. Similarly to categorisation, it is difficult to assess how well Jurisprudentie Tracker extracts mentions from different resources. If the algorithm fails to detect a mention, the case may simply be ignored. Recht.nl does not provide information on how the references are extracted. It is likely that it is done using regular expressions, which is a way to search for a specific character pattern within the text. We expect that Recht.nl recognises ECLI-numbers, which have a very specific and easily extractable structure. This would suggest that if the resource does not explicitly mention the ECLI number, the reference would not be extracted. If the tracker uses more complex methods to find mentions, for instance by recognising case names, the likelihood of errors and missing mentions is higher.
- According to the platform’s website the ranking of the cases is determined by multiple parameters, and not just the date and number of mentions, however, they do not provide a full list of said parameters, and thus it is impossible to independently assess how this specific ranking algorithm may affect which cases will end up on top of the ranking.
- Currently Recht.nl publishes a set number of ‘trending’ and ‘top’ cases (200 and 40, respectively). Since most of them are only referenced once or twice, it is not clear how the cases are ranked when more than the set number of cases have the same number of mentions. It is likely that it is done based on the date of the publication or reference, however, the more recent reference is not necessarily more relevant than the older one.
- It is likely that the ranking is also affected by which sources are referencing the court decision, this means that the journals are also pre-ranked among themselves. This would mean that the cases cited by a specific legal journal or public resource might appear higher in the ratings than by the others. Given that the ranking of the sources is not public, the tracker may prioritise some publications over the others without disclosing which ones over which ones.
- Jurisprudentie Tracker suggests that it can be used to track discussions about the published cases, however it is only providing the links to publicly-available resources. Their promotional material suggests using the tool will make subscription to most case law journals unnecessary, but without such subscriptions end-users will not actually have access to the comments and discussion. Moreover, without the subscription to the specific resources, it is also impossible to easily verify that the case discussion has indeed been published.
- Just because a case is cited in any public forum does not mean it is necessarily more relevant for a legal professional, especially given that some of mentions of particular cases on Recht.nl include sources such as LinkedIn blog posts, rather than curated case law journals. Jurisprudentie Tracker does not provide the full list of sources that they monitor, which reinforces the uncertainty about the relevance and accuracy of the case ranking based on case law discussions.
Potential legal impact and the effect on legal effect
Given that the Dutch judiciary aims to publish up to 75% of all case law, it would be impossible to navigate such a large amount of information without any type of curation. Curation in one form or another has always existed in the presentation of case law, e.g. Nederlandse Jurisprudentie (NJ), and more recently rechtspraak.nl. Given the number of documents that need to be processed, an automated system that helps end-users select the most relevant cases makes a lot of sense. However, we should be aware of potential inaccuracies as well as the broader implications of using such a system. Given the popularity of the platform and the likely popularity of the new service, legal professionals may come to rely on it excessively, and thus the cases that are selected for ‘trending’ or ‘top’ tabs would be cited even more, while the ones that are not mentioned are more likely to be ignored. Thus, instead of merely recognising the trends, it may start creating them.
Issues such as this, in addition to the drawbacks described in the sections above, can be understood in terms of their potential legal impact. This notion, in turn, hinges on the idea of the effect on legal effect. Here, legal effect can be narrowly understood as the “effects brought about by written and oral speech acts that are recognised by law” and is therefore intimately bound up with the affordances of natural language and the law’s technological articulation in text. The effect on legal effect then denotes the impact of transitions in the mode of existence of law, from natural language to data- or code-driven systems. By analysing legal technologies, like Jurisprudentie Tracker, in these terms, we can better understand whether the design choices that are made change how legal effects are brought about. If this happens, a fundamental characteristic of positive law is changed.
As the Typology FAQ explains, by way of example: “if a lawyer relies on a search application that uses natural language processing techniques, the outputs of that system might lead the lawyer to consider a different notion of what the law is than if she had assessed all the potentially relevant cases manually.” However indirectly, the design of the legal tech thus affects the legal effect of the lawyer’s work (e.g. an argument in court). In the case of legal search engines, suggestions output by the system may inform the work of lawyers and other legal professionals by providing information ranked according to very specific criteria that they do not have control over or even awareness and knowledge of, given that the algorithms behind such rankings often remain undisclosed.
Consequently, there is the potential to alter the legal effects their work brings about in a variety of ways. For one, as mentioned above, the notion of “trending” conflates what is most discussed with what is most relevant. The cases that are most talked about are not necessarily more relevant, but could, for instance, merely be the more controversial, and thus focus the attention on one particular (aspect of a) case while foregoing other potentially (more) legally relevant elements. Therefore, something being a ‘trending’ topic can lead to the ‘loudest’ arguments being given more weight, which may lead to considerably different outcomes if applied, for instance, in court. Moreover, it is difficult to ascertain the selection of case law journals Jurisprudentie Tracker relies on, and therefore also what sources are missed out.
The Typology as a mindset
Features and limitations similar to those of Jurisprudentie Tracker can be found in other systems in the Typology of Legal Technologies. We invite the reader to investigate the topic of search engines for case law further by engaging with the Typology. Click here to explore the similar systems CaseText, Jus Mundi, Moonlit and Westlaw Edge.
Adopting the approach set out by the Typology provides a roadmap to developing a solid framework of questions to assess what one actually requires of an application in Legal Search, taking into account not only users’ and business’ requirements, but also accounting for the potential legal impacts such system may have and for the effect on legal effect.
Discussion