Legal Innovation Lab Wales

June 2023

Final Evaluation Final Report

Contents

1. Introduction and context .............................................................................................................. 1

2. Assessment of inputs and activities ............................................................................................. 9

3. Assessment of Cross-Cutting Themes........................................................................................ 24

4. Assessment of core benefits....................................................................................................... 29

5. Assessment of wider benefits and assessment against the logic model............................... 37

6. Conclusions and recommendations ......................................................................................... 46

Annex A: Glossary ......................................................................................................................... A-1

Annex B: Strand level logic models..............................................................................................B-1

Contact:

Approved by:

Joanne Barber

Joe Duggett

Tel: 0161 240 1683

Director

email: jbarber@sqw.co.uk

Date: 08/06/2023

Disclaimer This report takes into account the particular instructions and requirements of our client. It is not intended for, and should not be relied upon by, any third party and no responsibility is undertaken to any third party. Whilst SQW has used reasonable care and skill throughout, it is unable to warrant either the accuracy or completeness of information supplied by the client or third parties, and it does not accept responsibility for any legal, commercial or other consequences that arise from its use.

Legal Innovation Lab Wales

1. Introduction and context

1.1 SQW Ltd (SQW) was appointed by Swansea University in July 2022 to undertake a Final Evaluation of the Legal Innovation Lab Wales operation. This follows Inception and Mid-Term Evaluations completed in 2020 and 2022 respectively.

About the Legal Innovation Lab Wales

1.2 The Legal Innovation Lab Wales (‘LILW’) was a £4.95 m, part-ERDF funded operation under the West Wales and the Valleys programme 2014-20. Specifically, the operation was funded under Priority Axis 1: Research and Innovation, and is expected to deliver against Specific Objective 1.1: to, “ increase the success of Welsh research institutions in attracting competitive and private research funding ” , and the associated result indicator of an increase in research income amongst HEIs.

1.3 Located within the School of Law’s Digital Technium building, the operation combined new purpose-built facilities, with additional expert research and technical capacity in three areas:

• LegalTech : focused on supporting the development of legal innovations, prototypes and toolkits, underpinned by research into artificial intelligence (AI) and machine learning.

• Cyber Threats : focused on supporting research to advance understanding of cybercrime, terrorist groups’ exploitation of new technologies, online grooming, misinformation, fake news and criminal internet use, and the development of cyber-security/terrorism prototypes/toolkits.

• Law Clinic : designed to support the access to justice agenda through facilitating the trialling and scoping of LegalTech innovations to address real world legal issues.

1.4 Delivery of the operation commenced in August 2019, with a formal project announcement by Welsh Government and Swansea University in October 2019. The close date for the operation is June 2023.

1.5 The operation included both capital and revenue elements:

• Capital funding from ERDF (£0.75m), to fund the design, procurement, and construction of a c.700 sqm facility comprising an: AI and Law Laboratory, a Cyber Threats Suite, a LegalTech R&D Centre, a Law Clinic (part-eligible for ERDF support 1 ), and an Executive Boardroom.

1 The focus of the project in relation to the Law Clinic was to fund the development of the space for prototype testing, not its wider remit (provision of pro bono legal advice), which is not within the scope of the ERDF eligible project.

Legal Innovation Lab Wales

• Revenue funding from ERDF (£2.6m) and Swansea University (£1.6m) to support the appointment of researchers in LegalTech and Cyber Threats, a software development team, and a project management team.

Evaluation purpose and coverage

1.6 Consistent with the requirements of ERDF funding, Swansea University committed to undertaking a robust evaluation of the operation. As such, the University commissioned SQW in February 2020 to lead Inception and Mid-Term Evaluations of the operation. The University then subsequently, and separately, commissioned SQW to lead a Final Evaluation in July 2022 (the subject of this report). 1.7 Completed in June 2020, the Inception Evaluation focused on the development of a logic model/theory of change for the operation, clarifying the data to be collected to support the programme of evaluation, and confirming the approach to the Mid-Term Evaluation. 1.8 The Mid-Term Evaluation, completed in January 2022, focused on assessing how effectively the operation was progressing against its aims and objectives at an interim point. Overall, the findings at the Mid-Term Evaluation stage were encouraging, indicating strong progress in delivering against its Business Plan. Key messages included:

• the operation had delivered a substantial amount of activity in a challenging context and was broadly on course to deliver outputs and outcomes in line with expectations

• the Covid-19 pandemic had inevitably influenced delivery, leading to a substantial underspend against budget and the need to re-profile the budget reducing the total ERDF grant from an initial £4m to £3.35m • securing research funding had proved challenging (owing to delays in the appointment of research staff and to the completion of the Cyber Threats Suite), with modest progress made at that stage against the ERDF target of £3.4m in research funding secured by the close of the operation • a range of collaborative R&D projects had progressed successfully, leading to tangible prototypes to be tested and evaluated with users, and in some cases platforms which were available for use

• there was evidence of wider emerging benefits in relation to enhanced partnerships, networks, and behaviours supporting collaborative R&D.

1.9 Some two years on from the Mid-Term Evaluation, the core purpose of this Final Evaluation is to provide an assessment of the impact of the operation as it nears the end of its ERDF period against the aims and objectives set out in the Business Plan. This includes an assessment of the extent to which the benefits realised are ‘additional’ compared to what

Legal Innovation Lab Wales

would have happened without the LILW The Final Evaluation also provides an updated assessment on delivery by the operation against the ERDF Cross Cutting Themes (CCTs), considers what has worked well and less well in delivery, and considers implications for the sustainability/priorities of the operation following the close of the ERDF period. 1.10 The Final Evaluation commenced in September 2022, with primary research completed over March-May 2023. The report was drafted in May 2023, ahead of the formal completion of the operation in June 2023. As such, the Final Evaluation is based on evidence provided as the operation drew to a close, but whilst activity was on-going. This needs to be taken into account when considering the evidence on progress in terms of expenditure and in delivery of ERDF target indicators, and the wider final outcomes and impacts.

Evaluation methodology

Logic model

1.11 Logic models are useful devices to inform evaluation, because they encourage thinking about the steps required for an intervention to have its desired effects, and the nature of effects that can be covered in evaluation. A logic model articulates explicitly the relationship between inputs, activities, outputs, outcomes and impacts:

• inputs are the resources used by the intervention, including ERDF funding

• activities are those tasks undertaken by the intervention

• outputs are the readily measurable results of those activities

• outcomes are the changes in the behaviours and performance of those involved with an intervention

• impacts are the ultimate benefits attributable to the intervention and changes in wider conditions, both directly and indirectly.

1.12 Logic models also set out the external factors which are expected to influence activity (either positively or negatively), and therefore the realisation of anticipated outputs, outcomes and impacts. 1.13 T he logic model and associated narrative ‘theory of change’ developed during the Inception Evaluation (and reflecting the position before the project started) are set out below. This Final Evaluation seeks to test the logic model and underpinning theory of change, particularly the extent to which the LILW has delivered the outputs, outcomes and impacts anticipated.

Legal Innovation Lab Wales

Theory of change for the LILW Swansea University’s Hillary Rodham Clinton School of Law, is an internationally leading hub for research into cyber threats, and a UK-leading institution in the emerging field of LegalTech research and innovation. The application of technologies such as AI and blockchain are becoming increasingly important in legal practice and national security. However, existing infrastructure barriers and capacity limitations at the School of Law act as a constraint to the University further pursuing world-leading research and securing competitive research income across the LegalTech and cyber threats disciplines. The Law School has no existing facilities to support legal innovation, research in legal informatics or the application of AI in a legal context, or the facilities necessary to work on secure or sensitive data, nor to meet in a secure environment with partners from law enforcement or international security agencies. Further, the University lacks research capacity across the across the legal services, computational social sciences, and cyber/counterterrorism spectrum. The LILW will enable the Law School to enhance its internationally leading position in LegalTech, cyber threats and associated technologies (AI, blockchain etc.), accelerating innovation and research for both academics and sector organisations across Wales, contributing to the recruitment and retention of academic talent in Wales, and supporting the development of a robust and vibrant LegalTech sector in Wales alongside the access to justice agenda. LILW will provide Swansea with a significant competitive advantage in securing competitive research income from public and private sources. These positive effects will be achieved through the development of a purpose-built LILW suite within the Law School’s Digital Technium building. This will be complemented by the expansion of the existing research and technical capacity by recruiting additional researchers and a DevOps software development team to work with academic and commercial partners to design, develop and implement software solutions. Engagement with external partners will be facilitated by the appointment of a LegalTech Engagement Officer, and funding to enable researchers to identify, scope and submit grant applications and secure competitive research funding.

Legal Innovation Lab Wales

Figure 1-1: Operation level logic model

Activities

Impacts

Remit

Context and rationale • Swansea University’s School of Law is an internationally leading hub for research into cyber threats, and a UK- leading institution in the emerging field of LegalTechresearch and innovation. • However, barriers and capacity constraints are preventing the School from maximising fully its potential ➢ the number of researchers in relevant legal technology disciplines ➢ technical and software development expertise to prototype and test technical solutions ➢ physical constraints, with insufficient space for collaborative R&D activities, and cyber threats research • Limited resources and capacity in the Welsh legal sector to invest in and adopt next generation technologies to

Inputs Total project funding of £5.6m • Capital funding ➢ ERDF: £1.2m • Revenue funding ➢ ERDF: £2.8m

ERDF indicators • Improved research infrastructure facilities • Researchers working in improved research infrastructure facilities • New researchers in supported entities • Partners cooperating on research projects • Research funding secured

➢ Swansea University: £1.6m

Activities • Capital investment , in the scoping, design, procurement and construction of the Legal Innovation Lab Wales suite covering three constituent strands ➢ Strand 1: LegalTech, AI & Law Lab, Legal R&D Centre, & Executive Boardroom ➢ Strand 2: Law Clinic, part-funded Law Clinic facility ➢ Strand 3: Cyber Threats, Cyber Threats Research Suite • Appointment of research staff (x7.6 FTEs), across in LegalTech (Strand 1) and Cyber Threats (Strand 3), to deliver collaborative research with partners, and apply for and deliver research funding from competitive sources (public and private) • Appointment of a Software Development Team (x3 FTEs) to develop prototypes for new and innovative legal technology products and services, working across the three Strands (including supporting the trialling of LegalTech prototypes in the Law Clinic) • Appointment of a project management team to ensure successful delivery of the project, including a Project Manager and LegalTechEngagement Officer • CCT activities in construction and operational phases

… which will be realised through all three Strands of project activity

Wider outputs and outcomes • Academic benefits , including advances in relevant disciplines, development of prototypes and toolkits, enhanced research and teaching outcomes, enhanced reputation of the Law School • Business and societal benefits , including the development of concepts/ideas for new technology applications, knowledge transfer leading to post-engagement commercialisation opportunities, supporting enhanced access to justice for communities, and contribution to policy and security regimes • Collaboration benefits , including new and enhanced partnerships, enhanced collaboration behaviours with industry and other research centres, and ecosystem development at the intersection of law and technology … which will vary in focus and scope across the three Strands of project activity

support commercial growth and improved legal service delivery.

Overarching strategic objectives • Increase the success of Welsh research institutions in attracting competitive and private research funding (ERDF Priority 1, SO1) • Establish a bespoke niche research facility, located within an expanded Law School that will ➢ drive the development of a robust and vibrant LegalTech sector in Wales, delivering economic impact and enabling growth in the Welsh legal services sector ➢ support the development of toolkits that enhance the business of law and access to justice, delivering societal impacts

➢ enable the development of technological platforms and frameworks to support counterterrorism, defence and security ➢ foster new opportunities for interdisciplinary and industrial collaboration

Legal Innovation Lab Wales

Key assumptions underpinning the ToC

Project wide  Recruitment & retention of academics/staff of high calibre staff with the necessary expertise  Understanding amongst stakeholders/partners regarding what does & does not fall within the remit of LILW  Sufficient sources of research & innovation income, & linked to this, an adequate number of successful research funding bids secured by LILW  LILW offers capabilities at the forefront of research & innovation in LegalTech/cyber threats throughout project delivery and beyond. The longer-term sustainability of LILW & its ability to deliver final outcomes will require additional investment to ensure that staff are retained & the facility/equipment remains maintained & up-to-date.  LILW is integral to and aligned with the strategic positioning & priorities of the wider Law School, ensuring strategic support & commitment from leadership  Interest, demand & culture of cross-disciplinary collaborations with other university disciplines & research centres  There is demand from industry for collaborative research with the LILW  Engagement from the wider research base & industry to utilise & engage in collaborative research with LILW  Effective relationships/synergies between the three strands of the project (LegalTech, Cyber Threats, & the Law Clinic), which are mutually re-enforcing & provide opportunity for cross-strand activity  There will be feedback loops, with further funding leveraged from new opportunities & collaborations, findings leading to further research activity, & changes in staff behaviours & attitudes leading to new activities & opportunities LegalTech  Willingness & capacity within the legal community to explore/adopt LegalTech  Local ecosystem/infrastructure that supports start ups  Economic conditions that support entrepreneurialism & investment in new technologies Cyber threats  Engagement from partners from law enforcement or international security agencies  Infrastructure & expertise align with the current cyber threats challenges  Research suite provides sufficient level of security – consistent with any necessary legal or regulatory requirements – to enable on-site data interrogation/collaboration activity Law Clinic  Community assured that the Law Clinic offers professional expertise in law  Partners (NGOs, charities, law firms) have the capacity to engage in collaborations  Steady demand for legal advice from the Clinic throughout delivery to support the trialling of innovations External factors influencing outcomes  Unforeseen delays to project delivery owing to COVID-19; including delays to staff recruitment, the construction of the cyber threats research suite (now due to be completed early 2021 rather than summer 2020 as planned) & the ability of the project to deliver collaborative projects.  Wider economic conditions (particularly in the context of COVID-19) & the policy/regulatory agendas will influence levels of industry capacity & demand for collaborative research & innovation.  The prioritisation of research & innovation funding & the level of competition , including Research Council / UKRI priorities & investment decisions related to legal tech, cyber threats & related disciplines.  Investments in equivalent facilities elsewhere , which will impact on the extent to which the LILW provides a unique offer to collaborators & industry. Significant investments by competitors in the Wales/UK may impact on the ability of the project to secure competitive research funding, & the retention and attraction of talent, but also provide opportunities for collaboration  The UK’s departure from the EU will change access to research funding & movement of people; this may influence the ability of the project to recruit & retain talented staff, & participation in EU-funded research programmes. Linked to this, WEFO-funded projects need to be completed by June 2023 leaving no scope for a change to project end-date  The location of LILW is not located in London - the ‘legal hub’ of the UK – therefore it may be challenging to attract high calibre academics & national/international interest from stakeholders.

Source: SQW

Legal Innovation Lab Wales

Sources of evidence

1.14 The Final Evaluation adopted a mixed methods approach, comprising both desk-based and primary research. The desk-based research included a review of project documents, and data analysis including monitoring data, expenditure, and performance against targets. The latest monitoring data submitted by the LILW and approved by WEFO available for the Final Evaluation covered the period to January 2023. 1.15 The primary research was divided into five strands (see Table 1-1). In total, 38 individuals were consulted for the evaluation, providing a broad and robust evidence base, and a wide range of different perspectives with different types/levels of engagement with the operation.

Table 1-1: Primary research Research strand

Main areas of focus

Project management and leadership (x7)

Consultations with project management and leadership

• Overall performance of the LILW against original aims & objectives

• Effectiveness of LILW management & monitoring processes

• Outcomes & impacts, including any unanticipated effects

• Lessons & reflections on delivery against Cross-Cutting Themes

• Key lessons learned from delivery & implications for the future

LILW academics / researchers (x13)

Consultations with academics and researchers, covering both LegalTech and Cyber Threats

Role and nature of activity undertaken

•

• Effects of the LILW on research activity and funding

Effects of the LILW on collaboration

•

Personal and wider benefits of the LILW

•

• Lessons & reflections on delivery against Cross-Cutting Themes

• Key lessons learned from delivery & priorities for the future

Software Development Team (x4)

Consultations with Lead Developer and all three current Developers

Role and nature of activity undertaken

•

• Lessons & reflections on delivery against Cross-Cutting Themes

• Key lessons learned from delivery & priorities for the future

Partners cooperating in research projects (x5)

Consultations with partners cooperating in research projects with the operation including private, public and third sector organisations

• Background/rationale for engagement with the LILW

• Nature of activity undertaken with the LILW

• Observed / expected benefits as a result of the engagement

Legal Innovation Lab Wales

Research strand

Main areas of focus

Additionality in activity & outcomes

•

Reflections and key lessons

•

Priorities for the LILW in future

•

Strategic stakeholders (x9)

Consultations with internal (x7) and external stakeholders (x2), including representatives from (other) departments/institutes in Swansea University, and external consultees from other organisations

• Engagement with the LILW, and any benefits realised

Strategic alignment

•

• What has worked well and less well overall

Priorities for the LILW in future

•

Source: SQW

Report structure

1.16 The reminder of the report is structured as follows:

• Section 2: Assessment of inputs and activities

• Section 3: Assessment of Cross-Cutting Themes

Section 4: Assessment of core benefits

•

• Section 5: Assessment of wider benefits and assessment against the logic model

• Section 6: Conclusions and recommendations

1.17 There are two supporting annexes: glossary and strand level logic models.

Legal Innovation Lab Wales

2. Assessment of inputs and activities

Overview of delivery stages

2.1 The operation commenced in August 2019, with a formal project announcement by Welsh Government and Swansea University in October 2019. A timeline showing both actual and planned milestones for the operation is illustrated in Figure 2-1. Overall, timings have remained broadly as set out in the original Business Plan, with two notable exceptions (the reason for which are discussed below):

• the delivery of elements of the physical infrastructure, with the completion of the Cyber Threats Suite delayed by over a year from summer 2020 to winter 2021

• staff recruitment was due to be completed by September 2020, however, recruitment challenges meant that in practice, the full scope of appointments had not been made by this point, and recruitment was ongoing for much of the operation’s lifetime.

Figure 2-1: Timeline of key milestone for the LILW

Source: SQW manipulation from project documentation and consultation evidence

Expenditure

2.2 The original budget for the operation was £5.6m, comprising £4.0m from ERDF and £1.6m in match funding from Swansea University. However, by mid-2021 approaching two years into delivery, the project was significantly underspent compared to profile, owing principally to:

Legal Innovation Lab Wales

• delays in the construction of the Cyber Threats facilities combined with lower than anticipated construction costs

• the impact of recruitment challenges on revenue spend

• the impact of the Covid-19 related restrictions (e.g. on travel / event expenses).

2.3 The operation identified a range of approaches to manage the underspend including the introduction of a new strand of activity (a ‘ Virtual LegalTech platform ’) , and changes to the staffing structure (see below for further details). However, it was also considered necessary to agree a re-profile with WEFO given the scale of the underspend. WEFO approved the re- profile in September 2021. As a result, the operation’s total ERDF capital funding commitment decreased from £1.2m to £0.75m (see Table 2-1). The revenue expenditure was not subject to re-profiling, although Swansea University’s match decreased slightly due to a reduction in the flat rate 2 .

Table 2-1: Original and re-profiled total anticipated costs Original total budget

Re-profiled total budget

ERDF

Total

ERDF

Total

Capital

£1.2m

£0.75m

Revenue

£2.8m

£1.6m

£4.4m

£2.60m

£1.58m

£4.18m

Total

£4.0m

£1.6m

£5.6m

£3.35m

£1.58m

£4.93m

Source: Business Plan, September 2019 and Re-profile Addendum, September 2021

2.4 By the Mid-Term Evaluation, actual expenditure (as at October 2021) was £1.6m, marginally below planned expenditure by that point (8% below re-profiled budget). At that stage, the majority of outstanding expenditure was revenue expenditure on staffing costs. 2.5 At this Final Evaluation stage, by January 2023, overall eligible expenditure was £3.7m, with an underspend of 13% against the planned expenditure of £4.26m. This is a material underspend, in aggregate terms, approximately £560k. 2.6 Table 2-2 provides a breakdown of expenditure by category. At January 2023, the underspend was pronounced in aggregate terms on professional services and ICT. The latest Progress Report (November 2022-January 2023) sets out that the main reasons for the underspend across these two categories are as follows:

• Professional services expenditure is below budget largely due to delays progressing the development of the Virtual LegalTech platform. Also, procurement and implementation

2 Note, the amount of SU match funding for staff remained unchanged, however, the flat rate (a simplified category for indirect costs at 25% on all revenue costs of the operation) commitment decreased reflecting the decrease in the total revenue budget.

Legal Innovation Lab Wales

of the Amazon Trusted Research Environment (TRE) had been delayed and this had not yet been claimed.

• ICT expenditure is below budget because the Amazon TRE being procured for the PASF will be claimed under the Professional Services Cost heading, rather than ICT as planned. Following the later than planned implementation of the TRE, the operation will then be able to procure the necessary hardware.

Table 2-2: Expenditure to January 2023 (£000s) Total expenditure to Jan 2023 Budget to Jan 2023

Variance to Jan 2023

Percentage of budget spent to Jan 2023

Estates

739

745

99%

ICT

137

249

112

55%

Staff

2,200

2,219

99%

Administration

100%

Marketing / Promotion

33%

Travel

56%

Training Costs

35%

Professional services

211

180

15%

Flat rate

508

703

195

72%

Total

3,696

4,259

563

87%

Source: LILW Variance by Claim (includes ERDF and SU match funding)

2.7 Error! Reference source not found. shows the variance between actual cumulative expenditure and budgeted expenditure by claim period. To the point of the re-profile in September 2021, there was a consistent and substantive underspend. Positively, since the re- profile, actual expenditure has aligned more closely overall with the planned expenditure. However, this said, there has been a consistent and material underspend (from 13% to 19% underspend) since the Mid-Term Evaluation point. Whilst this validates the case for the re- profile in late-2021, it suggests that the revised financial profile for operation was still not fully realistic and deliverable over its final 18 months of delivery.

Legal Innovation Lab Wales

Figure 2-2: Actual cumulative expenditure versus budget per claim period

Source: SQW analysis of data from LILW

2.8 The data to the end of January 2023 suggest that of the total LILW budget of c.£4.9m, approximately £1.2m of expenditure will be needed in the final five months of the operational period. The expectation from the project management team was that this would not be spent in full, with an anticipated underspend of circa £800k. By spend category, professional services, ICT, and staff are likely to account for the largest surplus spend in aggregate terms. However, this will need to be confirmed following the formal closure of the operation in June 2023.

Activities

Construction of the physical infrastructure

2.9 Construction of physical infrastructure commenced in 2019. The Law Clinic and Executive Boardroom opened in September 2019, followed by the LegalTech innovation facilities in March 2020; comprising the AI and Law Lab, and LegalTech R&D Centre respectively. 2.10 As noted in the Mid-Term Evaluation, construction of the Cyber Threats Suite fell behind schedule, and was impacted by the Covid-19 pandemic, and associated restrictions. Further, procurement challenges and the complexity of designing the space to adhere to Police- Assured Secure Facilities (PASF) status also led to delays. The construction of the Cyber Threats Research Suite was successfully completed in December 2021. This was around a year later than planned. However, given the challenges associated with the pandemic (both directly in relation to construction, and indirectly regarding effects and pressures on working

Legal Innovation Lab Wales

patterns and behaviours), and the need to ensure that the specification of the facility met PASF standards (which will be crucial over the long-term, enabling the University to handle secure or sensitive data), this is not considered unreasonable.

2.11 At the point of this Final Evaluation, the PASF accreditation process is ongoing, with external assessment for the accreditation expected in June 2023. A working group was established to support the process; however, this was temporarily suspended due to a change in Project Manager in Summer 2022. Subsequently, owing to the volume of work required to secure the accreditation, the operation sought to recruit a Data and Collaborations Officer in late-2022 to drive the implementation of the PASF. However, no suitable candidates were identified and the position was not filled, which led to some further delays. In February 2023, implementation of a cloud-based

This [Cyber Threats Suite] helps makes us unique. We have strong security features, and this help makes Swansea a good place to undertake really sensitive data research

Academic consultee

platform (Amazon Trusted Research Environment) was completed. Currently (at May/June 2023) the focus is on developing the necessary processes, policies, and collaboration required amongst stakeholders, prior to the external assessment. Images of the facilities are set out in Figure 2-3 below. 2.12 Consultees reported that the delay to the completion of the Cyber Threats Suite has led to some adverse effects on the outcomes realised by the operation by this stage; put simply, researchers have not yet been able to secure and deliver projects requiring the use of highly sensitive data that the facilities will allow. 2.13 However, positively, consultees highlighted that the accreditation will provide Swansea University with a distinctive and differentiated offer relative to competitor institutions, which will be of significant value over the longer-term in securing research grants and engaging with key partners across the cyber threats and security landscape. In this sense whilst the delays to the completion of the facility are not optimal, over the longer-term the benefits will be realised following a successful accreditation, and the additional time required to position the facilities to secure this accreditation is recognised as appropriate.

Legal Innovation Lab Wales

Figure 2-3: Pictures of the Law Clinic (top left), LegalTech R&D Centre (top and bottom right), and Cyber Threats Suite (bottom left)

Source: Swansea University, LILW

Recruitment activity

2.14 Reflecting the breakdown of expenditure noted above, core to the delivery of the operation was the appointment of academics/researchers (also an ERDF indicator, see Section 4), software developers, and a project team to support/manage the delivery of the operation. 2.15 As also noted above, recruitment activity has been on-going throughout delivery, which has reflected both challenges in appointing individuals for the roles intended, and retaining individuals in those roles; taken together, this has led to some changes in roles as well as individuals. As such, over the lifetime of the operation, the team structure and specific appointments have evolved, providing a shifting and complex landscape in relation to the operation’s activity.

2.16 Importantly, key changes were formalised as part of the operation’s re -profile in September 2021. The key changes were as follows:

• Changes to the LegalTech research team included increasing the Associate Professor post from 0.6 to 0.8 FTE and hiring one Post Doctorate Research Assistant (Research Officer), plus two additional Research Assistants (rather than the additional Senior Lecturer

Legal Innovation Lab Wales

proposed originally). The changes were in response to unsuccessful recruitment drives for the original posts.

• A Chair role for LegalTech (0.2 FTE) was not appointed and a decision was made not to seek to fill this role (with this position removed from the organisational chart in the August – October 2022 Progress Report).

• An additional Software Developer was appointed in response to strong demand for support.

• The project team was expanded with the appointment of three additional posts: a Project Administrator, Research Funding Officer, and an additional LegalTech Engagement Officer. However, the original Engagement Officer left in November 2021, coinciding with the new Engagement Officer joining. Instead of hiring a second Engagement Officer, the decision was taken to appoint a Project Officer to drive development of the Virtual LegalTech platform, however, two recruitment rounds were unsuccessful, so the role was not filled.

2.17 At the Final Evaluation stage (in May 2023), the operation included the following staffing structure:

• Two research teams (LegalTech and Cyber Threats) primarily responsible for conducting collaborative research with partners, applying for grant funding, and producing academic outputs. It is highlighted that the Cyber Treats team are part of the broader well-established Cyber Threats Research Centre (CYTREC) at the University, an interdisciplinary centre with experts spanning law, criminology, political science, linguistics and psychology. ➢ LegalTech: an Associate Professor/Co-Principal Investigator (1 FTE), a Senior Lecturer (1 FTE), a Research Officer (1 FTE), and three Research Assistants (x4 FTEs). 7 FTEs in total ➢ Cyber Threats: a Chair (0.2 FTE), Associate Professor (0.6 FTE), Senior Lecturer (0.6 FTE), two Lecturers (each 0.6 FTE), Post Doc Research Officer (1 FTE), and two Research Assistants (x2 FTE). The Cyber Threats research team is led by the Co- Principal Investigator, who is not supported by ERDF monies. 5.6 FTEs in total

• A Software Development Team with responsibility for developing and trialling prototypes for new and innovative legal technology products and services.

➢ a Lead Software Developer (1 FTE) and three Software Developers (x3 FTEs). 4 FTEs in total

• A project team responsibly for the day-to-day delivery of the project comprising,

Legal Innovation Lab Wales

➢ a Project Manager (1 FTE), Project Coordinator (1 FTE), LegalTech Engagement Officer (1 FTE), Research Funding Officer (0.8 FTE), Research Development Officer (0.4 FTE) and REIS Support Officer (0.4 FTE). A Project Administrator was also in post to February 2023; given the short period of delivery remaining a replacement was not appointed. The project team was supported by the two Co-Principal Investigators, and the Heads of the School of Law and Faculty of Humanities and Social Sciences (the latter not supported by ERDF monies). 4.6 FTEs in total

2.18 As indicated, overall, in its final months of delivery the operation was supporting directly 21.2 FTEs, with additional inputs provided by other senior staff at the University.

Delivery of collaborative projects

2.19 The operation has developed and delivered collaborative projects with a variety of partners including private sector companies, third sector organisations, sector bodies, and other HEIs. For example, data from May 2023 indicate that the operation has worked on over 15 collaborative projects across both the Cyber Threats and LegalTech strands of activity with partners, including those that have sought and/or secured external funding. Each project is distinct and tailored to the specific issue / opportunity that it is seeking to address; there is no ‘standard’ activity delivered by the operation.

2.20 Reflecting this, and demonstrating the range of activity progressed, Figure 2-4 provides a selection of examples of collaborative projects involving members of the LILW team.

Figure 2-4: Examples of Cyber Threats and LegalTech collaborative projects

Legal Innovation Lab Wales

Source: SQW analysis of consultation evidence, LILW Progress Reports, and the LILW website

2.21 The operation has a lso supported the delivery of several ‘ strategic ’ projects in the LegalTech strand of activity, where the Lab has sought to develop tools and resources that can be used to support enhanced understanding and use of LegalTech innovations and opportunities. 2.22 This has included a ‘ Virtual LegalTech Lab platform ’ , which was identified as a new strand of activity as part of the re-profiling process. Led by LILW, in collaboration with the Solicitors Regulation Authority (SRA), and Cambridge and Edinburgh universities, the platform is designed to connect LegalTech professionals in the UK and beyond, supporting the co- creation of LegalTech applications through a model of open innovation. It will address key challenges in the use of legal technology and developing apps that will allow all Welsh and UK law firms to access essential LegalTech tools to enable their effective participation in LegalTech. 2.23 Delivery of the Virtual Lab was ongoing at the Final Evaluation stage, and it not anticipated that the platform will be fully operational by the end of operation as planned. There have been two main strands of activity to date: (i) researching and producing an academic publication setting out the rationale for the platform and (ii) working with a developer towards a prototype to be completed by June 2023. On the latter, the University has worked with an external developer on a similar platform already, and is hoping to utilise this platform with some slight customisations to deliver a proof of concept, to help encourage stakeholder buy- in and drive further discussions.

Legal Innovation Lab Wales

2.24 Progressing the platform has been challenging. There was strong interest initially from several other institutions (notably Cambridge, Ulster, and Edinburgh), but it was reported to the evaluators that formalising this through a collaboration agreement, and subsequently co- designing the platform in partnership was not straightforward . Ulster’s involvement in the project ceased due to major structural changes in staffing at the institution. Further, two recruitment rounds to appoint a Project Officer to provide additional capacity to lead on developing the platform were unsuccessful. In addition, progress on the platform was impacted by the handover process associated with a change in Project Manager, although work recommenced subsequently. Given this range of challenges, the progress made at this stage by the activity is encouraging, although the successful delivery and subsequent usage and impact of the platform remains uncertain at this stage. Given the potential of the platform to deliver important legacy benefits from the operation, it is important that the University explores possible funding mechanisms to sustain delivery going forward.

2.25 Other LegalTech strategic projects include:

• Wales LegalTech start-up map: The goal of the map is to outline Welsh LegalTech innovation, understand the opportunities and potential challenges of being in LegalTech in Wales and influence policies around LegalTech entrepreneurship in Wales. The LegalTech research team conducted interview research with start-ups and law firms to inform the map, and key findings from the final report will be delivered at a LawTech UK 3 event in June. • The LegalTech Observatory: The observatory will act as a hub for news and updates on LegalTech, hosted online. Lab research projects, original content on LegalTech issues, and external projects and LegalTech news will be part of the Observatory. Posts will be organised chronologically allowing the evolution of LegalTech in the UK to be codified. The website has been designed and is ready to go live following the competition and signoff of final formatting/visuals.

Engagement and dissemination activities

2.26 Following the Mid-Term Evaluation, the operation has continued to deliver a broad range of engagement and dissemination activities with staff based at Swansea University and external stakeholders. Positively, as Covid-19 related restrictions eased, the operation was able to engage with/deliver in-person events and activities. Some recent examples include: • Attendance and/or contributing to events and conferences. For example, members of the LILW team delivered a webinar showcasing ‘The C ovid-19 Impact on Employment Law Forums’ project in collaboration with the LegalBeagles online advice forum; attended the Swansea Science Festival where they demonstrated a legal advice chatbot designed

3 LawTech UK is a government-backed initiative, supported by CodeBase and Legal Geek. It was established to support the transformation of the UK legal sector through technology, for the benefit of society and the economy.

Legal Innovation Lab Wales

by the Lab to help members of the public understand the recent Renting Homes Wales Act; and presented at a ‘Lunch’n’Learn’ event to raise awareness and engagement with the Lab amongst Swansea University academics. • Publication of research articles, including ‘A comparative analysis of Islamic State (IS) Channels on Telegram’ , in the International Journal of Security, Terrorism and Society; ‘M aking the most of cybercrime and fraud crime report data: a case study of UK Action Frau d’ in the International Journal of Population Data Science; a paper on ‘Taking stock o f available technologies for compliance checking on first- order knowledge’ was presented at the 37th Italian Conference on Computational Logic; and two conferences papers were submitted to the 14th International Conference On Global Security, Safety & Sustainability in London. • Contributions to reports, insights, and reviews. For example, a Senior Lecturer published a co- edited ‘Handbook of Legal AI’ , a CYTREC Lecturer had a chapter accepted for publication with the ‘ Crime Data Handbook ’, another CYTREC Senior Lecturer co - authored seven book chapters in 2022, whilst the Co-Principal Investigator and Director of CYTREC co-authored a report for the Global Network on Extremism and Technology (GNET). 2.27 In addition, the operation has created a substantial volume of online content, in both English and Welsh, to promote and educate the law sector about LegalTech, including ‘tech explainer’ videos and blogs (by both the research teams and Software Development Team). The LILW also delivered a series of knowledge exchange sessions over seven weeks in Summer 2022 targeted primarily at trainee solicitors. These aimed to provide insight into how different technologies could be implemented in the workplace. Around 50 individuals signed-up for the sessions, which were focused on providing practical examples and guidance on the use of and opportunities from LegalTech. The operation also hosted (with the Welsh Government) a Legal Hackathon Wales in July 2022. 2.28 Prototypes developed by the LILW are open source and available on the Developer page of the website. 4 By June 2023, this included details on 11 operation projects, including (where relevant) source code and links to prototypes/demonstrations of the applications and/or further details about the project.

4 https://developer.legaltech.wales/

Legal Innovation Lab Wales

Figure 2-5: Examples of project content on the LILW Developer Page

Source: Legal Innovation Lab Wales Developer Page (legaltech.wales)

2.29 Further, the ‘community outputs’ page 5 of the website provides information on activities related to the CCTs. These resources have been disseminated more widely via the LILW YouTube and Twitter (@LegalLabWales) accounts, where the latter has 474 followers (as at 02/06/23).

Project management and governance perspectives

2.30 Consultations for this Final Evaluation – consistent broadly with the findings at the Mid-Term Evaluation stage – suggest that project management and governance systems and processes of the operation have worked well overall, albeit governance and management structures changed over the course of the operation.

Project management …

2.31 Delivery of the operation has been led and managed by two Co-Principal Investigators (covering the LegalTech and Cyber Threats elements respectively) who were responsible for overall direction and oversight of the operation, and a Project Manager who is responsible for day-to-day management activities including internal and external liaison. Additional support has been provided through a wider project team which has included (although specific roles have changed over time) a Project Coordinator, LegalTech Engagement Officer, and Research Funding Officer. 2.32 Consultations suggest that strong project management and administration support underpinned delivery of the operation. The provision of a dedicated project team via the ERDF support for the operation was seen to be important, both from a monitoring and reporting

5 https://legaltech.wales/en/outputs

Legal Innovation Lab Wales

perspective, and to facilitate the delivery of a range of events, meetings, and wider initiatives which are important to the core purpose of the operation to help raise awareness and buy-in to LegalTech and enhancing the University’s existing strong position and profile in relation to Cyber Threats. 2.33 Frequent communication via weekly team meetings was valued by researchers and software developers. However, some feedback suggested a lack of interaction and communication between the researcher teams and the software developers in the day-to-day delivery of projects, resulting in less frequent engagement and communication between the constituent parts of the operation. This was likely influenced to some extent by the pandemic and changing working behaviours, and potentially exacerbated owing to changes in personnel throughout delivery (with changing contacts and the need to establish new relationships and understanding). 2.34 The original Project Manager left the University and was replaced in August 2022. This led to some short-term delays in the delivery of activity associated with the handover e.g. in relation to the PASF accreditation process and Virtual Platform (as noted above). However, overall it was reported that this transition was well-handled and did not adversely impact on overall progress; this is a positive finding, and was enabled in part through continuity in the academic leadership of the operation. 2.35 Monitoring and reporting processes have been implemented effectively overall. Feedback suggested that reporting to WEFO has been thorough and high quality, and the project team has effectively engaged with the University’s R esearch, Engagement and Innovation Services (REIS) department to support with the process. 2.36 However, there was scope for better collection of data on the wider benefits of the LILW outside of formal ERDF indicators. The operation sought to implement the use of a consistent and comprehensive process to gather data on key outputs (such as events hosted/attended, research publications, articles, or prototypes developed), involving the distribution of an online form to staff every six weeks. However, the quality and coverage of the material provided in response was very varied and not comprehensive. As such, the data does not provide an accurate reflection of the activity, outputs, and outcomes delivered by the operation. As a result, the evaluation has not drawn on this data source. This is an important learning lesson for any similar future interventions.

…and project governance

2.37 As set out in the Mid-Term Evaluation Report, the original governance structure of the operation – with a Project Board, a Project Advisory Board 6 and a single Principal Investigator

6 Called a Project Steering Group in the Business Plan

Legal Innovation Lab Wales

Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Page 7 Page 8-9 Page 10-11 Page 12 Page 13 Page 14 Page 15 Page 16 Page 17 Page 18 Page 19 Page 20 Page 21 Page 22 Page 23 Page 24 Page 25 Page 26 Page 27 Page 28 Page 29 Page 30 Page 31 Page 32 Page 33 Page 34 Page 35 Page 36 Page 37 Page 38 Page 39 Page 40 Page 41 Page 42 Page 43 Page 44 Page 45 Page 46 Page 47 Page 48 Page 49 Page 50-51 Page 52-53 Page 54 Page 55 Page 56 Page 57 Page 58-59 Page 60-61 Page 62-63 Page 64-65 Page 66-67 Page 68-69 Page 70

www.swansea.ac.uk

Made with FlippingBook HTML5