In light of the digital transformation taking place in international youth work, a rising number of digital projects have been implemented in recent years. To ensure that these, too, meet the quality criteria applicable in this field, it appears expedient to observe and analyse this transformation systematically, for instance by incorporating evaluation processes in digital formats. The research project “IJA.digital“ developed and tested such a set of questions.
Research project “IJA.digital” Development and testing of a set of questions
In light of the digital transformation taking place in international youth work, a rising number of digital projects have been implemented in recent years. To ensure that these, too, meet the quality criteria ap- plicable in this field, it appears expedient to observe and analyse this transformation systematically, for in- stance by incorporating evaluation processes in digi- tal formats. In the case of analogue (offline) formats, there are already platforms that help team leaders to do so. One example is www.i-eval.eu, which enables participants and project staff to self-evaluate inter- national youth exchanges using questionnaires. The anonymised data from the questionnaires is also fed into a panel study on youth exchanges and camps. However, at the present time this platform does not contain any questions that are designed specifically to evaluate digital projects in international youth work. The research project “IJA.digital“ developed and tested such a set of questions, bearing in mind the following considerations: • What should a quantitative set of questions for evaluating digital international youth work activi- ties ideally look like in terms of length and integ- ration in the activity? • What questions should be asked? To obtain key contextual information on the use of such a set of questions, team leaders working in inter- national youth work were invited to attend structured one-on-one interviews online and asked to share their thoughts on the above aspects. The interviewees had previously run digital projects that were observed during the IJA.digital project. According to them, the following aspects should be considered when deve- loping a set of questions for evaluation purposes: • In terms of length, the set of questions should be as compact as possible. • The questions should focus above all on the fol- lowing themes: • Suitability of the technical equipment and online tools used
Maximising the level of participation among young people in online settings
•
• Atmosphere in the group. • Generally speaking, the interviewees felt there was no need to make a major difference between questions for digital projects and questions for in-person projects. When developing the set of questions, besides con- sidering the feedback from the team leaders the re- searchers also built on the outcomes of a previous project. For instance, they considered an existing tool for evaluating digital activities in the cultural educa- tion field. The tool gathers feedback from each parti - cipant in an online activity individually, but also provi- des that feedback online to all members of the group via the tool itself. This approach allows for the data to be recorded but also integrates the tool in the feed- back phase towards the end of an activity. It is availa- ble at www.kubi-digital.jff.de. The IJA.digital research team used this as a basis to develop the questions for use in the international youth work context. The resulting set of questions was turned into an on- line questionnaire using the online survey applica- tion LimeSurvey; this in turn was tested in three on- line sessions by young people who had taken part in three separate digital projects that were observed by the IJA.digital research team. A written record was produced of the outcomes. The links below lead to the online test questionnaire and to the responses as exported by the tool. The test questionnaire and the responses are older versions that were used to test the new questions. The final version is reproduced on page 4 and 5 of this do- cument. During the test, the team briefly interrupted the process twice in order to discuss the responses so far and to make a qualitative assessment of the participants’ quantitative answers.
1
The first link leads to the first part of the set of questi - ons, the second to the second. • Part 1: https://befragung.jff.de/index.php/726815 • Part 2: https://befragung.jff.de/index.php/195383 • Responses: https://feedback.jff.de/
Fig. 1: Screenshot of the LimeSurvey test tool
2
Set of questions Application This section outlines the developed set of questions and explains how it can be used. • The set consists of twelve items and can be used for both purely online projects as well as for hyb- rid and blended activities. • The questions centre around themes that are par- ticularly relevant to digital projects. Questions on aspects that are equally relevant to physical/in- person settings (improving one’s language skills, getting to know the partner country, developing social skills, etc.) are deliberately not included. For this, reference is made to the above-mentio- ned i-EVAL platform, which contains questions on these topics. If the aim is to evaluate a digital project, the i-EVAL questions could be added to the questions from the set described here .
• The questions can also be used as an ice-brea- ker to start a discussion with the participants about the results of the survey. In this case the results should be made available directly after- wards to all team leaders and participants. The results should be clearly identifiable as belon - ging to a specific project. This can be done by using a dedicated project ID, for instance. • Naturally, any discussion with the participants should be voluntary; also, participants should be able to opt to remain anonymous.
3
The questions The following table contains the questions in the set and lists the aspects they focus on.
Aspect
Questions
Online sessions: General assessment
1. How much do you enjoy working with digital media (smartphone, tablet, etc.) in this project? • A lot • Somewhat • I have mixed feelings • Not a lot • I really don’t like it 2. Have the online sessions encouraged you to take part in
other international activities? • Yes, if they are only online • Yes, if they are offline/on-site • Yes, if there’s a mix of digital and offline • No
Number and duration of online sessions
3. How do you feel about the number of online sessions in this project? • Just right • Not enough • Too many 4. How do you feel about the duration of the online sessions in the project? • The sessions had just the right length
• The sessions were too long • The sessions were too short Group atmosphere during the online sessions 5. During the online sessions, there was a feeling of be- longing within the group of people I already knew from before.
• Completely agree • Somewhat agree • Neither agree nor disagree • Somewhat disagree • Totally disagree • I didn’t know anyone
6. During the online sessions, there was a feeling of be- longing within the group of people I did not know from before.
• Completely agree • Somewhat agree • Neither agree nor disagree • Somewhat disagree • Totally disagree • I already knew everyone
4
Informal interaction during the online sessions 7. During the online sessions I could interact with the other participants outside of the schedule and away from the
tasks we were given. • Completely agree • Somewhat agree • Neither agree nor disagree • Somewhat disagree • Totally disagree
Technical equipment and online tools
8. I had difficulties taking part in the online sessions becau - se the equipment (camera, microphone, etc.) didn’t work. • Completely agree • Somewhat agree • Neither agree nor disagree • Somewhat disagree • Totally disagree 9. I think the online tools (software, apps, etc.) were used appropriately.
• Completely agree • Somewhat agree • Neither agree nor disagree • Somewhat disagree • Totally disagree
Participants’ media skills and experience
10. I was able to suggest certain online tools (e.g., Discord, Instagram, TikTok) that were then used during the online sessions.
• Completely agree • Somewhat agree • Neither agree nor disagree • Somewhat disagree • Totally disagree
11. I was able to contribute my skills and experience with digital media (e.g., social media, hardware, software) during the online sessions.
• Completely agree • Somewhat agree • Neither agree nor disagree • Somewhat disagree • Totally disagree
Suggested improvements
12. I would like to suggest the following improvements to future online sessions: [open answer]
The set of questions was developed as part of the scientific monitoring of the project „IJA.digital - international youth work digital“. Further information and contact addresses can be found at: https://www.jff.de/kompetenzbereiche/projektdetail/internationale- jugendarbeit-digital/ 5
Page 1 Page 2 Page 3 Page 4 Page 5Made with FlippingBook - Online catalogs