DigiBot: Personalized AI-Driven Digital Skills Assessment

Can AI personalize assessment tests?
The DigiBot project explores this question. The chatbot uses ChatGPT to adapt standardized assessment questions based on individual user profiles, making tests feel more relevant and engaging. The project tested how personalized questions compare to traditional ones—revealing surprising insights about AI’s role in education.

The Challenge: Making Tests More Relevant

Standardized tests often feel disconnected from real-life experiences, making them less effective. This project tackled the challenge by leveraging the EU’s DigComp framework, which assesses digital skills like information literacy and cybersecurity. The goal was to personalize these assessments while maintaining consistency and reliability.

The Innovative Approach: AI-Powered Personalization

DigiBot personalizes assessments in two key steps:

1. User Profiling: DigiBot first engages users in a brief conversation, asking about their job role, daily tasks, and self-assessed digital skills. This creates a detailed profile to tailor questions effectively.

2. Question Personalization: Using ChatGPT, DigiBot transforms standardized questions to match the user’s context. For instance, a marketer might receive a question about identifying fake customer inquiries, while an IT admin might focus on email security.

Users were then asked to choose between original and AI-personalized questions, providing feedback on relevance and clarity.

Key Results

• Clarity vs. Relevance: Original questions were easier to understand, while AI-generated ones felt more detailed and context-specific.
• User Preferences: Beginners preferred simpler, original questions. Advanced users appreciated the deeper, profession-specific AI-generated versions.
• Personalization Impact: Tailored questions were valued when they accurately reflected the user’s background.

Future Steps

To enhance DigiBot’s performance, the next developments include:

• Short-Term Improvements:

  • Enhance user profiling with smarter follow-up questions.
  • Simplify and clarify AI-generated questions using readability metrics.
  • Refine how user preferences are collected and analyzed.

• Long-Term Developments:

  • Fine-tune AI models for better personalization.
  • Implement user-driven feedback loops for continuous improvement.
  • Optimize system efficiency by caching frequent questions and using cost-effective models.

DigiBot shows that AI can make assessments more engaging by personalizing them to individual experiences. The next phase will focus on refining this balance between relevance and simplicity, paving the way for smarter, user-centered testing solutions.