Enhancing AI Literacy in Higher Education: A Mixed-Methods Study on Student Experiences with AI Tools

Artificial intelligence (AI) has become increasingly prevalent in education, transforming how students learn and interact with information. As AI technologies like generative AI (GenAI) continue to evolve, there is a growing need for AI literacy among students and educators. This study contributes to the emerging literature on AI literacy in higher education by exploring how graduate students' exposure to AI review mechanisms and AI image generation tools influences their perceived AI literacy development. AI literacy encompasses the ability to critically understand, evaluate, and apply AI technologies in various contexts. In higher education, developing AI literacy is crucial for preparing students to navigate an AI-driven world and leverage AI tools effectively in their academic and professional lives. This research investigates students' perspectives on effective ways to enhance their AI literacy through hands-on experiences with AI tools in postgraduate education courses.

This study employed a convergent, mixed-methods case study approach, combining quantitative survey data with qualitative insights from student reflections. The research was conducted across three 8-week online courses within the College of Education at a midwestern university in the United States. Out of 61 enrolled students, 37 volunteered to participate in the study. Participants were primarily white females between 25 and 45 years old, pursuing graduate degrees in education and humanities. Their prior exposure to AI technologies varied, with 22% reporting high familiarity and 22% having no familiarity with AI concepts. The study focused on students' experiences with two key AI applications: a specialized AI review tool for assessing complex essays and AI-based image generation tools for reflecting on learning experiences.

The study utilized a holistic, cyber-social approach to explore students' perceived AI literacy development. This approach involved: 1. A social learning platform's GenAI review tool, designed and developed by the research team 2. GenAI image generation tools for student reflections 3. Critical exposure to AI-related topics through course resources and discussions The AI review tool interfaced with OpenAI's GPT to provide automated feedback on student projects, complementing peer and instructor feedback. The tool was enhanced through prompt engineering, precision fine-tuning, transparency, human moderation, and integration of disciplinary ontologies. Students worked on multimodal projects examining technology, educational theory, and practice throughout the semester. They received both AI and peer feedback at different stages of the development process, allowing them to compare and reflect on the two types of reviews.

Data collection involved pre- and post-course surveys and student reflections on their AI literacy progress. The surveys probed participants' familiarity with AI concepts, confidence in using AI tools, and experience with AI image generation. Quantitative data from the surveys were analyzed using descriptive and inferential statistics, including paired samples t-tests to determine significant changes in participants' reported AI literacy. Qualitative data from open-ended survey responses and student reflections were subjected to thematic analysis to identify common themes and experiences related to AI literacy development.

The study's findings revealed several key insights into students' perceived AI literacy development: 1. Increased familiarity with AI concepts: Students reported a significant increase in their understanding of AI and machine learning concepts after the course (mean score increased from 2.62 to 3.22 on a 5-point scale). 2. Enhanced confidence in using AI tools: Participants' perceived confidence in utilizing AI tools for educational purposes improved substantially (mean score increased from 2.41 to 3.27). 3. Improved prompt creation skills: Students reported greater proficiency in crafting prompts for AI image generation (mean score increased from 2.16 to 3.35). 4. Recognition of AI utility: Post-course, 67% of participants found AI image generation tools at least moderately useful for their learning experience. 5. Critical assessment of AI feedback: Students developed the ability to identify advantages and disadvantages of AI feedback compared to human reviews, demonstrating enhanced critical thinking skills related to AI applications.

Thematic analysis of student reflections revealed several important aspects of their AI experiences: 1. Iterative learning process: Students described their interaction with AI image generation tools as an iterative process of trial and error, leading to improved prompt creation skills. 2. Creative expression: Participants used AI-generated images to express their experiences with peer and AI reviews metaphorically, demonstrating creative applications of AI tools. 3. Positive perceptions of AI: Students viewed AI as a powerful, intelligent, and collaborative tool that enhances productivity and supports cognitive development. 4. Increased interest in AI applications: Exposure to AI reviews sparked heightened interest in exploring potential AI applications in future academic and professional contexts. 5. Integration of AI in personal practices: Students who had more experience with the AI review tool reported incorporating AI tools like ChatGPT more extensively into their pedagogical practices and personal studies.

The study's findings have several implications for integrating AI into higher education: 1. Hands-on experience: Providing students with opportunities to interact directly with AI tools can significantly enhance their AI literacy and confidence in using these technologies. 2. Complementary feedback: Combining AI and peer reviews can offer students a more comprehensive understanding of their work and develop critical assessment skills. 3. Creative applications: Encouraging students to use AI tools for creative expression can foster innovative thinking and deeper engagement with AI technologies. 4. Customized strategies: Educators should consider developing customized educational strategies to maximize AI tool adoption and literacy development based on varied student needs and levels of AI literacy. 5. Ongoing skill development: As AI technologies continue to evolve, higher education institutions should prioritize ongoing AI literacy development for both students and educators.

While this study provides valuable insights into AI literacy development in higher education, it has some limitations: 1. Self-reported data: The study relied on students' self-reported perceptions of AI literacy, which may not fully reflect actual skill development. 2. Limited sample size: The study involved a relatively small sample of 37 participants from a single university, limiting generalizability. 3. Specific educational context: The research focused on graduate education courses, and findings may not apply equally to other disciplines or educational levels. Future research directions could include: 1. Objective assessment: Developing and implementing objective measures of AI literacy to complement self-reported data. 2. Longitudinal studies: Investigating long-term impacts of AI integration on students' AI literacy and academic performance. 3. Cross-disciplinary research: Exploring AI literacy development across various academic disciplines and educational levels. 4. Ethical considerations: Examining the ethical implications of AI integration in higher education and developing frameworks for responsible AI use. 5. Pedagogical strategies: Investigating effective pedagogical approaches for enhancing AI literacy across diverse student populations.