NewtBot
Concept to Implementation to Data Collection & Analysis
I was accepted to participate in an NSF-funded research program in collaboration with UNH and LMU Munich in which I worked within the Media Informatics division of LMU. Our project, conducted in collaboration with the Didactic Physics department at LMU, involved investigating how different methods of priming an LLM would impact its usability and function in an educational context, specifically within physics education. I worked with a team of collaborators to design and implement a chatbot built on the ChatGPT API with three different methods of priming the LL, and additionally had the opportunity to gather data on education effectiveness by testing the application with student participants. We are currently working on preparing our findings for reporting and publication.
Conceptual Design
One of the first things I worked on was defining what features we wanted our application to contain, and how the data tables should be set up. I sketched out what our various interactions would look like and what they would encompass, as well as the types of information we would gather and how we would retrieve and display responses from the ChatGPT API. We varied the system message for each version of our chatbot to create different effects and compare their effectiveness in education, so I also drafted different messages based on a literature review in this stage.
Implementation and Testing
A sample interaction with one version of the chatbot. We integrated specific tasks for students to solve based on our collaboration with the Didactic Physics department in order to ensure our student participants would have a smooth and meaningful experience. Each version of the chatbot had a different internal system message, one of which was left untouched so as to return the ChatGPT unaltered response as a control. We were able to observe around 100 students using our application and provide on-site assistance to troubleshoot any technical issues.
Each of the above shows one of the three versions of our chatbot. We stored anonymous participant interactions with the bot in our database, and additionally gathered data by integrating a Qualtrics survey into our application as the enter and exit screens. We gathered participant data on their experience using the application, the extent to which they felt it assisted them with the provided tasks, and other general questions about their comfort with using AI assistance for educational tasks. We are currently in the process of formalizing our analysis of this data and their conversational for publication.