Maizey: University of Michigan AI Chatbot
Goal
A University of Michigan professor I know through my church approached me about a project she was working on. Her vision was an AI chatbot tool that would help her 250+ students in a way that she and her limited TA staff couldn't. The goal was to create an AI chatbot to both answer student questions, and help with test prep by generating and grading practice problems. The problem she asked me to help solve was the process of transforming the unstructured course materials—33 hours of video lectures and a 1,200-page textbook—into a clean, structured knowledge base for the LLM. Maizey, U of M's in house AI tool, is a weaker model than the current generation, and must use text only data sources for its training, meaning we had to find a way to turn the semester's content into text only representations. Initially, I was only asked to work with the direct transcripts of the lectures, but I convinced the professor of the value and potential of being able to include the slide content, and she happily agreed.
My Contributions
- Scripted an automatic textbook scraper, spending a few days to create a script that processed the entire 1,200-page book in about an hour. This task previously took an entire summer of manual work for a student worker under the professor.
- Engineered a sophisticated video processing script using SIFT feature matching and OCR to match timestamped portions of the video to their corresponding presentation slides. This then made it possible to synchronize the slide content with the lecture transcript, creating a rich knowledge base for the LLM.
- Utilized the Gemini API for data transformation, primarily for merging timestamped slide content with its respective speaker transcript, as well as for synthesizing the final lecture transcript into a cohesive, textbook style chapter.
- Implemented a manual verification stage to ensure 100% accuracy in the final data. This demonstrates a commitment to quality when using AI generated output, which will inevitably have imperfections.
Challenge
The primary technical challenge was accurately synchronizing the fast paced spoken lecture with the correct on-screen slide. The video feed was often imperfect, and couldn't reliably always have a clear picture of the slides. This required a two stage verification process (feature matching + OCR) to ensure reliability. A secondary challenge was parsing the textbook's complex layout, particularly in deciding how to handle nonlinear elements like asides and diagrams.
Solution
My solution was a multi stage pipeline. For lectures, the script first aligns the transcript with slide changes, then uses the Gemini API to group content by topic, and synthesize the whole lecture into a polished chapter. For the textbook, the script extracts text while identifying images and tables, using Gemini to generate descriptive captions. It then integrates these figures into the text content of the chapter, looking to where it is referenced and using the generated description to stitch together a cohesive text-only rendition of the chapter content. Both processes include a manual verification step for asides and matches, ensuring the final output is of the highest quality.
Outcome & Impact
The completed pipelines were effectively used to process the semester's lectures and textbook, creating not just a foundational knowledge base for this class' specific chatbot, but a set of tools that any other professor or class could use. My automation work saved months of manual labor and raised the quality of the chatbot's data sources through the inclusion of lecture slides to accompany the transcript. This tool is now being used by the professor at the University of Michigan in her Microbiology course for students to ask questions of and generate practice problems. We are now aiming to bring this kind of tool to other professors and their courses, and work currently to understand how best to teach others how to use and adapt the pipeline for their own courses. This project also was an introduction in responsible use of generative AI for me, and I learned a signifcant amount on how to best leverage it as a tool while understanding its limitations.