Create an AI Assistant to Summarize YouTube Videos: Part 1 - Transcribing with OpenAI Whisper

The AI assistant for summarizing YouTube videos is designed with a clear, multi-stage architecture. The process begins with obtaining the video's transcript. If a transcript is readily available on YouTube, it is directly downloaded. Otherwise, a powerful open-source voice-to-text model, OpenAI's Whisper, is employed for transcription. Following transcription, the text is processed using Langchain and a large language model fine-tuned for instruction following, specifically Falcon-7b-instruct, to generate a concise summary. Finally, a user-friendly interface is created using Gradio, allowing users to interact with the AI, generate summaries, and view them alongside the original video. Throughout the development process, various architectural alternatives were explored, including on-premise versus cloud-based inference and hosting options.

For YouTube videos that already have a transcript, the process is straightforward. Libraries like `youtube_transcript_api` and `pytube` can be used to extract this information. The `pytube` library helps in extracting the video ID from the URL, which is then passed to `YouTubeTranscriptApi.get_transcript()`. This function returns a list of dictionaries, each containing the transcript text, start time, and duration. By iterating through this list and concatenating the 'text' field from each dictionary, the full transcript can be obtained as a single string.

For users who prefer not to run computationally intensive models locally or lack sufficient hardware, Hugging Face offers cloud-based inference through its Hub APIs. An `InferenceClient` can be initialized with the Whisper model and a Hugging Face API token. However, a common issue encountered with the inference API is result truncation for longer audio inputs. To overcome this, the audio file needs to be split into smaller segments before being sent to the API. Libraries like `librosa` can load the audio, and `soundfile` can save the split chunks. Each chunk is then processed individually by the `client.automatic_speech_recognition()` method, and the results are concatenated. It's important to note that free-tier API usage has rate limits, which might affect the transcription of very long videos.

To create a versatile transcription function, it's beneficial to combine the different approaches. A function, such as `transcribe_youtube_video(url, force_transcribe=False, use_api=False)`, can be implemented. This function first attempts to retrieve the transcript directly from YouTube. If this fails or if `force_transcribe` is set to `True`, it proceeds to transcription. The `use_api` parameter then determines whether to use the local Whisper model (`transcribe_yt_vid`) or the Hugging Face Hub API (`transcribe_yt_vid_api`). This approach provides flexibility, allowing users to choose the most suitable transcription method based on their resources and needs.