Chat about a video clip using the powerful OpenAI GPT-4 Vision or GPT-4o.

chat-about-video is an open-source NPM package designed to accelerate the development of conversation applications about video content. Harnessing the capabilities of OpenAI GPT-4 Vision or GPT-4o services from Microsoft Azure or OpenAI, this package opens up a range of usage scenarios with minimal effort.

Key features:

• ChatGPT models hosted in both Azure and OpenAI are supported.
• Frame images are extracted from the input video, and uploaded for ChatGPT to consume.
• It can automatically retry on receiving throttling (HTTP status code 429) responses from the API.
• Options supported by the underlying API are exposed for customisation.

There are two approaches for feeding video content into GPT-4 Vision. chat-about-video supports both of them.

Frame image extraction:

• Integrate GPT-4 Vision or GPT-4o from Microsoft Azure or OpenAI effortlessly.
• Utilize ffmpeg integration provided by this package for frame image extraction or opt for a DIY approach.
• Store frame images with ease, supporting Azure Blob Storage and AWS S3.
• GPT-4 Vision hosted in Azure allows analysis of up to 10 frame images.
• GPT-4 Vision or GPT-4o hosted in OpenAI allows analysis of more than 10 frame images.

Video indexing with Microsoft Azure:

• Exclusively supported by GPT-4 Vision from Microsoft Azure.
• Ingest videos seamlessly into Microsoft Azure's Video Retrieval Index.
• Automatic extraction of up to 20 frame images using Video Retrieval Indexer.
• Default integration of speech transcription for enhanced comprehension.
• Flexible storage options with support for Azure Blob Storage and AWS S3.

Add chat-about-video as a dependency to your Node.js application using the following command:

npm i chat-about-video

If you intend to utilize ffmpeg for extracting video frame images, ensure it is installed on your system. You can install it using either a system package manager or a helper NPM package:

sudo apt install ffmpeg
# or
npm i @ffmpeg-installer/ffmpeg

If you plan to use Azure Blob Storage, include the following dependency:

npm i @azure/storage-blob

For using AWS S3, install the following dependencies:

npm i @handy-common-utils/aws-utils @aws-sdk/s3-request-presigner @aws-sdk/client-s3

To integrate chat-about-video into your Node.js application, follow these simple steps:

1. Instantiate the ChatAboutVideo class by creating an instance. The constructor allows you to pass in configuration options.

• Most configuration options come with sensible default values, but you can specify your own for further customization.
• The second constructor argument is a logger. If not specified, a default logging will be created for logging to the console. If logging is not needed, you can pass in undefined.

2. Use the startConversation(videoFilePath) function to initiate a conversation about a video clip. This function returns a Conversation object. The video file or its frame images are sent to Azure Blob Storage or AWS S3 during this step.
3. Interact with GPT by using the say(question, { maxTokens: 2000 }) function within the conversation. You can pass in a question, and will receive an answer.

• Message history is automatically kept during the conversation, providing context for a more coherent dialogue.
• The second parameter of the say(...) function allows you to specify your own for further customization.

4. Wrap up the conversation using the end() function. This ensures proper clean-up and resource management.

Below is an example chat application, which

• uses GPT deployment (in this example, it is named 'gpt4vision') hosted in Microsoft Azure;
• uses ffmpeg to extract video frame images;
• stores video frame images in Azure Blob Storage;
  • container name: 'vision-experiment-input'
  • object path prefix: 'video-frames/'
• reads credentials from environment variables
• reads input video file path from environment variable 'DEMO_VIDEO'

import readline from 'node:readline';
import { ChatAboutVideo } from 'chat-about-video';

const rl = readline.createInterface({ input: process.stdin, output: process.stdout });
const prompt = (question: string) => new Promise<string>((resolve) => rl.question(question, resolve));

async function demo() {
  const chat = new ChatAboutVideo({
    openAiEndpoint: process.env.AZURE_OPENAI_API_ENDPOINT!, // This line is not needed if you are using GTP provided by OpenAI rather than by Microsoft Azure.
    openAiApiKey: process.env.OPENAI_API_KEY!, // This is the API key.
    azureStorageConnectionString: process.env.AZURE_STORAGE_CONNECTION_STRING!, // This line is not needed if you'd like to use AWS S3.
    openAiDeploymentName: 'gpt4vision', // For GPT provided by OpenAI, this is the model name. For GPT provided by Microsoft Azure, this is the deployment name.
    storageContainerName: 'vision-experiment-input', // Blob container name in Azure or S3 bucket name in AWS
    storagePathPrefix: 'video-frames/',
  });

  const conversation = await chat.startConversation(process.env.DEMO_VIDEO!);
  
  while(true) {
    const question = await prompt('\nUser: ');
    if (!question) {
      continue;
    }
    if (['exit', 'quit'].includes(question.toLowerCase().trim())) {
      break;
    }
    const answer = await conversation.say(question, { maxTokens: 2000 });
    console.log('\nAI:' + answer);
  }
}

demo().catch((error) => console.error(error));

Below is an example showing how to create an instance of ChatAboutVideo that

• uses GPT provided by OpenAI;
• uses ffmpeg to extract video frame images;
• stores video frame images in AWS S3;
  • bucket name: 'my-s3-bucket'
  • object path prefix: 'video-frames/'
• reads API key from environment variable 'OPENAI_API_KEY'

  const chat = new ChatAboutVideo({
    openAiApiKey: process.env.OPENAI_API_KEY!,
    openAiDeploymentName: 'gpt-4-vision-preview', // or 'gpt-4o'
    storageContainerName: 'my-s3-bucket',
    storagePathPrefix: 'video-frames/',
    extractVideoFrames: {
      limit: 30,    // override default value 10
      interval: 2,  // override default value 5
    },
  });

Below is an example showing how to create an instance of ChatAboutVideo that

• uses GPT deployment (in this example, it is named 'gpt4vision') hosted in Microsoft Azure;
• uses Microsoft Video Retrieval Index to extract frames and analyse the video
  • A randomly named index is created automatically.
  • The index is also deleted automatically when the conversation ends.
• stores video file in Azure Blob Storage;
  • container name: 'vision-experiment-input'
  • object path prefix: 'videos/'
• reads credentials from environment variables

  const chat = new ChatAboutVideo({
    openAiEndpoint: process.env.AZURE_OPENAI_API_ENDPOINT!,
    openAiApiKey: process.env.AZURE_OPENAI_API_KEY!,
    azureStorageConnectionString: process.env.AZURE_STORAGE_CONNECTION_STRING!,
    openAiDeploymentName: 'gpt4vision',
    storageContainerName: 'vision-experiment-input',
    storagePathPrefix: 'videos/',
    videoRetrievalIndex: {
      endpoint: process.env.AZURE_CV_API_ENDPOINT!,
      apiKey: process.env.AZURE_CV_API_KEY!,
      createIndexIfNotExists: true,
      deleteIndexWhenConversationEnds: true,
    },
  });

• aws
• azure
• azure/video-retrieval-api-client
• chat
• client-hack
• index
• storage
• storage/types
• video
• video/ffmpeg
• video/types

azure/video-retrieval-api-client.VideoRetrievalApiClient

• new VideoRetrievalApiClient(endpointBaseUrl, apiKey, apiVersion?)

▸ createIndex(indexName, indexOptions?): Promise<void>

Promise<void>

▸ createIndexIfNotExist(indexName, indexOptions?): Promise<void>

Promise<void>

▸ createIngestion(indexName, ingestionName, ingestion): Promise<void>

Promise<void>

▸ deleteDocument(indexName, documentUrl): Promise<void>

Promise<void>

▸ deleteIndex(indexName): Promise<void>

Promise<void>

▸ getIndex(indexName): Promise<undefined | IndexSummary>

Promise<undefined | IndexSummary>

▸ getIngestion(indexName, ingestionName): Promise<IngestionSummary>

Promise<IngestionSummary>

▸ ingest(indexName, ingestionName, ingestion, backoff?): Promise<void>

Promise<void>

▸ listDocuments(indexName): Promise<DocumentSummary[]>

Promise<DocumentSummary[]>

▸ listIndexes(): Promise<IndexSummary[]>

Promise<IndexSummary[]>

chat.ChatAboutVideo

• new ChatAboutVideo(options, log?)

▸ Protected prepareVideoFrames(conversationId, videoFile, extractVideoFramesOptions?): Promise<PreparationResult>

Promise<PreparationResult>

▸ Protected prepareVideoRetrievalIndex(conversationId, videoFile, videoRetrievalIndexOptions?): Promise<PreparationResult>

Promise<PreparationResult>

▸ startConversation(videoFile, options?): Promise<Conversation>

Start a conversation about a video.

Promise<Conversation>

The conversation.

chat.Conversation

• new Conversation(client, deploymentName, conversationId, messages, options?, cleanup?, log?)

▸ end(): Promise<void>

Promise<void>

▸ say(message, options?): Promise<undefined | string>

Say something in the conversation, and get the response from AI

Promise<undefined | string>

The response/completion

azure/video-retrieval-api-client.CreateIndexOptions

azure/video-retrieval-api-client.DocumentSummary

azure/video-retrieval-api-client.IndexFeature

azure/video-retrieval-api-client.IndexMetadataSchema

azure/video-retrieval-api-client.IndexMetadataSchemaField

azure/video-retrieval-api-client.IndexSummary

azure/video-retrieval-api-client.IngestionRequest

azure/video-retrieval-api-client.IngestionStatusDetail

azure/video-retrieval-api-client.IngestionSummary

azure/video-retrieval-api-client.VideoIngestion

chat.ChatAboutVideoOptions

Option settings for ChatAboutVideo

▸ createAwsS3FileBatchUploader(s3Client, expirationSeconds, parallelism?): FileBatchUploader

FileBatchUploader

Re-exports CreateIndexOptions

Re-exports DocumentSummary

Re-exports IndexFeature

Re-exports IndexMetadataSchema

Re-exports IndexMetadataSchemaField

Re-exports IndexSummary

Re-exports IngestionRequest

Re-exports IngestionStatusDetail

Re-exports IngestionSummary

Re-exports PaginatedWithNextLink

Re-exports VideoIngestion

Re-exports VideoRetrievalApiClient

▸ createAzureBlobStorageFileBatchUploader(blobServiceClient, expirationSeconds, parallelism?): FileBatchUploader

FileBatchUploader

• VideoRetrievalApiClient

• CreateIndexOptions
• DocumentSummary
• IndexFeature
• IndexMetadataSchema
• IndexMetadataSchemaField
• IndexSummary
• IngestionRequest
• IngestionStatusDetail
• IngestionSummary
• VideoIngestion

Ƭ PaginatedWithNextLink<T>: Object

• ChatAboutVideo
• Conversation

• ChatAboutVideoOptions

Ƭ ChatAboutVideoConstructorOptions: Partial<Omit<ChatAboutVideoOptions, "videoRetrievalIndex" | "extractVideoFrames">> & Required<Pick<ChatAboutVideoOptions, "openAiDeploymentName" | "storageContainerName">> & { extractVideoFrames?: Partial<Exclude<ChatAboutVideoOptions["extractVideoFrames"], undefined>> ; videoRetrievalIndex?: Partial<ChatAboutVideoOptions["videoRetrievalIndex"]> & Pick<Exclude<ChatAboutVideoOptions["videoRetrievalIndex"], undefined>, "endpoint" | "apiKey"> } & { azureStorageConnectionString?: string ; downloadUrlExpirationSeconds?: number ; openAiApiKey: string ; openAiEndpoint?: string }

Ƭ ChatOptions: GetChatCompletionsOptions & { throttleBackoff?: number[] }

Ƭ ExtractVideoFramesOptions: Exclude<ChatAboutVideoOptions["extractVideoFrames"], undefined>

Ƭ VideoRetrievalIndexOptions: Exclude<ChatAboutVideoOptions["videoRetrievalIndex"], undefined>

▸ fixClient(openAIClient): void

void

Re-exports ChatAboutVideo

Re-exports ChatAboutVideoConstructorOptions

Re-exports ChatAboutVideoOptions

Re-exports ChatOptions

Re-exports Conversation

Re-exports ExtractVideoFramesOptions

Re-exports FileBatchUploader

Re-exports VideoFramesExtractor

Re-exports VideoRetrievalIndexOptions

Re-exports extractVideoFramesWithFfmpeg

Re-exports lazyCreatedFileBatchUploader

Re-exports lazyCreatedVideoFramesExtractor

Re-exports FileBatchUploader

▸ lazyCreatedFileBatchUploader(creator): FileBatchUploader

FileBatchUploader

Ƭ FileBatchUploader: (dir: string, fileNames: string[], containerName: string, blobPathPrefix: string) => Promise<string[]>

▸ (dir, fileNames, containerName, blobPathPrefix): Promise<string[]>

####### Parameters

####### Returns

Promise<string[]>

Re-exports VideoFramesExtractor

Re-exports extractVideoFramesWithFfmpeg

▸ lazyCreatedVideoFramesExtractor(creator): VideoFramesExtractor

VideoFramesExtractor

▸ extractVideoFramesWithFfmpeg(inputFile, outputDir, intervalSec, format?, width?, height?, startSec?, endSec?): Promise<string[]>

Promise<string[]>

Ƭ VideoFramesExtractor: (inputFile: string, outputDir: string, intervalSec: number, format?: string, width?: number, height?: number, startSec?: number, endSec?: number) => Promise<string[]>

▸ (inputFile, outputDir, intervalSec, format?, width?, height?, startSec?, endSec?): Promise<string[]>

####### Parameters

####### Returns

Promise<string[]>