11-01-2021, 01:38 PM
(This post was last modified: 11-01-2021, 01:44 PM by chelllman. Edited 2 times in total.)
The topic of logging staff/donor command usage is something that @RussEfarmer and I have both been exploring, but from different angles. In particular, he has been exploring real-time command logging while I have been exploring extracting command usage from recorded gameplay. I will try to have my code publicly available in the next few days, but I’ve been swamped with work. I wanted to make this post weeks ago.
I record 100% of my time on Dink’s prop hunt both because I can reference footage to supplement bans, but also for my own accountability. However, when you have 100s of hours of footage, simply making a couple of notes on every map of PH isn’t enough. Example: I warn and gag someone for racism. Later, another staff member gags the same person for racism. In order for us to make a ban, we would need to show footage of the actions by both staff members. In order to get this person banned, I would need to potentially watch several hours of footage.
High Level Overview
To solve this issue, I’ve written a Python script that automatically extracts my command usage from recorded gameplay footage. At a high level, the program indexes through frames of a video looking for text. Once all text on a frame has been found, it then parses through the text looking for statements that conform to command usage. This is relatively simple since each command has a pattern associated with it. For example, a kick command follows the following format: !kick [NAME] (REASON) where the [NAME] part is required and the (REASON) part is optional. Additionally, following command usage, each command will print out on your screen stating that “you [COMMAND]ed [NAME]...”. Here’s an example where the program found instances of me gagging the former president of the US for micspam:
The output of the program is a .csv file containing a table of commands used by staff/donors, the name of the command user, and finally the target of the command. After a video has been analyzed, a secondary Python script can be used to quickly search through the output .csv files to search for particular command usage, particular users, etc.
More Details
Tesseract is an excellent optical character recognition engine currently sponsored by Google, and tesseract has support for a wide variety of programming languages including Python. We will use tesseract as the means to detect text on video frame images.
The process for command text extraction is relatively simple. We first use Python CV to determine the fps of the video because if your video was recorded at 60 fps and text from the chat stays on your screen for 5ish seconds, then you have at most 300 frames where the text will appear. Many back-to-back frames will contain the same text from the chat, so it is not necessary to extract the text from all frames. In fact, this will make our process extremely slow. Therefore, we have the option to simply skip ahead X number of frames to avoid repeatedly searching nearly identical images for text. In my case, I record at 30 fps, so I query text every 10 frames.
We can make additional optimizations as well. For example, the text chat always appears on the lower left quadrant of my screen (you can set this to appear somewhere else btw). We can limit our search area to just 1/4th the size of the original frame to save time as well. With these optimizations, I can get through a 1 hour video in about 10ish minutes (which is relatively fast considering we are extracting text from an image!).
Once all the text has been found, I perform pattern analysis using a few regular expressions to identify command usage. This is better left to be explained via comments in my code (that I will release soon).
Right now this is built for prop hunt, but the architecture could quickly be adapted for the other game modes. I’m also adding in the functionality to search for the tables showing which players are online, scores, etc. This is more for if you need to narrow your search for a particular interaction with a player where a command was not used.
Other Projects
I created a simple Python program for managing youtube video uploads. The program is done, I just need to make a video tutorial because you need to go through Google authentication in order to link with your youtube channel. I put the videos I want to upload in a folder, run the script, and they are automatically uploaded to my youtube channel as unlisted videos. Simple, effective, and youtube serves as cloud storage for videos. You can also make your videos public with such an obscure name that no one will ever find them through the search bar. I’ll put this on my github soon.
EDIT: I don't know why the text was so small, fixed it so you don't need a microscope to read this post.
I record 100% of my time on Dink’s prop hunt both because I can reference footage to supplement bans, but also for my own accountability. However, when you have 100s of hours of footage, simply making a couple of notes on every map of PH isn’t enough. Example: I warn and gag someone for racism. Later, another staff member gags the same person for racism. In order for us to make a ban, we would need to show footage of the actions by both staff members. In order to get this person banned, I would need to potentially watch several hours of footage.
High Level Overview
To solve this issue, I’ve written a Python script that automatically extracts my command usage from recorded gameplay footage. At a high level, the program indexes through frames of a video looking for text. Once all text on a frame has been found, it then parses through the text looking for statements that conform to command usage. This is relatively simple since each command has a pattern associated with it. For example, a kick command follows the following format: !kick [NAME] (REASON) where the [NAME] part is required and the (REASON) part is optional. Additionally, following command usage, each command will print out on your screen stating that “you [COMMAND]ed [NAME]...”. Here’s an example where the program found instances of me gagging the former president of the US for micspam:
The output of the program is a .csv file containing a table of commands used by staff/donors, the name of the command user, and finally the target of the command. After a video has been analyzed, a secondary Python script can be used to quickly search through the output .csv files to search for particular command usage, particular users, etc.
More Details
Tesseract is an excellent optical character recognition engine currently sponsored by Google, and tesseract has support for a wide variety of programming languages including Python. We will use tesseract as the means to detect text on video frame images.
The process for command text extraction is relatively simple. We first use Python CV to determine the fps of the video because if your video was recorded at 60 fps and text from the chat stays on your screen for 5ish seconds, then you have at most 300 frames where the text will appear. Many back-to-back frames will contain the same text from the chat, so it is not necessary to extract the text from all frames. In fact, this will make our process extremely slow. Therefore, we have the option to simply skip ahead X number of frames to avoid repeatedly searching nearly identical images for text. In my case, I record at 30 fps, so I query text every 10 frames.
We can make additional optimizations as well. For example, the text chat always appears on the lower left quadrant of my screen (you can set this to appear somewhere else btw). We can limit our search area to just 1/4th the size of the original frame to save time as well. With these optimizations, I can get through a 1 hour video in about 10ish minutes (which is relatively fast considering we are extracting text from an image!).
Once all the text has been found, I perform pattern analysis using a few regular expressions to identify command usage. This is better left to be explained via comments in my code (that I will release soon).
Right now this is built for prop hunt, but the architecture could quickly be adapted for the other game modes. I’m also adding in the functionality to search for the tables showing which players are online, scores, etc. This is more for if you need to narrow your search for a particular interaction with a player where a command was not used.
Other Projects
I created a simple Python program for managing youtube video uploads. The program is done, I just need to make a video tutorial because you need to go through Google authentication in order to link with your youtube channel. I put the videos I want to upload in a folder, run the script, and they are automatically uploaded to my youtube channel as unlisted videos. Simple, effective, and youtube serves as cloud storage for videos. You can also make your videos public with such an obscure name that no one will ever find them through the search bar. I’ll put this on my github soon.
EDIT: I don't know why the text was so small, fixed it so you don't need a microscope to read this post.
