Clever Geek Handbook

The power of multiple cores to tame the AV1 codec

image



Prologue



From time to time, I am interested in video codecs and how much more efficient they are compared to their predecessors. At one time, when HEVC came out after H264, I was very interested in touching it, but my hardware of that time left much to be desired.



Now the hardware has tightened up, but HEVC has long been outdated, it is eager to replace it with the open AV1, which promises us up to 50% savings compared to 1080p H264, but if the speed of high-quality encoding in HEVC seems slow (compared to H264), then AV1 is its ~ 0.2 fps demoralizes completely. When something is encoded so slowly, it means that even a simple 10 minute video will take about a day to be processed. Those. just to see if the encoding parameters are suitable or if you need to add a little bitrate, you will have to wait not just for hours, but for days ...



And so, once, admiring the beautiful sunset (H264 codec), I thought: "What if we put all the hardware that I have on AV1 at the same time?"



Idea



I tried to encode AV1 using tiles and multicore, but the performance gain seemed to me not so effective for each added processor core, giving about one and a half FPS at the fastest settings and 0.2 at the slowest, so a radically different idea came to my mind.



After looking at what we have for today on AV1, I made a list:



  • Ffmpeg's built-in libaom-av1 encoder
  • Rav1e project
  • SVT-AV1 project


From all of the above, I chose rav1e. It showed very good single-threaded performance and fit perfectly into the system I came up with:



  • The encoder will cut the original video into pieces for n seconds
  • Each of my computers will have a web server with a special script
  • We encode in one stream, which means that the server can simultaneously encode as many pieces as it has processor cores
  • The encoder will send the pieces to the servers, and download the encoded results back
  • When all the pieces are ready, the encoder will glue them into one and overlay the sound from the original file


Implementation



I must say right away that the implementation is made under Windows. In theory, nothing prevents me from doing the same thing for other OSs, but I did it for what I had.



So we need:



  • PHP web server
  • ffmpeg
  • rav1e


1. First, we need a Web server, I will not describe what and how I set up, for this there are a lot of instructions for every taste and color. I used Apache + PHP. It is important for PHP to make a setting that allows it to receive large files (by default in the settings 2MB and this is not enough, our pieces may be larger). Nothing special about plugins, CURL, JSON.



I will also mention security, which does not exist. Everything that I did - I did inside the local network, so no checks and authorizations were done, and there are plenty of opportunities for harm by intruders. Therefore, if this is to be tested in non-secured networks, security issues need to be taken care of yourself.



2. FFmpeg - I downloaded ready binaries from Zeranoe builds



3.rav1e - you can also download the binary from the rav1e project releases



PHP script for each computer that will participate
encoding.php, http: // HOST/remote/encoding.php

:



  1. ,
  2. CMD CMD
  3. CMD


:



  1. , CMD —
  2. , CMD —


, - , , , … , , .



, , . , , .



encoding.php: 



<?php

function getRoot()
{
	$root = $_SERVER['DOCUMENT_ROOT'];
	if (strlen($root) == 0)
	{
		$root = dirname(__FILE__)."\\..";
	}
	return $root;
}

function getStoragePath()
{
	return getRoot()."\\storage";
}


function get_total_cpu_cores()
{
	$coresFileName = getRoot()."\\cores.txt";
	if (file_exists($coresFileName))
	{
		return intval(file_get_contents($coresFileName));
	}
	return (int) ((PHP_OS_FAMILY == 'Windows')?(getenv("NUMBER_OF_PROCESSORS")+0):substr_count(file_get_contents("/proc/cpuinfo"),"processor"));
}

function antiHack($str)
{
	$strOld = "";
	while ($strOld != $str)
	{
		$strOld = $str;
  		$str = str_replace("\\", "", $str);
  		$str = str_replace("/", "",$str);
  		$str = str_replace("|","", $str);
  		$str = str_replace("..","", $str);
	}
  return $str;
}


$filesDir = getStoragePath()."\\encfiles";
if (!is_dir($filesDir))
{
	mkdir($filesDir);
}
$resultDir = $filesDir."\\result";
if (!is_dir($resultDir))
{
	mkdir($resultDir);
}

$active = glob($filesDir.'\\*.cmd');
$all = glob($resultDir.'\\*.*');

$info = [
	"active" => count($active),
	"total" => get_total_cpu_cores(),
	"inProgress" => [],
	"done" => []
];

foreach ($all as $key)
{
	$pi = pathinfo($key);
	$commandFile = $pi["filename"].".cmd";
	$sourceFile = $pi["filename"];
	if (file_exists($filesDir.'\\'.$sourceFile))
	{
		if (file_exists($filesDir.'\\'.$commandFile))
		{
			$info["inProgress"][] = $sourceFile;
		}
		else
		{
			$info["done"][] = $sourceFile;
		}
	}
}

if (isset($_GET["action"]))
{
	if ($_GET["action"] == "upload" && isset($_FILES['encfile']) && isset($_POST["params"]))
	{
		$params = json_decode(hex2bin($_POST["params"]), true);
		$fileName = $_FILES['encfile']['name'];
		$fileToProcess = $filesDir."\\".$fileName;
		move_uploaded_file($_FILES['encfile']['tmp_name'], $fileToProcess);
		$commandFile = $fileToProcess.".cmd";
		$resultFile = $resultDir."\\".$fileName.$params["outputExt"];

		$command = $params["commandLine"];
		$command = str_replace("%SRC%", $fileToProcess, $command);
		$command = str_replace("%DST%", $resultFile, $command);
		$command .= PHP_EOL.'DEL /Q "'.$commandFile.'"';
		file_put_contents($commandFile, $command);
		pclose(popen('start "" /B "'.$commandFile.'"', "r"));
	}
	if ($_GET["action"] == "info")
	{		
		header("Content-Type: application/json");
		echo json_encode($info);
		die();
	}
	if ($_GET["action"] == "get")
	{
		if (isset($_POST["name"]) && isset($_POST["params"]))
		{
			$params = json_decode(hex2bin($_POST["params"]), true);

			$fileName = antiHack($_POST["name"]);
			$fileToGet = $filesDir."\\".$fileName;
			$commandFile = $fileToGet.".cmd";
			$resultFile = $resultDir."\\".$fileName.$params["outputExt"];
			if (file_exists($fileToGet) && !file_exists($commandFile) && file_exists($resultFile))
			{
				$fp = fopen($resultFile, 'rb');

				header("Content-Type: application/octet-stream");
				header("Content-Length: ".filesize($resultFile));

				fpassthru($fp);
				exit;
			}
		}
	}
	if ($_GET["action"] == "remove")
	{
		if (isset($_POST["name"]) && isset($_POST["params"]))
		{
			$params = json_decode(hex2bin($_POST["params"]), true);

			$fileName = antiHack($_POST["name"]);
			$fileToGet = $filesDir."\\".$fileName;
			$commandFile = $fileToGet.".cmd";
			$resultFile = $resultDir."\\".$fileName.$params["outputExt"];
			if (file_exists($fileToGet) && !file_exists($commandFile))
			{
				if (file_exists($resultFile))
				{
					unlink($resultFile);
				}
				unlink($fileToGet);
				header("Content-Type: application/json");
				echo json_encode([ "result" => true ]);
				die();
			}
		}
		header("Content-Type: application/json");
		echo json_encode([ "result" => false ]);
		die();
	}
}
echo "URL Correct";
?>




Local script to run encode.php encoding
. : , . :



  • c:\Apps\OneDrive\commands\bin\ffmpeg\ffmpeg.exe — Zeranoe builds
  • c:\Apps\OneDrive\commands\bin\ffmpeg\rav1e.exe — rav1e


: 



$servers = [
	"LOCAL" => "http://127.0.0.1:8000/remote/encoding.php",
	"SERVER2" => "http://192.168.100.25:8000/remote/encoding.php",
];


encode.php: 



<?php

$ffmpeg = '"c:\Apps\OneDrive\commands\bin\ffmpeg\ffmpeg.exe"';

$params = [
	"commandLine" => '"c:\Apps\OneDrive\commands\bin\ffmpeg\ffmpeg" -i "%SRC%" -an -pix_fmt yuv420p -f yuv4mpegpipe - | "c:\Apps\OneDrive\commands\bin\ffmpeg\rav1e" - -s 5 --quantizer 130  -y --output "%DST%"',
	"outputExt" => ".ivf"
];


$paramsData = bin2hex(json_encode($params));

$servers = [
	"LOCAL" => "http://127.0.0.1:8000/remote/encoding.php",
	"SERVER2" => "http://192.168.100.25:8000/remote/encoding.php",
];

if (isset($argc))
{
	if ($argc > 1)
	{
		$fileToEncode = $argv[1];

		$timeBegin = time();
		$pi = pathinfo($fileToEncode);
		$filePartName = $pi["dirname"]."\\".$pi["filename"]."_part%04d.mkv";
		$fileList = $pi["dirname"]."\\".$pi["filename"]."_list.txt";
		$joinedFileName = $pi["dirname"]."\\".$pi["filename"]."_joined.mkv";
		$audioFileName = $pi["dirname"]."\\".$pi["filename"]."_audio.opus";
		$finalFileName = $pi["dirname"]."\\".$pi["filename"]."_AV1.mkv";
		exec($ffmpeg.' -i "'.$fileToEncode.'" -c copy -an -segment_time 00:00:10 -reset_timestamps 1 -f segment -y "'.$filePartName.'"');
		exec($ffmpeg.' -i "'.$fileToEncode.'" -vn -acodec libopus -ab 128k -y "'.$audioFileName.'"');

		$files = glob($pi["dirname"]."\\".$pi["filename"]."_part*.mkv");

		$sourceParts = $files;
		$resultParts = [];
		$resultFiles = [];
		$inProgress = [];
		while (count($files) || count($inProgress))
		{
			foreach ($servers as $server => $url)
			{
				if( $curl = curl_init() )
				{
					curl_setopt($curl, CURLOPT_URL, $url."?action=info");
					curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
					$out = curl_exec($curl);
					curl_close($curl);

					$info = json_decode($out, true);
					//var_dump($info);

					if (count($files))
					{
						if (intval($info["active"]) < intval($info["total"]))
						{
							$fileName = $files[0];
							$key = pathinfo($fileName)["basename"];
							$inProgress[] = $key;
							//echo "Server: ".$url."\r\n";
							echo "Sending part ".$key."[TO ".$server."]...";
							if (!in_array($key, $info["done"]) && !in_array($key, $info["inProgress"]))
							{
								$cFile = curl_file_create($fileName);

								$post = ['encfile'=> $cFile, 'params' => $paramsData];
								$ch = curl_init();
								curl_setopt($ch, CURLOPT_URL, $url."?action=upload");
								curl_setopt($ch, CURLOPT_POST,1);
								curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
								curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
								$result = curl_exec($ch);
								curl_close ($ch);
							}
							echo " DONE\r\n";
							echo "  Total: ".count($sourceParts).", In Progress: ".count($inProgress).", Left: ".count($files)."\r\n";
							$files = array_slice($files, 1);
						}
					}

					if (count($info["done"]))
					{
						foreach ($info["done"] as $file)
						{
							if (($key = array_search($file, $inProgress)) !== false)
							{
								set_time_limit(0);
								
								echo "Receiving part ".$file."... [FROM ".$server."]...";
								$resultFile = $pi["dirname"]."\\".$file.".result".$params["outputExt"];
								$fp = fopen($resultFile, 'w+');
								$post = ['name' => $file, 'params' => $paramsData];
								$ch = curl_init();
								curl_setopt($ch, CURLOPT_URL, $url."?action=get");
								curl_setopt($ch, CURLOPT_POST,1);
								curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
								curl_setopt($ch, CURLOPT_FILE, $fp); 
								curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
								//curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
								curl_exec($ch); 
								curl_close($ch);
								//fclose($fp);

								$resultFiles[] = "file ".$resultFile;
								$resultParts[] = $resultFile;

								$post = ['name' => $file, 'params' => $paramsData];
								$ch = curl_init();
								curl_setopt($ch, CURLOPT_URL, $url."?action=remove");
								curl_setopt($ch, CURLOPT_POST,1);
								curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
								curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
								curl_exec($ch); 
								curl_close($ch);
								fclose($fp);

								unset($inProgress[$key]);

								echo " DONE\r\n";
								echo "  Total: ".count($sourceParts).", In Progress: ".count($inProgress).", Left: ".count($files)."\r\n";
							}
						}
					}
				}
			}
			usleep(300000);
		}

		asort($resultFiles);
		file_put_contents($fileList, str_replace("\\", "/", implode("\r\n", $resultFiles)));

		exec($ffmpeg.' -safe 0 -f concat -i "'.$fileList.'" -c copy -y "'.$joinedFileName.'"');
		exec($ffmpeg.' -i "'.$joinedFileName.'" -i "'.$audioFileName.'" -c copy -y "'.$finalFileName.'"');

		unlink($fileList);
		unlink($audioFileName);
		unlink($joinedFileName);
		foreach ($sourceParts as $part)
		{
			unlink($part);
		}
		foreach ($resultParts as $part)
		{
			unlink($part);
		}

		echo "Total Time: ".(time() - $timeBegin)."s\r\n";
	}
}

?>






The file to run the encoding script is next to the script. You configure the path to PHP yourself.

encoding.cmd: 

@ECHO OFF
cd /d %~dp0
SET /p FILENAME=Drag'n'Drop file here and Press Enter: 
..\php7\php.exe -c ..\php7\php_standalone.ini encode.php "%FILENAME%"
PAUSE


Go?



For the test, I used the famous Big Bucks Bunny cartoon about a rabbit , 10 minutes long and 150MB in size.



Iron



  • AMD Ryzen 5 1600 (12 Threads) + 16GB DDR4 (Windows 10)
  • Intel Core i7 4770 (8 threads) + 32GB DDR3 (Windows 10)
  • Intel Core i5 3570 (4 threads) + 8GB DDR3 (Windows 10)
  • Intel Xeon E5-2650 V2 (16 threads) + 32GB DDR3 (Windows 10)


Total: 40 threads



Command line with parameters



ffmpeg -i "%SRC%" -an -pix_fmt yuv420p -f yuv4mpegpipe - | rav1e - -s 5 --quantizer 130  -y --output "%DST%


results



Encoding time: 55 minutes

Video size: 75MB



I will not speak for the quality, because the selection of the optimal encoding parameters is a task of the day before, and today I was pursuing the goal of achieving a reasonable encoding time and it seems to me it worked out. I was afraid that the glued pieces would stick together badly and there would be twitching at these moments, but no, the result went smoothly, without any jerks.



Separately, I note that 1080p requires about a gigabyte of RAM per stream, so there should be a lot of memory. Also note that towards the end the herd is running at the speed of the slowest ram and while the Ryzen and i7 have long since finished coding, the Xeon and i5 were still chugging over their pieces. Those. a longer video in general would be encoded at a higher overall fps at the expense of the faster cores doing more work.



Running the conversion on one Ryzen 5 1600 with multithreading, the maximum I had was about 1.5 fps. Here, given that the last 10 minutes of encoding are finishing off the last pieces with slow cores, we can say that it turned out about 5-6 fps, which is not so little for such an advanced codec. That's all I wanted to share, I hope someone might find it useful.


More articles:


All Articles