Down- or Upscaling with AVF Capture Presets will result in corrupted files, when using 10-Bit video mode. 8-Bit video mode will work fine. This seems to be a macOS Mojave bug, as with macOS Catalina Beta it works fine.
Recording in HD Uncompressed (8- or 10-Bit) will result in an Error. This seems to be a macOS Mojave bug, as with macOS Catalina Beta it works fine.