2

I currently have two ffmpeg videos. Video one is a matte key file , that looks like so

Example of output

Video two, would be the normal video without the background and just the person.

How would I go about merging these two videos in ffmpeg to have it look like this

enter image description here

I've tried the command

ffmpeg -i word.mp4 -i word.matte.mp4 -filter_complex "[0:v][1:v]alphamerge" -shortest -c:v qtrle -an output.mp4

but I get the following error

ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'word.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.29.100
  Duration: 00:00:06.24, start: 0.000000, bitrate: 741 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 738 kb/s, 25.79 fps, 25.79 tbr, 11040 tbn, 51.59 tbc (default)
    Metadata:
      handler_name    : VideoHandler
Input #1, mov,mp4,m4a,3gp,3g2,mj2, from 'word.matte.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf58.29.100
  Duration: 00:00:06.19, start: 0.000000, bitrate: 493 kb/s
    Stream #1:0(und): Video: mpeg4 (Simple Profile) (mp4v / 0x7634706D), yuv420p, 568x320 [SAR 1:1 DAR 71:40], 491 kb/s, 26 fps, 26 tbr, 13312 tbn, 26 tbc (default)
    Metadata:
      handler_name    : VideoHandler
Stream mapping:
  Stream #0:0 (h264) -> alphamerge:main
  Stream #1:0 (mpeg4) -> alphamerge:alpha
  alphamerge -> Stream #0:0 (qtrle)
Press [q] to stop, [?] for help
[swscaler @ 0x5619d30b9900] No accelerated colorspace conversion found from yuv420p to argb.
[Parsed_alphamerge_0 @ 0x5619d2f57700] Input frame sizes do not match (1280x720 vs 568x320).
[Parsed_alphamerge_0 @ 0x5619d2f57700] Failed to configure output pad on Parsed_alphamerge_0
Error reinitializing filters!
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #1:0

Any help would be appreciated. Here is a link to both video files, the matte file, and the normal video

Edit: Here is the example of this working. https://backgroundremover.app/video

nadermx
  • 667
  • 1
  • 10
  • 32

2 Answers2

2

Make a new mask to match the video size

Your mask is smaller than the video (568x320 vs 1280x720). Optimal solution is to make a new mask that matches the size and aspect ratio of the video. This will avoid the need to upscale the mask and will provide the best results. Avoid upscaling when possible.

If you have to use the original mask

You can use the scale2ref filter to upscale the mask to match the video:

ffmpeg -i video.mp4 -i mask.mp4 -filter_complex "[1][0]scale2ref[mask][main];[main][mask]alphamerge=shortest=1" -c:v qtrle output.mov

See the scale2ref filter documentation for more details.

llogan
  • 57,139
  • 15
  • 118
  • 145
  • LIterally what I said. – WG481 May 10 '21 at 17:27
  • @llogan I have updated the question with the example video files. Your command does not give an error, but does not remove the background and instead just gives the same video with the background – nadermx May 10 '21 at 18:22
  • @llogan I have gone ahead and put the originals, with the different sizes. Also is there any way to get it to auto detect the range, although that may be outside the scope of this question. And I currently need to upscale the matte file, as to create a original size matte is too expensive in terms of cpu/gpu for my program currently. – nadermx May 11 '21 at 17:46
  • @nadermx See updated answer showing basic scale2ref usage. – llogan May 11 '21 at 18:03
  • @llogan it errores out with the shortest command, if I take it out, it just converts the original mp4 to a mov, with the background still there. – nadermx May 11 '21 at 18:23
  • @nadermx Your ffmpeg is too old for `shortest=1`. Remove it, or better option is to use a recent ffmpeg. [Download](https://johnvansickle.com/ffmpeg/) and put it in `/usr/local/bin`, or follow [Ubuntu ffmpeg compile/install](https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu) guide. Output works for me if I try 4.2.4 and play using `mpv`. – llogan May 11 '21 at 18:51
  • @llogan I have have 4.2.4, it gives this error `[Parsed_alphamerge_1 @ 0x564520e36100] This filter does not take any options, but options were provided: shortest=1. [AVFilterGraph @ 0x564520e2d940] Error initializing filter 'alphamerge' with args 'shortest=1' ` – nadermx May 11 '21 at 20:44
  • 1
    @nadermx Your version of ffmpeg is too old and does not support `shortest=1`. Use `ffmpeg -i video.mp4 -i mask.mp4 -filter_complex "[1][0]scale2ref[mask][main];[main][mask]alphamerge" -c:v qtrle output.mov` or upgrade your ffmpeg. What I meant in my last comment is that I tried 4.2.4 **without** `shortest=1`, and then viewed `output.mov` with the player named `mpv` and it had a transparent background. Best recommendation is to upgrade your ffmpeg. – llogan May 11 '21 at 21:25
  • As I said, I have `ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers` and when I run this, when I see the video in VLC it still shows it with the background, https://backgroundremover.app/static/output.mov – nadermx May 12 '21 at 03:05
  • 1
    @nadermx Blame VLC. Try mpv. Verify that the mask works by outputting PNG: `ffmpeg -i word.mp4 -i word.matte.mp4 -filter_complex "[1][0]scale2ref[mask][main];[main][mask]alphamerge" output_%03d.png` or overlaying on color `ffmpeg -i word.mp4 -i word.matte.mp4 -filter_complex "[1][0]scale2ref[mask][main];[main][mask]alphamerge1[fg];color=c=magenta:s=1280x720[bg];[bg][fg]overlay=format=auto:shortest=1,format=yuv420p" output.mp4` – llogan May 12 '21 at 16:20
  • @llogan I believe the reason shortest wasn't working, is cause your sytax was wrong, i think it should be `-shortest` as an alternative to yours https://stackoverflow.com/a/65911572/4180276 – nadermx May 27 '21 at 07:38
0

I read through your error log and found this interesting line of text:
[Parsed_alphamerge_0 @ 0x5619d2f57700] Input frame sizes do not match (1280x720 vs 568x320).

In this error line, it says the sizes of both frames don't match, and thus, cannot be put together. By downscaling/upscaling one of your files to match the other's resolution, the videos should be put together with ease.

WG481
  • 240
  • 1
  • 9
  • I agree with llogan that your answer simply showed me the error log, which I had already read, but was unsure of how to fix. – nadermx May 12 '21 at 16:51