fftools/ffmpeg_opt: Improve checks for truncation/alloc error

Do this by switching from the dynamic buffer API to the AVBPrint API;
the former has no defined way to check for errors.
This also avoids allocating an AVIOContext.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>