rtmp: Always call rtmp_close() on rtmp_open() failure

Prevent possible memory leaks.

Connect to nginx and request a non-existent resource to
trigger the issue.

CC: libav-stable@libav.org

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Uwe L. Korn <uwelk@xhochy.com>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>