Segfault in VideoOutputXv::PrepareFrameMem() with certain aspect ratios.

[myth@…]

When running mythfrontend in a window in a vnc session (for testing purposes), I noticed that with certain window sizes I get a segfault in VideoOutputXv::PrepareFrameMem(). The problem appears to be related to the allocation of the sbuf buffer in that method, if I double the size, the segfault goes away, but I get skewed video output (the bottom left corner is shifted over to being the bottom right corner, at a 45 degree angle, so something seems to be off by one pixel per line or somesuch). The allocation in question is this line:

unsigned char *sbuf = new unsigned char[

display_visible_rect.width() * display_visible_rect.height() * 3 / 2];

The segfault is from deleting that buffer later on. I see the problem when I run mythfrontend in a window with the size 711x400px, if I change the size to 712x400, the segfault goes away.

The mythtv version is as follows:

MythTV Version : 21870M MythTV Branch : trunk Network Protocol : 48 Library API : 0.22.20090912-2 QT Version : 4.5.2 Options compiled in:

linux release using_oss using_alsa using_backend using_dvb using_frontend using_hdhomerun using_hdpvr using_iptv using_ivtv using_joystick_menu using_lirc using_mheg using_opengl_video using_opengl_vsync using_qtwebkit using_v4l using_x11 using_xrandr using_xv using_xvmc using_xvmc_vld using_xvmcw using_bindings_perl using_bindings_python using_opengl using_ffmpeg_threads using_live using_mheg

This is all using the standard CPU+ playback profile.

Valgrind had this to say about the segfault:

==14485== ==14485== Thread 11: ==14485== Invalid write of size 1 ==14485== at 0x6D28E3A: swScale_MMX2 (swscale.c:536) ==14485== by 0x6D1E4B7: sws_scale (swscale.c:3206) ==14485== by 0x56BAB40: VideoOutputXv::PrepareFrameMem(VideoFrame_*, FrameScanType) (videoout_xv.cpp:2207) ==14485== by 0x56BB4E3: VideoOutputXv::PrepareFrame(VideoFrame_*, FrameScanType) (videoout_xv.cpp:2258) ==14485== by 0x554E9BB: NuppelVideoPlayer::AVSync() (NuppelVideoPlayer.cpp:2489) ==14485== by 0x5552DD8: NuppelVideoPlayer::DisplayNormalFrame() (NuppelVideoPlayer.cpp:2946) ==14485== by 0x5553E22: NuppelVideoPlayer::OutputVideoLoop() (NuppelVideoPlayer.cpp:3085) ==14485== by 0x5553F9D: NuppelVideoPlayer::kickoffOutputVideoLoop(void*) (NuppelVideoPlayer.cpp:3163) ==14485== by 0x99EF3D9: start_thread (in /lib64/libpthread-2.9.so) ==14485== by 0xC38D2BC: clone (in /lib64/libc-2.9.so) ==14485== Address 0x2e0eadb0 is 0 bytes after a block of size 426,600 alloc'd ==14485== at 0x4C2561C: operator new[](unsigned long) (vg_replace_malloc.c:274) ==14485== by 0x56BA604: VideoOutputXv::PrepareFrameMem(VideoFrame_*, FrameScanType) (videoout_xv.cpp:2176) ==14485== by 0x56BB4E3: VideoOutputXv::PrepareFrame(VideoFrame_*, FrameScanType) (videoout_xv.cpp:2258) ==14485== by 0x554E9BB: NuppelVideoPlayer::AVSync() (NuppelVideoPlayer.cpp:2489) ==14485== by 0x5552DD8: NuppelVideoPlayer::DisplayNormalFrame() (NuppelVideoPlayer.cpp:2946) ==14485== by 0x5553E22: NuppelVideoPlayer::OutputVideoLoop() (NuppelVideoPlayer.cpp:3085) ==14485== by 0x5553F9D: NuppelVideoPlayer::kickoffOutputVideoLoop(void*) (NuppelVideoPlayer.cpp:3163) ==14485== by 0x99EF3D9: start_thread (in /lib64/libpthread-2.9.so) ==14485== by 0xC38D2BC: clone (in /lib64/libc-2.9.so) ==14485== ==14485== Invalid read of size 4 ==14485== at 0x6D2E720: yuv420_rgb16_MMX2 (yuv2rgb_template.c:193) ==14485== by 0x6D1E4B7: sws_scale (swscale.c:3206) ==14485== by 0x530906B: myth_sws_img_convert(AVPicture*, PixelFormat, AVPicture*, PixelFormat, int, int) (myth_imgconvert.cpp:54) ==14485== by 0x56BAC01: VideoOutputXv::PrepareFrameMem(VideoFrame_*, FrameScanType) (videoout_xv.cpp:2219) ==14485== by 0x56BB4E3: VideoOutputXv::PrepareFrame(VideoFrame_*, FrameScanType) (videoout_xv.cpp:2258) ==14485== by 0x554E9BB: NuppelVideoPlayer::AVSync() (NuppelVideoPlayer.cpp:2489) ==14485== by 0x5552DD8: NuppelVideoPlayer::DisplayNormalFrame() (NuppelVideoPlayer.cpp:2946) ==14485== by 0x5553E22: NuppelVideoPlayer::OutputVideoLoop() (NuppelVideoPlayer.cpp:3085) ==14485== by 0x5553F9D: NuppelVideoPlayer::kickoffOutputVideoLoop(void*) (NuppelVideoPlayer.cpp:3163) ==14485== by 0x99EF3D9: start_thread (in /lib64/libpthread-2.9.so) ==14485== by 0xC38D2BC: clone (in /lib64/libc-2.9.so) ==14485== Address 0x2e0eadb0 is 0 bytes after a block of size 426,600 alloc'd ==14485== at 0x4C2561C: operator new[](unsigned long) (vg_replace_malloc.c:274) ==14485== by 0x56BA604: VideoOutputXv::PrepareFrameMem(VideoFrame_*, FrameScanType) (videoout_xv.cpp:2176) ==14485== by 0x56BB4E3: VideoOutputXv::PrepareFrame(VideoFrame_*, FrameScanType) (videoout_xv.cpp:2258) ==14485== by 0x554E9BB: NuppelVideoPlayer::AVSync() (NuppelVideoPlayer.cpp:2489) ==14485== by 0x5552DD8: NuppelVideoPlayer::DisplayNormalFrame() (NuppelVideoPlayer.cpp:2946) ==14485== by 0x5553E22: NuppelVideoPlayer::OutputVideoLoop() (NuppelVideoPlayer.cpp:3085) ==14485== by 0x5553F9D: NuppelVideoPlayer::kickoffOutputVideoLoop(void*) (NuppelVideoPlayer.cpp:3163) ==14485== by 0x99EF3D9: start_thread (in /lib64/libpthread-2.9.so) ==14485== by 0xC38D2BC: clone (in /lib64/libc-2.9.so)

--14485-- VALGRIND INTERNAL ERROR: Valgrind received a signal 11 (SIGSEGV) - exiting --14485-- si_code=80; Faulting address: 0x0; sp: 0x410D46DD0

Attached is also the output of:

mythfrontend -w -geometry 711x400 -v playback

of me attempting to play back a recording (720p H264, from an HD-PVR).

Let me know if I can provide any additional information.

[myth@…]

Output of mythfrontend -w -geometry 711x400 -v playback

[markk]

(In [21918]) Prevent a segmentation fault when using odd (i.e. not even) dimensioned output windows and X11 shared memory video output. Closes #7083.