diff options
Diffstat (limited to 'TODO')
| -rw-r--r-- | TODO | 99 |
1 files changed, 79 insertions, 20 deletions
@@ -9,7 +9,6 @@ Allow recording a region by recording the compositor proxy window / nvfbc window Use nvenc directly, which allows removing the use of cuda. Handle xrandr monitor change in nvfbc. Implement follow focused in drm. -Support fullscreen capture on amd/intel using external kms process. Support amf and qsv. Disable flipping on nvidia? this might fix some stuttering issues on some setups. See NvCtrlGetAttribute/NvCtrlSetAttributeAndGetStatus NV_CTRL_SYNC_TO_VBLANK https://github.com/NVIDIA/nvidia-settings/blob/d5f022976368cbceb2f20b838ddb0bf992f0cfb9/src/gtk%2B-2.x/ctkopengl.c. Replays seem to have some issues with audio/video. Why? @@ -35,23 +34,9 @@ NVreg_RegistryDwords. Restore nvfbc screen recording on monitor reconfiguration. Window capture doesn't work properly in _control_ game after going from pause menu to in-game (and back to pause menu). There might be some x11 event we need to catch. Same for vr-video-player. -Fix constant framerate not working properly on amd/intel because capture framerate gets locked to the same framerate as game framerate, which doesn't work well when you need to encode multiple duplicate frames. We can skip multiple encode if we duplicate frame once and then use that same frame data as the difference between frames will be exactly the same, but hevc complains about that. Is there a way to make hevc shut up? - -JPEG color range on amd seems to produce too bright video with h264 but not hevc, why? - -Properly handle monitor reconfiguration (kms vaapi, nvfbc). - -Better configure vaapi. The file size is too large. -Better colors for vaapi. It looks a bit off when recording vscode for example. - -Clear vaapi surface (for focused window). - -Window capture performance on steam deck isn't good when playing the witcher 3 for example. The capture itself is fine but video encoding puts it to 30fps even if the game runs at 57 fps. - Monitor capture on steam deck is slightly below the game fps, but only when capturing on the steam deck screen. If capturing on another monitor, there is no issue. Is this related to the dma buf rotation issue? different modifier being slow? does this always happen? -Make sure rgb to yuv color conversion is 100% correct. Fallback to vaapi copy in kms if opengl version fails. This can happen on steam deck for some reason (driver bug?). Also vaapi copy uses less gpu since it uses video codec unit to copy. Test if vaapi copy version uses less memory than opengl version. @@ -59,10 +44,84 @@ Intel is a bit weird with monitor capture and multiple monitors. If one of the m Is that only the case when the primary monitor is rotated? Also the primary monitor becomes position 0, 0 so crtc (x11 randr) position doesn't match the drm pos. Maybe get monitor position and size from drm instead. How about if multiple monitors are rotated? -When using multiple monitors kms grab the target monitor instead of the whole screen. -Enable opus/flac again. It's broken right now when merging audio inputs. The audio gets a lot of static noise! +Support vp8/vp9. This is especially important on amd which on some distros (such as Manjaro) where hardware accelerated h264/hevc is disabled in the mesa package. + +Support screen (all monitors) capture on amd/intel and nvidia wayland when no combined plane is found. Right now screen just takes the first output. +Use separate plane (which has offset and pitch) from combined plane instead of the combined plane. + +Both twitch and youtube support variable bitrate but twitch recommends constant bitrate to reduce stream buffering/dropped frames when going from low motion to high motion: https://help.twitch.tv/s/article/broadcasting-guidelines?language=en_US. Info for youtube: https://support.google.com/youtube/answer/2853702?hl=en#zippy=%2Cvariable-bitrate-with-custom-stream-keys-in-live-control-room%2Ck-p-fps%2Cp-fps. + +Limit fps recording with x damage. This is good when running replay mode 24/7 and being afk or when not much is happening on the screen. + +On nvidia some games apparently causes the game to appear to stutter (without dropping fps) when recording a monitor but not using + when using direct screen capture. Observed in Deus Ex and Apex Legends. + +Support "screen" (all monitors) capture on wayland. This should be done by getting all drm fds and multiple EGL_DMA_BUF_PLANEX_FD_EXT to create one egl image with all fds combined. + +Support pipewire screen capture? +CPU usage is pretty high on AMD/Intel/(Nvidia(wayland)), why? opening and closing fds, creating egl, cuda association, is slow when done every frame. Test if desktop portal screencast has better performance. + +Capture is broken on amd on wlroots. It's disabled at the moment and instead uses kms capture. Find out why we get a black screen in wlroots. + +Support vulkan video encoding. That might workaround forced p2 state nvidia driver "bug". Ffmpeg supports vulkan video encoding if it's encoding with --enable-vulkan + +It may be possible to improve color conversion rgb->yuv shader for color edges by biasing colors to an edge, instead of letting color overlaying with bilinear filtering handle it. + +When webcam is supported mention that nvidia_drm.modeset=1 must be set on nvidia x11 (it's required on wayland so it's not needed there. Or does eglstream work without it??). Check if this really is the case. + Support green screen removal, cropping, shader effects in general (circle mask, rounded corners, etc). + +Preset is set to p5 for now but it should ideally be p6 or p7. + This change is needed because for certain sizes of a window (or monitor?) such as 971x780 causes encoding to freeze + when using h264 codec. This is a new(?) nvidia driver bug. + Maybe dont choose p6 or p7 again? it causes micro stutter for some users (?). + +For low latency, see https://developer.download.nvidia.com/compute/nvenc/v4.0/NVENC_VideoEncoder_API_ProgGuide.pdf (section 7.1). +Remove follow focused option. + +Exit if X11/Wayland killed (if drm plane dead or something?) + +Use SRC_W and SRC_H for screen plane instead of crtc_w and crtc_h. + +Make it possible to select which /dev/dri/card* to use, but that requires opengl to also use the same card. Not sure if that is possible for amd, intel and nvidia without using vulkan instead. + +Support I915_FORMAT_MOD_Y_TILED_CCS (and other power saving modifiers, see https://trac.ffmpeg.org/ticket/8542). The only fix may be to use desktop portal for recording. This issue doesn't appear on x11 since these modifiers are not used by xorg server. + +Test if p2 state can be worked around by using pure nvenc api and overwriting cuInit/cuCtxCreate* to not do anything. Cuda might be loaded when using nvenc but it might not be used, with certain record options? (such as h264 p5). + nvenc uses cuda when using b frames and rgb->yuv conversion, so convert the image ourselves instead.- + +Mesa doesn't support global headers (AV_CODEC_FLAG_GLOBAL_HEADER) with h264... which also breaks mkv since mkv requires global header. Right now gpu screen recorder will forcefully set video codec to hevc when h264 is requested for mkv files. + +Drop frames if live streaming cant keep up with target fps, or dynamically change resolution/quality. + +Support low power option (does it even work with vaapi in ffmpeg??). Would be very useful for steam deck. + +Instead of sending a big list of drm data back to kms client, send the monitor we want to record to kms server and the server should respond with only the matching monitor, and cursor. + +Tonemap hdr to sdr when hdr is enabled and when hevc_hdr/av1_hdr is not used. + +Add 10 bit record option, h264_10bit, hevc_10bit and av1_10bit. + +Rotate cursor texture properly (around top left origin). + +Setup hardware video context so we can query constraints and capabilities for better default and better error messages. + +Use CAP_SYS_NICE in flatpak too on the main gpu screen recorder binary. It makes recording smoother, especially with constant framerate. + +Show error when using compressed kms plane which isn't supported. Also do that in the gui. + +Modify ffmpeg to accept opengl texture for nvenc encoding. Removes extra buffers and copies. + +When vulkan encode is added, mention minimum nvidia driver required. (550.54.14?). + +Support drm plane rotation. Neither X11 nor any Wayland compositor currently rotates drm planes so this might not be needed. + +Investigate if there is a way to do gpu->gpu copy directly without touching system ram to enable video encoding on a different gpu. On nvidia this is possible with cudaMemcpyPeer, but how about from an intel/amd gpu to an nvidia gpu or the other way around or any combination of iGPU and dedicated GPU? + Maybe something with clEnqueueMigrateMemObjects? on AMD something with DirectGMA maybe? + +Go back to using pure vaapi without opengl for video encoding? rotation (transpose) can be done if its done after (rgb to yuv) color conversion. + +Implement scaling and use lanczos resampling for better quality. Lanczos resampling can also be used for YUV chroma for better color quality on small text. -Support vp8/vp9/av1. This is especially important on amd which on some distros (such as Manjaro) where hardware accelerated h264/hevc is disabled in the mesa package. +Try fixing HDR by passing HDR+10 data as well, and in the packet. Run "ffprobe -loglevel quiet -read_intervals "%+#2" -select_streams v:0 -show_entries side_data video.mp4" to test if the file has correct metadata. -Support screen (all monitors) capture on amd/intel when no combined plane is found. -Use separate plane (which has offset and pitch) from combined plane instead of the combined plane.
\ No newline at end of file +Flac is disabled because the frame sizes are too large which causes big audio/video desync.
\ No newline at end of file |