path: root/TODO

Check for reparent.
Quickly changing workspace and back while recording under i3 breaks the screen recorder. i3 probably unmaps windows in other workspaces.
See https://trac.ffmpeg.org/wiki/EncodingForStreamingSites for optimizing streaming.
Look at VK_EXT_external_memory_dma_buf.
Allow setting a different output resolution than the input resolution.
Use mov+faststart.
Allow recording all monitors/selected monitor without nvfbc by recording the compositor proxy window and only recording the part that matches the monitor(s).
Allow recording a region by recording the compositor proxy window / nvfbc window and copying part of it.
Use nvenc directly, which allows removing the use of cuda.
Handle xrandr monitor change in nvfbc.
Implement follow focused in drm.
Support amf and qsv.
Disable flipping on nvidia? this might fix some stuttering issues on some setups. See NvCtrlGetAttribute/NvCtrlSetAttributeAndGetStatus NV_CTRL_SYNC_TO_VBLANK https://github.com/NVIDIA/nvidia-settings/blob/d5f022976368cbceb2f20b838ddb0bf992f0cfb9/src/gtk%2B-2.x/ctkopengl.c.
Replays seem to have some issues with audio/video. Why?
Cleanup unused gl/egl functions, macro, etc.
Add option to disable overlapping of replays (the old behavior kinda. Remove the whole replay buffer data after saving when doing this).
Set audio track name to audio device name (if not merge of multiple audio devices).
Add support for webcam, but only really for amd/intel because amd/intel can get drm fd access to webcam, nvidia cant. This allows us to create an opengl texture directly from the webcam fd for optimal performance.
Reverse engineer nvapi so we can disable "force p2 state" on linux too (nvapi profile api with the settings id 0x50166c5e).
Support yuv444p on amd/intel.
fix yuv444 for hevc.
Do not allow streaming if yuv444.
Re-enable yuv444.
Support 10 bit output because of better gradients. May even be smaller file size. Better supported on hevc (not supported at all on h264 on my gpu).
Add nvidia/(amd/intel) specific install script for ubuntu. User should run install_ubuntu.sh but it should run different install dep script depending on if /proc/driver/nvidia/version exists or not. But what about switchable graphics setup?
Test different combinations of switchable graphics. Intel hybrid mode (running intel but possible to run specific applications with prime-run), running pure intel. Detect switchable graphics.

https://web.archive.org/web/20210306020203/https://forums.developer.nvidia.com/t/performance-power-management-problem-on-shared-vgpu/161986
https://djdallmann.github.io/GamingPCSetup/CONTENT/RESEARCH/FINDINGS/registrykeys_displayadapter_class_4d36e968-e325-11ce-bfc1-08002be10318.txt

The video output will be black if if the system is suspended on nvidia and NVreg_PreserveVideoMemoryAllocations is not set to 1. This happens because I think that the driver invalidates textures/cuda buffers? To fix this we could try and recreate gsr capture when gsr_capture_capture fails (with timeout to retry again).

NVreg_RegistryDwords.
Restore nvfbc screen recording on monitor reconfiguration.
Window capture doesn't work properly in _control_ game after going from pause menu to in-game (and back to pause menu). There might be some x11 event we need to catch. Same for vr-video-player.

Fix constant framerate not working properly on amd/intel because capture framerate gets locked to the same framerate as game framerate, which doesn't work well when you need to encode multiple duplicate frames. We can skip multiple encode if we duplicate frame once and then use that same frame data as the difference between frames will be exactly the same, but hevc complains about that. Is there a way to make hevc shut up?

Properly handle monitor reconfiguration (kms vaapi, nvfbc).

Better configure vaapi. The file size is too large.

Clear vaapi surface (for focused window).

Window capture performance on steam deck isn't good when playing the witcher 3 for example. The capture itself is fine but video encoding puts it to 30fps even if the game runs at 57 fps.

Monitor capture on steam deck is slightly below the game fps, but only when capturing on the steam deck screen. If capturing on another monitor, there is no issue.
    Is this related to the dma buf rotation issue? different modifier being slow? does this always happen?

Fallback to vaapi copy in kms if opengl version fails. This can happen on steam deck for some reason (driver bug?). Also vaapi copy uses less gpu since it uses video codec unit to copy.
Test if vaapi copy version uses less memory than opengl version.

Intel is a bit weird with monitor capture and multiple monitors. If one of the monitors is rotated then all the kms will be rotated as well.
    Is that only the case when the primary monitor is rotated? Also the primary monitor becomes position 0, 0 so crtc (x11 randr) position doesn't match the drm pos. Maybe get monitor position and size from drm instead.
    How about if multiple monitors are rotated?

Support vp8/vp9. This is especially important on amd which on some distros (such as Manjaro) where hardware accelerated h264/hevc is disabled in the mesa package.

Support screen (all monitors) capture on amd/intel and nvidia wayland when no combined plane is found. Right now screen just takes the first output.
Use separate plane (which has offset and pitch) from combined plane instead of the combined plane.

Both twitch and youtube support variable bitrate but twitch recommends constant bitrate to reduce stream buffering/dropped frames when going from low motion to high motion: https://help.twitch.tv/s/article/broadcasting-guidelines?language=en_US. Info for youtube: https://support.google.com/youtube/answer/2853702?hl=en#zippy=%2Cvariable-bitrate-with-custom-stream-keys-in-live-control-room%2Ck-p-fps%2Cp-fps.

Limit fps recording with x damage. This is good when running replay mode 24/7 and being afk or when not much is happening on the screen.

On nvidia some games apparently causes the game to appear to stutter (without dropping fps) when recording a monitor but not using
    when using direct screen capture. Observed in Deus Ex and Apex Legends.

Support "screen" (all monitors) capture on wayland. This should be done by getting all drm fds and multiple EGL_DMA_BUF_PLANEX_FD_EXT to create one egl image with all fds combined.

Support pipewire screen capture?
Support screen rotation.
When nvidia supports hardware cursor then capture the cursor. Right now the cursor is captured because it's a software cursor so it's composed on the dma buf.
CPU usage is pretty high on AMD/Intel/(Nvidia(wayland)), why? opening and closing fds, creating egl, cuda association, is slow when done every frame. Test if desktop portal screencast has better performance.

Capture is broken on amd on wlroots. It's disabled at the moment and instead uses kms capture. Find out why we get a black screen in wlroots.

First video and audio frame should be posted immediately instead of waiting 1000/fps milliseconds, to improve latency for remote desktop future functionality.

Support vulkan video encoding. That might workaround forced p2 state nvidia driver "bug". Ffmpeg supports vulkan video encoding if it's encoding with --enable-vulkan

It may be possible to improve color conversion rgb->yuv shader for color edges by biasing colors to an edge, instead of letting color overlaying with bilinear filtering handle it.

When webcam is supported mention that nvidia_drm.modeset=1 must be set on nvidia x11 (it's required on wayland so it's not needed there. Or does eglstream work without it??). Check if this really is the case.
  Support green screen removal, cropping, shader effects in general (circle mask, rounded corners, etc).

Use vfr on nvidia x11 as well, otherwise network data could slow it down to below target fps and mess it up.

Preset is set to p5 for now but it should ideally be p6 or p7.
    This change is needed because for certain sizes of a window (or monitor?) such as 971x780 causes encoding to freeze
    when using h264 codec. This is a new(?) nvidia driver bug.
    Maybe dont choose p6 or p7 again? it causes micro stutter for some users (?).

For low latency, see https://developer.download.nvidia.com/compute/nvenc/v4.0/NVENC_VideoEncoder_API_ProgGuide.pdf (section 7.1).
Remove follow focused option.

Overclocking (-oc) can overclock too much on some systems. Maybe remove the option?

Exit if X11/Wayland killed (if drm plane dead or something?)

Use SRC_W and SRC_H for screen plane instead of crtc_w and crtc_h.

Make it possible to select which /dev/dri/card* to use, but that requires opengl to also use the same card. Not sure if that is possible for amd, intel and nvidia without using vulkan instead.

Support I915_FORMAT_MOD_Y_TILED_CCS (and other power saving modifiers, see https://trac.ffmpeg.org/ticket/8542). The only fix may be to use desktop portal for recording. This issue doesn't appear on x11 since these modifiers are not used by xorg server.

Test if p2 state can be worked around by using pure nvenc api and overwriting cuInit/cuCtxCreate* to not do anything. Cuda might be loaded when using nvenc but it might not be used, with certain record options? (such as h264 p5).
    nvenc uses cuda when using b frames and rgb->yuv conversion, so convert the image ourselves instead.-

Mesa doesn't support global headers (AV_CODEC_FLAG_GLOBAL_HEADER) with h264... which also breaks mkv since mkv requires global header. Right now gpu screen recorder will forcefully set video codec to hevc when h264 is requested for mkv files.

Drop frames if live streaming cant keep up with target fps, or dynamically change resolution/quality.