diff options
Diffstat (limited to 'TODO')
| -rw-r--r-- | TODO | 177 |
1 files changed, 160 insertions, 17 deletions
@@ -2,15 +2,12 @@ Check for reparent. Quickly changing workspace and back while recording under i3 breaks the screen recorder. i3 probably unmaps windows in other workspaces. See https://trac.ffmpeg.org/wiki/EncodingForStreamingSites for optimizing streaming. Look at VK_EXT_external_memory_dma_buf. -Allow setting a different output resolution than the input resolution. Use mov+faststart. Allow recording all monitors/selected monitor without nvfbc by recording the compositor proxy window and only recording the part that matches the monitor(s). -Allow recording a region by recording the compositor proxy window / nvfbc window and copying part of it. Support amf and qsv. Disable flipping on nvidia? this might fix some stuttering issues on some setups. See NvCtrlGetAttribute/NvCtrlSetAttributeAndGetStatus NV_CTRL_SYNC_TO_VBLANK https://github.com/NVIDIA/nvidia-settings/blob/d5f022976368cbceb2f20b838ddb0bf992f0cfb9/src/gtk%2B-2.x/ctkopengl.c. Replays seem to have some issues with audio/video. Why? Cleanup unused gl/egl functions, macro, etc. -Add option to disable overlapping of replays (the old behavior kinda. Remove the whole replay buffer data after saving when doing this). Set audio track name to audio device name (if not merge of multiple audio devices). Add support for webcam, but only really for amd/intel because amd/intel can get drm fd access to webcam, nvidia cant. This allows us to create an opengl texture directly from the webcam fd for optimal performance. Reverse engineer nvapi so we can disable "force p2 state" on linux too (nvapi profile api with the settings id 0x50166c5e). @@ -69,16 +66,12 @@ Exit if X11/Wayland killed (if drm plane dead or something?) Use SRC_W and SRC_H for screen plane instead of crtc_w and crtc_h. -Make it possible to select which /dev/dri/card* to use, but that requires opengl to also use the same card. Not sure if that is possible for amd, intel and nvidia without using vulkan instead. - Test if p2 state can be worked around by using pure nvenc api and overwriting cuInit/cuCtxCreate* to not do anything. Cuda might be loaded when using nvenc but it might not be used, with certain record options? (such as h264 p5). nvenc uses cuda when using b frames and rgb->yuv conversion, so convert the image ourselves instead.- -Mesa doesn't support global headers (AV_CODEC_FLAG_GLOBAL_HEADER) with h264... which also breaks mkv since mkv requires global header. Right now gpu screen recorder will forcefully set video codec to hevc when h264 is requested for mkv files. - Drop frames if live streaming cant keep up with target fps, or dynamically change resolution/quality. -Support low power option (does it even work with vaapi in ffmpeg??). Would be very useful for steam deck. +Support low power option. Instead of sending a big list of drm data back to kms client, send the monitor we want to record to kms server and the server should respond with only the matching monitor, and cursor. @@ -103,12 +96,10 @@ Investigate if there is a way to do gpu->gpu copy directly without touching syst Go back to using pure vaapi without opengl for video encoding? rotation (transpose) can be done if its done after (rgb to yuv) color conversion. -Implement scaling and use lanczos resampling for better quality. Lanczos resampling can also be used for YUV chroma for better color quality on small text. +Use lanczos resampling for better scaling quality. Lanczos resampling can also be used for YUV chroma for better color quality on small text. Flac is disabled because the frame sizes are too large which causes big audio/video desync. -Add 10-bit capture option. This is good because it reduces banding and quality in very dark areas while reducing the file size compared to doing the same thing with 8-bits. - Enable b-frames. Support vfr matching games exact fps all the time. On x11 use damage tracking, on wayland? maybe there is drm plane damage tracking. But that may not be accurate as the compositor may update it every monitor hz anyways. On wayland maybe only support it for desktop portal + pipewire capture. @@ -152,17 +143,169 @@ Allow prime-run on x11 if monitor capture and the prime gpu is not nvidia. Enable 2-pass encoding. -Add cbr option. - Restart replay/update video resolution if monitor resolution changes. Fix pure vaapi copy on intel. -ffmpeg supports vulkan encoding now (h264!). Doesn't work on amd yet because mesa is missing VK_KHR_video_maintenance1, see https://gitlab.freedesktop.org/mesa/mesa/-/issues/11857. Test on nvidia! - -Test vaapi low latency mode (setenv("AMD_DEBUG", "lowlatencyenc", true);), added in mesa 24.1.4, released on july 17, 2024. Note that this forces gpu power usage to max at all times, even when recording at 2 fps. Use nvidia low latency options for better encoding times. Test ideal async_depth value. Increasing async_depth also increased gpu memory usage a lot (from 100mb to 500mb when moving from async_depth 2 to 16) at 4k resolution. Setting it to 8 increases it by 200mb which might be ok. -Replace -encoder cpu with -k h264_software?
\ No newline at end of file +Replace -encoder cpu with -k h264_software? + +Change vp8/vp9 quality options, right now the file size is too large (for vp9 at least at very_high quality). + +Support recording while in replay mode. This will be needed when enabling replay on system startup with systemd service and wanting to record a video besides that. + The harder and more bloat solution for this would be to make an IPC. + The simple solution would be to use SIGUSR2 for starting/stopping recording since SIGUSR2 is unused for replays. That would mean SIGUSR2 for pausing recording would be ignored. + It also means that the video will be created in the same directory as the replay (or have option to specify another location for that) but the filename would have to be generated automatically. + To rename the file you would have to use -sc to rename it with a script, or add an option to provide a template for the name. + +Dynamically change bitrate/resolution to match desired fps. This would be helpful when streaming for example, where the encode output speed also depends on upload speed to the streaming service. +Implement opengl injection to capture texture. This fixes VRR without having to use NvFBC direct capture and also allows perfect frame timing. +Always use direct capture with NvFBC once the capture issue in mpv fullscreen has been resolved (maybe detect if direct capture fails in nvfbc and switch to non-direct recording. NvFBC says if direct capture fails). + +Support ROI (AV_FRAME_DATA_REGIONS_OF_INTEREST). + +Default to hevc if capture size is larger than 4096 in width or height. + +Set low latency mode on vulkan encoding. + +Support recording/replay/livestreaming at the same time by allowing commands to be run on an existing gpu screen recorder instance. + +Test if `xrandr --output DP-1 --scale 1.5` captures correct size on nvidia. + +Fix cursor position and scale when scaling x11 display. + +Support application audio recording without pipewire combined sink. + +Support transposing (rotating) with vaapi. This isn't supported on many devices with rgb buffer, but its supported with nv12 buffer (on intel at least). + +Cleanup pipewire_audio.c (proper error handling and memory cleanup of proxies). + +Hide application audio module-null-sink by using sink_properties=media.class="Audio/Sink/Internal". + +Improve software encoding performance. + +Add option to record audio from the recorded window only. + +Add option to automatically select best video codec available. Add -k best, -k best_10bit and -k best_hdr. + +Use wayland color management protocol when it's available: https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/14. + +Use different exit codes for different errors. Use one for invalid -w option, another one for invalid -a option for audio devices, etc. This is to make UI error reporting better. + Document these exit codes in an exit code .md file, or finally create a manpage where this can be documented. + +Ffmpeg fixed black bars in videos on amd when using hevc and when recording at some resolutions, such as 1080p: + https://github.com/FFmpeg/FFmpeg/commit/bcfbf2bac8f9eeeedc407b40596f5c7aaa0d5b47 + https://github.com/FFmpeg/FFmpeg/commit/d0facac679faf45d3356dff2e2cb382580d7a521 + Disable gpu screen recorder black bar handling when using hevc on amd when the libavcodec version is the one that comes after those commits. + Also consider the mesa version, to see if the gpu supports this. + The version is libavcodec >= 61.28.100 + +Use opengl compute shader instead of graphics shader. This might allow for better performance when games are using 100% of graphics unit which might fix issue with 100% gpu usage causing gpu screen recorder to run slow when not using vaapi to convert rgb to nv12(?). + +Always disable prime run/dri prime and list all monitors to record from from all cards. + Do this instead of adding an option to choose which gpu to use. + On X11 the primary gpu will always have the framebuffer for all monitors combined. + Use randr to list all monitors and always record and encode with the primary gpu. + On Wayland each gpu will have its own list of monitors with framebuffers. + Iterate through all cards with drm and list all monitors with associated framebuffers and when choosing a monitor to record + automatically use the associated gpu card. + +Allow flv av1 if recent ffmpeg version and streaming to youtube (and twitch?) and for custom services. +Use explicit sync in pipewire video code: https://docs.pipewire.org/page_dma_buf.html. + +Support vaapi rotation. Support for it is added in mesa here: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/32919. + +Replay (and recording?) fails to save properly sometimes (especially for long videos). This is noticable with mp4 files since they get corrupt and become unplayable. + The entire video does seem to get saved (it's a large video file) and it seems to have the correct headers but it's not playable. + +Make it possible to save a shorter replay clip remotely. Maybe implement ipc first, to then also allow starting recording/stream while a replay is running. + +Add an option to pass http headers when streaming. Some streaming services require streaming keys to be passed in a http header instead of in the url as a parameter. + +When adding vulkan video support add VK_VIDEO_ENCODE_TUNING_MODE_LOW_LATENCY_KHR. + +Implement screenshot without invoking opengl (which is slow to start on some systems). + +Automatically use desktop portal on wayland when hdr is enabled (or night light) by checking if kms hdr metadata exists, if hdr video codec is not used. + Or maybe do this in the ui? + +Detect if cached portal session token is no longer valid (this can happen if the user switches to another wayland compositor). + +Support reconnecting (and setting things up again) if the audio server is restarted (for both device recording and app recording). + +Find out how nvidia-smi fixes nvenc not working on opensuse and do that ourselves instead of relying on nvidia-smi that is not always installed. + +Pulseaudio code: add "running" variable to loops to allow stopping the running code when quitting. + +Scale screenshot frame libswscale or implement lanczos shader for improved scaline for video as well. + +Support high quality scaling with -s by using lanczos. + +Support spanning multiple monitors with region capture. This would also allow the user to record multiple monitors at the same time, the same way screen-direct works on nvidia x11. + +When webcam support is added also support v4l2loopback? this is done by using avdevice_register_all(); and -c v4l2 -o /dev/video0; but it needs to output raw data as well instead of h264 and possibly yuv420p. Maybe add a -k yuv420p option to do that or -k rgb. + This would be implemented by outputting the raw data directly into the output file, without using the video encoder. + +Do proper exit, to call gsr_capture_destroy which will properly stop gsr-kms-server. Otherwise there can be zombie gsr-kms-server on error. + +Replace all scissors with clearing textures if the cursor hits the outside of the frame image. + +Cursor position might be slightly wrong on rotated monitor. + +External texture doesn't work on nvidia x11, probably because of glx context (requires gles es). External texture is not used on nvidia x11 right now so it's not an issue. + +Add option to save replay buffer on disk instead of ram. + +nvfbc capture cursor with cursor.h instead and composite that on top. This allows us to also always get a cursor in direct capture mode. This could possible give better performance as well. + +Maybe remove external shader code and make a simple external to internal texture converter (compute shader), to reduce texture sampling. Maybe this is faster? + +Fix opengl context broken after suspend on nvidia by using this: https://registry.khronos.org/OpenGL/extensions/NV/NV_robustness_video_memory_purge.txt requires glx context creation flags and GetGraphicsResetStatusARB() == PURGED_CONTEXT_RESET_NV check to recreate all graphics. + +HDR looks incorrect, brightest point gets cut off. + +Make "screen" capture the preferred monitor. + +When webcam support is added add the option to add it as a second video track, to make it easier to edit in video editors. + +Fix constant framerate not working properly on amd/intel because capture framerate gets locked to the same framerate as + game framerate, which doesn't work well when you need to encode multiple duplicate frames (AMD/Intel is slow at encoding!). + It also appears to skip audio frames on nvidia wayland? why? that should be fine, but it causes video stuttering because of audio/video sync. + +Add option to pass a fd (from socketpair) to use for rpc. In the rpc have a common header, with protocol version, data type and data in an enum. + +Add the option to set audio track name, for example with -a "track-name:blabla|device:default_output|app:firefox" + +Maybe disable qp/vbr for replay. In that case we can preallocate all replay data (for both ram and disk) and write to that directly when receiving packet (dont do that when also recording at the same time). + That could improve performance/disk write optimization and maybe even reduce ram usage because of less blocks/fragmentation. + +When rpc is added add the option to add/remove audio devices/app audio and also overlays (from new capture sources). + +Support hdr screenshot. + +Recreate opengl context on loss. This can happen if there is a gpu driver bug, causing context to need to be recreated. This is a nice improvement to not break recording even with buggy driver. + +Support saving video with surround sound. Surround sound audio capture does work, but it gets downmixed to stereo. + +Add (render) plugin support. To simplify it (and possibly best performance) create one rgba texture (with the size of the output video) that is used across all plugins. + Create a framebuffer and set this texture and the target and set the framebuffer as active before calling the plugins. + Then the plugins can render simply by doing simple opengl draw functions. + Maybe send some metadata to the plugin, such as video (and framebuffer) size. Although this data can be retrieved from the active framebuffer. + +Either support webcam support with raw yuyv, mapping the buffer directly to opengl. Or use mjpeg, mapping the buffer directly to vaapi jpeg decoder and then get then map the decoded buffer to opengl. + Some webcams dont support raw yuyv and many webcams support higher framerates for mjpeg. + +Allow medium, high, very_high and ultra quality for -bm cbr. If that is used then it will automatically estimate the best bitrate for that quality based on resolution and fps. + Maybe do this in the ui instead (or both?), to show estimated file size. + +Maybe remove shader compute code. It doesn't seem necessary anymore now that glSwapBuffer/glFinish isn't used. dbus server isn't needed anymore either, the code can be moved back to the gpu screen recorder process. + +Add proper check if opengl functions are supported. dlsym for the symbol will return a no-op function if it's not supported, so it silently fails if used. + +Colors are offset to bottom left by 1 pixel or so on steam deck in landscape mode. + +When constant framerate is used (and for audio) multiple frames need to be encoded after resuming from suspend. + The clock jumps forward by around 2-3 seconds (on my machine). Is there a way to make sure the clock doesn't jump forward?
\ No newline at end of file |