I have 92 x 2160p clips at 400mbps, which have also been transcoded to DNxHR_HQX at 728mbps in 3 batches. The transcodes need to be re-imported in one batch for editing. Importing them into the Appimage uses 55.71Gb of RAM and it fills the RAM (64Gb) when trying to generate proxies, freezing the system, necessitating manually switching off and rebooting. Importing the same clips into the last Arch single user release (20201031) uses 28.03Gb, which rises to 32.54Gb towards the end of generating proxies. Importing the raw (not RAW!) camera files into the Appimage takes longer to freeze, but it still happens.
The import strategy is replace current project and concatenate tracks.
There is no point in importing the clips into the Appimage in batches for editing because, obviously, they will eventually need to be imported all at once for final assembly and rendering. Also, shouldnt the Appimage handle them the same as the single user version?
I have been happily using the Appimage to edit smaller sections of the project til now.
Good to know that up until now, AppImage has been working for you. I will run a test here to compare memory usage, which I would think should be comparable to what is used with a tar file, to see what my computer shows. Possibilities include:
A non-authoritative person on the internet says that the disadvantages are that your application executables are larger and take much more memory but I do not see how that can be true in this case.
The computer where the AppImage for newer distros was created has 32GB so I wonder if that is relevant. Do you have more or less than that? Again, that should not be relevant.
Other possibilities? I will see what I can find out. Maybe MatN has more ideas.
Since you are using Arch also and do your own builds too, could you watch the output of top from a terminal window while playing a video - once on a built system and once using AppImage? I am not sure what is going on.
Importing 4 ProRes files at 1440p for a total of 6.5GB (with your same Insertion Strategy) I also notice a slight difference in RAM consumption.
Using TOP:
I use Arch Linux; Ryzen 3700 (8/16) and 32GB of RAM; driver X11-OpenGL. The smaller values may depend on the 4 Prores being less extreme than your files. Also, the non-std resolution of your media may affect how it is handled by CinGG.
One thing I cant judge is that I have to restart every CinGG test because otherwise the RAM consumption always increases as you use CinGG and never goes back, even if you go back to idle (the test I did with result 3.2% became 4.6% in a second consecutive test).
PS: How much do you set the cache size in CinGG? Im doing various tests but I cant figure out which value is the best (now Im trying 4096, but I dont notice any improvement)!
Hi Andrea, its good to know you find the Appimage uses more RAM, the difference being not so far away from my result, in your case 72% and mine 84%.
Im not sure why you say my resolution is non-standard, I am importing 92 3840x2160p clips, maybe its the way I wrote it? Sorry for any confusion!
Interesting observation about Cin not releasing RAM, I have noticed this effect before when in the middle of a project, but then Im not prepared to try and find out why, I just want to get the project finished, then usually forget about it. I had not realised it is a regular thing. Needs fixing perhaps?
I just leave cache size at default (256).
I was using the 20210228 Appimage, I have downloaded the other two newer distro ones and found the unnumbered Newer Distro one to do the same, but the 20201031 Appimage acts like the single user release I tested against it.
I have done some simple testing with various CinGG releases. Determining the actual memory usage under Linux is difficult, I settled on ps_mem.py. Following are the test results on Mint 19.3 XFCE on AMD Ryzen 5 2400G, 32G ram.
I tested just after the program loaded, and then with a 300 M 4K file loaded which plays fine, LG Dolby Trailer 4K Demo.ts, I might have gotten that from Youtube.
Interesting observation about Cin not releasing RAM, I have noticed this effect before when in the middle of a project, but then Im not prepared to try and find out why, I just want to get the project finished, then usually forget about it. I had not realised it is a regular thing. Needs fixing perhaps?
Cinelerra is supposed to accumulate RAM as it runs in order to run faster by keeping things in memory for potential future use instead of having to pull it up again from disk. It releases memory upon exiting routines and hopefully the Operating System handles some of this. However, third party libraries used by Cinelerra may not be as diligent in releasing RAM.
Running Valgrind, which Andrea has done in the pass and sent to GG who was able to find any memory leaks, shows if memory is getting lost. For example at the end there is the following from an old one of Andreas dated Oct. 28, 2018:
==2100== LEAK SUMMARY:
==2100== definitely lost: 8,184 bytes in 46 blocks
==2100== indirectly lost: 626,096 bytes in 7 blocks
==2100== possibly lost: 0 bytes in 0 blocks
==2100== still reachable: 323,861 bytes in 5,044 blocks
==2100== suppressed: 0 bytes in 0 blocks
In the above case, when GG analyzed where the leaks came from, they turned out to be from somewhere other than Cinelerra so there was no way to fix them. But now changes made since Oct. 31, 2020 could have introduced new leaks in Cinelerra itself. Unfortunately, I do not know how to fix the code to resolve any of these but it would be worth looking at to see if there is any large new leak (maybe Andrea will have time to run Valgrind on a current build).
After reviewing all of the feedback, it looks like the extra memory usage is not from the tar/build versus AppImage BUT rather the difference between Oct. 31, 2020 and Feb. 28, 2021 (or in the case MatN tested after 20201131).
So based on the feedback, I tested again to confirm and can definitively report that the new version uses more memory on the same playback video. I will need to narrow down which modification has increased the need for more memory. Also, if Andrea has time to run Valgrind and send me the log, I can at least look at the statistics on the leakage to see if it looks like there is a problem.
I dont have that DNxHR_HQX file format. Could you tell me how to transcode into that format? Maybe the large memory increase is because of this specific format.
Do you have swapfile enabled? I find it strange the system freezes, I only had that when my disk was completely full. If you have a swapfile or partiton, and it is not full, does it grow when you do this large video in the AppImage?
Yes, in the render box File format FFMPEG, select mov, click the video spanner icon, in the Compression window open the drop down list and select dnxhr_hq.mov. It can also be found deep in the qt menu. I doubt very much it is the format because a) it happens whether I import DNxHR_HQX, DNxHR444 which I tried first, or raw camera files which are 10-bit h265 .mov at 400mbps and b) it doesnt happen if I import any of those into 20201031.
Normally no, but I temporarily set up a 64Gb swap partition and tried again. It filled the RAM and swap, froze and crashed Cin. Thats (nominally) 128Gb of memory!
Thank you for running valgrind. I never have the patience to do so. There are a lot of lost bytes and this is possibly the cause of the extra memory being used now. I will have to see if I can understand which mod may have caused a problem but it will be a difficult learning curve to get there. You can see millions of bytes lost and that is bad:
==215823== LEAK SUMMARY:
==215823== definitely lost: 169,064 bytes in 1,432 blocks
==215823== indirectly lost: 5,223,805 bytes in 3,188 blocks
==215823== possibly lost: 7,988,859 bytes in 23,395 blocks
==215823== still reachable: 973,283 bytes in 13,564 blocks
I did some more structured testing, because I wondered if settings might have
an effect. And indeed they do. Same testing environment as before, Mint XFCE 19.3,
AMD Ryzen 5 2400G, 32G ram, movie LG Dolby Trailer 4K Demo 300MB, memory size
total as reported by ps_mem.py.
Build source / Exe Disk Mem GB Mem GB Mem GB
system git date form MB just after play after play
started no hwacell vaapi
CinGG 2020-10 cin 227 0.091 2.0 1.1
distro static
Mint 20210308 cin 188 0.091 2.9 1.3
Mint 20210308 AppImage 82 0.092 3.0 1.3
CinGG 2021-02 AppImage 80 0.090 2.9 n.a.
older
On the last entry, the older AppImage has an incompatible vaapi lib,
so hwaccel gives errors in the starting terminal when enabled. The normal
AppImage does not run on Mint 19.2, but runs on Mint 20.1 . If I actually build the latest source on Mint, it works OK.
Basically, there is not much difference between the versions in memory
use. The only thing really making a difference is enabling the vaapi
hwaccel. I have no Nvidia, so did not test other hwaccels.
Thanks for the DNxHR_HQX tip, Ill try that next, on Manjaro.
I found there are considerable differences in memory usage if the hwaccel settings match the movie (see my other post). What are your hwaccel settings on the AppImage, and if you start it from a terminal, do you see any errors there while importing?
** Posted by: @matn **What are your hwaccel settings on the AppImage, and if you start it from a terminal, do you see any errors there while importing?
I have no idea what hwaccel settings are, or how to find them. I never use the terminal unless I absolutely have to and have no desire to learn how. Yes, there is at least one Linux user on Earth that only wants to use it!
I understand your reluctance to use the terminal. It would be nice if CinGG had some built-in way to display system hardware relevant to video editing. How do you determine memory usage?
Regarding hwaccel, I was referring to the Use HW device in the Settings-> Preferences->Perfomance tab. That makes a big difference in memory usage in my testing, at least for a H.265 coded movie where a hardware decoder is supported via vaapi.
Also, what are your Settings->Format? DNxHR_HQX is a 12-bit format, right? So you probably use RGBA float?
I have tried doing a DNxHR render with the 20201031 release, but the format is not supported. If I look in the ffmpeg/video folder of CinGG, there is no format file for any dnx. So that version cannot render dnxhr. But you are able to import and edit it?
Did you render the DNxHR format with the 2021-02 version?
