A few users have emailed reports of ‘sluggish’ behavior. Some reports sound similar to this bug: Stuttering frames in version 1.5.3b5126, but others make it seem like a separate issue.
Some notes:
restarting the applicaiton generally clears up the problem for a little while
issue was present in earlier versions, but may have gotten worse in v1.5.3
seems like it may be related to the number of stills grabbed, or number of clips added to a reel, or something.
I’ll spend some time trying to reproduce this today.
Here are some verbatim quotes from reports so far:
As I have mentioned in the past I have been experiencing some sluggishness in cortex. For the most part my sessions start smoothly and everything works fine then at some point in the evening the sluggishness begins. Lately I have been noticing that seems to start when reels are created and clips begin to be added to the reels. I am not sure if this step in the process is what is causing the issue but I figured I’d mention it.
…
Basically cortex starts slowing down. If I hit play it takes a second to start to play, when it’s playing it freezes and skips frames in play, 2x, 3x speed. When I hit stop it takes a second to stop. If I was hitting cut to still it would not cut then it would cut after it caught up to key strokes I hit as I would hit cut more than once to see if worked then waited and it would catch up. I’ve tried reconnecting to panel as well as restarting cortex. Sometimes it helps but sometimes no change. Hope this helps.
…
Tonight it started before we started adding clips to reels. I tried the restart and it seemed ok for a bit but then went back to what I was experiencing before. skipping of frames and slow to respond to commands. Only thing different tonight was we had much less footage. 2:47 compared to over 5 hours last night.
I have issues with the Cortex not responding/lagging about an hr. into my dailies session. The only way to fix this issue is by shutting down the System. I go through this ritual about 4/5 times a day.
It started lagging about 3 weeks ago, it’s happened on older versions as well.
What is happening is that, when I hit play, it doesn’t play back, I adjust the color then it takes about 10 sec. to take effect, then it gets slower and slower, then it gets to a point where it just doesn’t do anything it freezes. Then I have to shut it down
The only thing I can think of is that there are too many things going at the same time, Rendering, Coloring, Syncing , My understanding is that is happening to other users as well, If I can think of anything else I will let you know.
I believe the other guys are having the same issue.
We are not doing anything differently, File Format, same as last year, Same Lut from last year. I use the same Lut every day, AlexaV3_K1S1_LogC2Video_Rec709_EE_lustre3d.lut.
I’ve spoken with all of the operators about this issue and it appears as if it is happening on all of the systems in multiple projects. After about 2-3 hours of working the system starts to get slugish.
The operator will reboot the system and things begin to function normally again.
One of the more experienced operators noticed that the more that the still-store is used the quicker the system gets to that sluggish point.
We have noticed some of the same behavior. I think there’s two contributing factors:
the type of media being using and the decode/debayer settings
there may be a memory leak when rendering stills.
Item 1):
Looking back at this post, when the debayer is using the MTI GPU Debayer on Sony RAW clips, the still files get to be about 100MB per still. So there’s time involved to read/write the stills and also it eats into memory.
Which leads me to Item 2):
I’ve started experiencing what I think is a memory leak when Cortex is generating stills (just noticed this morning). I don’t have all the details yet, but it appears that when you import a ton of footage and see the “Updating stills” message that RAM usage keeps going up, up, up. (At one point it filled all the RAM then crashed.) Looking at Task Manager, you can see the steady increase of RAM as Cortex was updating stills:
I’ll try to look into this further and make a separate post if this is the case.
So it could be that RAM is getting scare, causing sluggishness in Cortex. Any way for Cortex to check on available RAM and report it in the log?
Based on the input so far, it does sound possible that a memory leak could be causing the sort of behavior being described. But I know some of the reports have come from people working on Alexa ProRes shows, and I haven’t been able to reproduce any obvious leak just be importing clips or saving many stills or different color corrections, so I’m not sure.
@joeral and @antonio1971818 - next time you notice this behavior, can you open up take manager and look on the performance tab to see what the memory usage is? (Like in the above screenshot its showing that 12.8 GB are being used).
In my experience when I saw this memory leak, simply closing the job then reopening also cleared up memory. Right-click on the job tab and select ‘Close’. Then reopen it again.
But quitting and relaunching will definitely make sure everything is cleaned up and working properly.
We made some changes in v1.5.3 b5238 that we hope will improve this. However, we’re not sure we really found the root cause.
Please try the new version and let us know if things have improved. If the application still gets sluggish after long use, it’d also be worth trying what @24pdailies mentioned above:
It’d be helpful to know if that does help in this case, or if you need to restart the application to get things back to normal.
Just to let you know, I believe the core cause of this problem (from a user standpoint) is definitely the still store. Specifically, how often the user toggles between saved stills and live images. When I first started my season, I was a still adjusting to the learning curve of the new system (from Control Dailies to Cortex) and I toggled to my stills heavily, and yes the system would crash multiple times a night. As I got more confident with the system, I toggled less, and saw less reboots. I used to need 3 reboots a night, now I typically only need 1. A fellow colorist confirmed similar results. To clarify, this has nothing to do with how MANY stills you record, but simply how often you toggle to those images. The more you toggle, the more crashes and reboots. This is all just an educated guess, but I honestly believe this is the main culprit.
Thanks for the insights @wired525. This one has proven difficult to reproduce outside of a real production environment.
We are going to be testing a new version Wednesday night to see if any of the changes we’ve made so far in v1.5.4 improve this issue, but we’ll take a closer look at the impact of toggling stills. Hopefully we’ll get to the bottom of this one soon.
I’m getting a new version installed possibly today. I’ll deliberately try to ‘break’ it with heavy toggling the first night, and the following night I’ll try light toggling to see if there’s still a difference.
OK, you may also want to keep your eye on the performance tab of the task manager to see whether RAM usage climbs faster depending on what you’re doing…
I tried using the still store a lot compared to using it sparingly and found no difference at all anymore. So clearly that gremlin was killed. Hey, I tried.
