Well since a while this dosent work, i start and after 5-7 sec it says out of memory, and stop recording. Also there is some problem to stop recording on my own .
It's a known issue - I don't recall whether there has been a fix for it yet (It happens on completely overpowered systems too) Your best bet is probably to use a different external tool to record your gameplay for now.