sv_memlimit

If set, whenever a game ends, if the total memory used by the server is greater than this # of megabytes, the server will exit.

For this command to work you must first run sv_cheats true in the console. It only works on a local map or a private server — not in matchmaking. sv_cheats is a built-in Valve command and is completely safe from VAC.