To deactivate the history of recent files opened with VLC displayed in the OSX dock can be done with the following commands (execute in terminal):
defaults write org.videolan.vlc NSRecentDocumentsLimit 0 defaults delete org.videolan.vlc.LSSharedFileList RecentDocuments defaults write org.videolan.vlc.LSSharedFileList RecentDocuments -dict-add MaxAmount 0
Restart dock:
killall Dock