You can bypass the Deleted Items folder by holding Shift when you delete an Outlook item. Because you are permanently deleting the item(s), you'll receive a dialog asking if you are sure you want to permanently delete it, unless you disable the 'Warn before permanently deleting items' alert in Tools, Options, Other, Advanced Options.
Keep in mind that it can be quite easy to accidentally delete an entire folder and unless you use Exchange server, you won't be able to recover the deleted folder and contents.
Items deleted using this method are still added to Exchange Server's Deleted Item Recovery.
Published March 31, 2005. Last updated on April 22, 2016.