When you delete an Outlook item — email message, contact, task, appointment, and so on–Outlook moves the item to the Deleted Items folder. You control whether Outlook then empties the Deleted Items folder when you close Outlook. That’s probably adequate for most of you, but you can override the settings for individual items. To do so, press [Shift]+Delete instead of just Delete. Doing so permanently deletes the selected item instead of moving it to the Deleted Items folder.
Microsoft calls this trick a hard delete. As always, use caution when using this shortcut, as you might not be able to recover the permanently deleted item.
Might not?
Yes, might not. Permanently deleted doesn’t mean completely wiped off the face of the earth. It means the item is permanently deleted from the current window. You can often find other copies, if you know where to look:

• There might be a copy of the permanently deleted item in your backup file.
• A copy of a permanently deleted email might still exist in the Sent Items folder, as an original item or as part of a continued conversation.
• If you sent or forwarded the message, one of more of the recipients might have a copy on their system.

The idea that you can permanently delete an Outlook item is iffier than you might think. For most of us, it doesn’t really matter, but if you permanently delete an item you probably can find a copy somewhere; whether that’s good or bad depends on your circumstances.
By the way, to configure the way Outlook empties the Deleted Items folder, choose Options from the Tools menu, click the Other tab, and check or uncheck the Empty Deleted Items Folder Upon Exiting option, accordingly.