There are two ways to delete an invoice:
On the Invoices list view
Select the invoices you wish to delete using the checkboxes next to the invoices.
On the individual edit screen
You can also delete them by going into the individual edit invoice screen and select ‘Delete’ from the Status dropdown:
Then click Update.
The difference between the above deletion method and the Delete Permanently link is that you can reset the status of an invoice deleted in the the former manner, while Delete Permanently removes it forever.