As contradictory as that might sound, it's true for the normal way of deleting files. The only way to truly delete a file is to use a "shredder" program that overwrites the file.
This has two important consequences: you can recover deleted files, and believing your deleted files are actually deleted can pose a security risk.
How Your Computer Stores and Deletes Files
Your computer divides up storage devices like hard drives and Solid State Drives (SSDs) into Logical Block Addresses (LBAs). Somewhere on your storage device there is a table that says file 1 is at LBA 5, file 2 is at LBA 20, etc. 
Let's say you delete file 2 (which is at LBAs 20 to 35). The data at LBAs 20-35 isn't overwritten. Instead, your computer deletes the entry in the table that specifies file 2 is at LBA 20. Now LBAs 20 to 35 are free for other files to be written to.
This is done for several reasons, but the primary reason is performance: if you were to actually overwite a file when you deleted it, then it would have to write out the file size's worth of data. Instead, we only have to change a handful of bytes in the file table.
You Can Recover Deleted Files
Because file deletion only deletes the entry in the file table, the file is still available for recovery. If you find yourself in this situation, either unplug the storage device immediately if possible, or shut down your computer. The longer the storage device is on and available for writing, the higher the chance that something will overwrite the file (or part of the file). Then find yourself a file recovery tool (there's plenty that show up via Google search).
If your storage device is removable, such as a USB flash drive, ideally you want to mount the drive as read only to prevent any possible overwriting of deleted files. If your storage device isn't removable (e.g. it is the same storage device that has your operating system), then ideally you would use a data recovery tool that is bootable - you load the software onto a USB flash drive and boot your computer from that. Once booted, the data recovery tool would ideally mount your storage device as read only.
All that being said, some recovery tools don't mount storage as read only for you. The more free space you have, and the sooner you try to recover the file, the lower the chances of the data being overwritten.
In reality, you should have a good backup system so you can recover deleted files, but nobody's perfect.
Deleted Files Can Pose a Security Risk
If you delete sensitive files and believe they truly are deleted, when in fact they're not, then this is a security issue. In reality, I'm of the opinion that if files are sensitive enough for you to be concerned that someone could recover the deleted files, then they should be encrypted to begin with.
Regardless, it is good security practice to wipe your storage device before you sell it, donate it, or otherwise get rid of it. The only exception would be is if you are physically destroying it.
Wiping Data on a Conventional Hard Drive
Any typical "shredder" program that you can find through Google can handle wiping both individual files and the whole hard drive.
They typically let you specify the number of passes to make. In theory, even if you overwrite the data once, there are lingering magnetic fields of the old data that might be readable for data recovery. I've seen some sources say this risk is overblown. I'm not sure what to make of it.
Wiping Data on a SSD
Solid state drives need special consideration because of the way they are designed. For a conventional hard drive, every time the computer tells it to write to LBA 20, the hard drive will always write to the same physical sector on the drive. Therefore, to overwrite a file stored at LBA 20, writing to LBA 20 is sufficient.
However, with a SSD, this is not true. Consecutive writes to the same LBA are not guaranteed to write to the same physical flash cell on the SSD (and in general, they will not) (your computer can only specify writes via LBA, not physical flash cell numbers). This is because the flash cells in SSDs can only be written to a finite number of times before they become unwriteable. To prevent this from happening, when the SSD is asked to write data it, picks an available flash cell with low wear to write to, and writes this mapping of LBAs to physical flash cells to a table stored on the SSD.
Therefore a conventional shredder program will not work with SSDs.
In the SATA standard there is a special secure erase command that can be issued to SSDs to securely wipe them. However, this will wipe the entire drive. You can't select individual files.
To the best of my knowledge there is no program available for any SSD to securely erase individual files.
In general, files are not stored contiguously on disk. Files are broken up into blocks, and its these blocks that are stored at various LBAs. However, for the purposes of this discussion, this detail doesn't materially change the conclusions ↩︎