In order to remove favicon completely without replacing it in magento, we need to do 3 steps:

1/ First step :
Remove all the favicon from ALL the folders

2/ Remove these codes in :  app/design/frontend/default/your_theme/template/page/html/head.phtml

<code><link rel="icon" href="<?php echo $this->getSkinUrl('favicon.ico') ?>" type="image/x-icon" />
<link rel="shortcut icon" href="<?php echo $this->getSkinUrl('favicon.ico') ?>" type="image/x-icon" /></code>


3/ The final step is : clear magento caches and session. Don’t forget to ctrl+f5 your browser to see the favicon is removed.

Hopefully that the tutorial will help anyone having the same problem.