In order to remove favicon completely without replacing it in magento, we need to do 3 steps:

1/ First step :
Remove all the favicon from ALL the folders

2/ Remove these codes in :  app/design/frontend/default/your_theme/template/page/html/head.phtml


3/ The final step is : clear magento caches and session. Don’t forget to ctrl+f5 your browser to see the favicon is removed.

Hopefully that the tutorial will help anyone having the same problem.