How bad is you when it come to verifying a cert? People use cert from CA because browser already trusted the root cert that being used to sign the cert. So browser can do the verification. Using self-sign cert, you have to do the verification yourself. If it's only you to access the site, it's not that difficult. You have the cert, so you can add it to your browser to be trusted.

But if Joe, Anna and Foo also need to access the site, you have to "securely" hand over the cert to them. Probably still not much a problem as you can go and meet them in person. But imagine if there's 100 more, or 1000 more people need to access your site and you don't even know them. Now you start seeing a problem with using self-signed cert.