I'll start with the unhelpful half of my answer answer - practice 😅 As you do it you'll get better (on average) at estimating the time it will take. Sometimes there will be something you didn't consider or couldn't have predicted, but for the non-outliers you'll get better.

Something that is slightly more actionable is to break down larger tasks into small tasks, and estimate the time for those - then add them all up for the total estimate.

It can also be helpful to use time tracking and to track similar tasks over time so that you have a reference point for how long things actually take you. Warning: It can be really jarring when you first use time tracking and see how long some things that are "10 minute tickets" really take 😅

The first thing to know about estimations is that they are never accurate. Noone can pin you down on those.

Most people will tell you that you'll get better at it with practice, but that's only half the truth. Even when estimating things inside a project that you built yourself from scratch, your estimations will probably be very inaccurate most of the time, especially with larger tasks.

I find it helpful to cut large tasks down to small bite-sized packages, that I can confidently estimate in hours. Let's say, the client wants to implement a data tracking point.

• Identify the part of the code that triggers the data tracking, add pre-existing tracking function -> 1h
• Sample tracking data from user behavior -> 2h
• Create a payload and verify it can be read in your analytics tool -> 1h
• Refactoring and Code Cleanup -> 1h
• Testing -> 2h
• Documentation -> 2h

That amounts to 9h, which seem a lot considering such a small feature request, but all the little things start to add up.

I don't. It's mostly a total waste of time

I'm being completely serious here. I've been a professional developer for 26 years, and the most productive, engaged, happy teams I've worked on have been ones that are allowed to self organise and not estimate.

Making estimates consumes time that could be better spent simply getting on with the work. They also are the source of conflict and unnecessary pressure from management who treat them as 'written in stone'. They create friction where there need not be any.

Breaking down features ahead of time is mostly a futile exercise. Only at a very high level does any kind of task 'breakdown' make sense. The lower levels develop organically, thought processes change, new problems are found, new approaches are considered, new requirements are added. For these reasons, everything works better and smoother without estimates - which would just be a meaningless overhead tacked onto a very rough 'broad strokes' plan.

Further interesting reading - Software Estimation Considered Harmful

'Estimation is ultimately a futile effort. Software, more or less, is like writing poetry. Or solving mathematical proofs. It takes as long as it takes, and it’s done when it’s done. Perfect, or imperfect, estimation won’t change how long it takes to complete the work. If that sounds horrible to you, then go do something else.'

When I estimate, I perform a breakdown into smaller tasks. Then I do three point estimates: best case, expected case and worst case. Another poster suggested using ranges which is similar.

Looking at the variation/range between best/worst gives you an idea of how "accurate"/risky your estimate is. For example, I estimate a task at 5, 2, 1 hours vs 16, 8, 7 hours. The second task has more risk (almost an additional day) compared to the first (1/2 day). Combining the three point estimates is a process on it's own (e.g. Monte Carlo or triangle distribution with three point estimate) but by producing a range rather than a single point you:

1 can present a better picture of the risk of the estimate
2 are forced to re-estimate a task under different conditions

I had somebody mention that planning poker (an agile method) also estimates the task under different conditions (different people with different viewpoints/experiences). Planning poker ultimately converges on a single number rather than a range (which I think is an issue), but it does seem to help counteract anchoring bias.

Doing estimates well takes discipline, analysis and experience.
When I have done post-mortems on my own and other's estimations, I have found a bias I didn't account for or an evaluation dimension (e.g. alignment between documentation for a library and what we needed the library to do). Addressing the issue by adding it to an estimation checklist improved future estimates.

An interesting book on it is "How to Measure Anything: Finding the Value of “Intangibles” in Business." Hubbard and Seiersen. The chapter on calibrated estimators includes some exercises as well as mechanisms to prevent biases in estimating. [[hubbardresearch.com/publications/h...]

Just wanna chime in here to put the spotlight on this very relevant post by @arielbk:

This is a much more realistic way to handle it

Regarding SCRUM or similar approaches, it looks like there is nothing complicated about this. It is just estimating and trying to get better at it. For SCRUM, people would try to explain, why the picked a certain "value" (the estimation), if it is much different from the other estimated values, but this is totally fine, because it not only is for finding a value, which people finally agree on, but also helps the team to improve on the estimations for future.
Nothing to worry about, if this is also your case.

Imagine doing the thing and take the time to record how much each thing might take given your previous experience

Then multiply that by 3 to handle edges cases that you didn't think of

It's very difficult to base estimations on time, so I usually estimate in points. What's is important when measuring in points is to consider how many points you have implemented in previous sprints when you calculate how many points you will have in the next sprint. If you skip that your points will be bound to time.

You can read more about it here

After 10+ years of experience, I always do the same routine.

Specifications

First of all, I do one (or several depending on the complexity of the project) meeting with the client, to gather all his/her needs and writing down specifications to translate them into technical stuff.

Usually it get it's own invoice : if the project is too expensive and refused I at least I get paid for this, and the client has the document to give to other devs.

Time split

Once it's validated from both parts, I split it as much as possible to get single tasks, easy to estimate, in hours. Then do a big old SUM to get total hours.

Do a division by 7 (hours per day) and you'll get the total days.

Depending on the length of the project I usually add 25% (short project) to 50% (long project) to this, to account for client management (mails, phone calls, visio) and unpredicted stuff (documentation writing, bugs, slowdowns, learn and tests, blackouts, illness, etc.)

Do a division by 5 (days per open weeks) and you'll get the planning estimation.

And voila.

Concrete example with random numbers

A client want to get a display website. Simple enough.

I go for a 1 day invoice, including half a day of meeting, 2 hours writing specs, 2 hours validations.

Website is to create from scratch (no texts nor images to re-use from a previous website), I estimate :

• Server stuff/hosting : 1 day
• Set a WordPress & configuration : 1 day
• Add content : 3 days
• Custom style and developpement : 5 days.

Total days of effective developpements : 10 days.

Rather short project, I add 3 days for project & client management : 13 days in total.

How much time will it take (planning) ? 13 days / 5 open days a week ~= 2 weeks and half, rounded to 3 weeks (sometimes clients don't respond quick enough).

So in resumé 13 days (x daily price) invoice, website up in 3 weeks.

Too much time ?

If the client is pretty reactive or if you get to dev really fast and a couple of days are left at the end, do some SEO, or spend some time explaining how works the back office, etc.

Conclusion

It isn't an exact science and it will always be pretty hard to estimate project, and the more projects you'll made, the more you'll be able to fine tune ;)

I also recommand to get a time management software, or a simplier excel sheet :

• 1 sheet for estimation
• 1 sheet for "time spent".

It requires some discipline but this will allow you to really note how time you spend on what, and can also help justify costs to your client

Estimate

Time spent

It comes with experience. You just remember something similar that you worked on before and the code is not too different. So you can estimate how long it is going to take to build it. With practice it becomes easier although its hard to be 100% accurate because you cant predict what bugs or issues are going to arise.