This question conflates two steps, which is why it is hard to answer in a sentence. The first step is whether the work is notifiable at all: that is decided by the work, not by the neighbour. The second is whether an agreement follows, and that is decided by the reply. So the honest answer is that if the work is notifiable you always need to serve, and you need an award only if the neighbour dissents or stays silent. Here is how to settle each half.
Is the work notifiable?
Three questions. Are you altering a structure you share with a neighbour: cutting in, raising it, underpinning it, taking out a chimney breast that forms part of it? Are you building a new wall on or at the boundary where nothing stands now? Are you excavating near the neighbour's building and going deeper than its foundations, within three metres, or within six metres on the forty-five degree line? A yes to any of them makes the work notifiable, and a loft conversion with a rear extension is routinely a yes to two.
Then it is the neighbour's reply that decides the rest
Serve the right notice on every adjoining owner, with the right period. If they consent in writing within a fortnight, you are done, and no award is needed. If they dissent, or if they simply do not reply, the Act deems a dispute to have arisen and surveyors are appointed. That is the whole test, and it means the answer to when do you need party wall agreement is: whenever the fortnight passes without a written consent.
What happens if you skip it
You lose the rights the Act gives you and the protection that comes with them. Without a schedule of condition you have no record of the neighbour's property before the work, so a crack that was there for twenty years becomes an argument you cannot win. Without an award you have no authority to do work to the shared structure, and the neighbour's remedy moves from a surveyor to a court. Neither is a good trade for the two months it would have taken.
Questions people ask about do i need a party wall agreement
Does a loft conversion always need one?
Almost always a notice, because the new floor steels usually bear into the shared wall, and often a chimney breast comes out as well. Whether an agreement follows still depends on the reply.
What about replacing a fence?
A fence is not a wall, and replacing one is not notifiable under this Act. A masonry garden wall astride the boundary is a party fence wall and is a different matter.
Does my neighbour's permission count if it is verbal?
Consent has to be in writing to end the process. A verbal yes leaves you relying on a fortnight passing with nothing served, which is exactly the deemed dispute you were trying to avoid.