В случае, если атмега не отвечает, поможет снижение тактовой частоты.
The error message you are seeing means that the chip is not responding to the «enable programming mode» command that the programmer is sending. Either you have got the connections wrong, or the programmer is sending the commands too quickly (programming frequency too high).
New AVR chips come configured to use the chip’s internal oscillator as a clock source, with the CLOCKDIV8 («divide clock frequency by 8») fuse set. This means that they run at 1MHz. The maximum programming frequency the chip supports is clock speed / 4, i.e. 250kHz. The USBASP defaults to 375kHz, which is too fast, so you need to slow it down.
You do this either by setting the «slow SCK» jumper of your USBASP, if it has one, or by using the avrdude -B option:
avrdude -c usbasp -p m328p -B 5 -U flash:w:bootloader.hex:i
The number after the -B option selects the programming speed (specifically, the clock period, the inverse of the frequency, so higher numbers mean slower frequencies). If -B 5 doesn’t work, try a higher number, like -B12 or even -B60.
Update: this line in the avrdude output suggests that your USBASP firmware version doesn’t support the -B option, so you will need to use the jumper:
avrdude: warning: cannot set sck period. please check for usbasp firmware update.