ResolutionMostly, the problem is not related to the amount of available memory in the system. By default, FreeBSD 4.x kernel is compiled with 'datasize' 512MB . See:
# limits -d
Resource limits (current):
datasize 524288 kb
You cannot increase this value without the kernel recompilation. It means that, by default, a process cannot allocate more than 512 MB.
The possible solutions are:
1. Recompile the kernel with a larger datasize.
2. Dump not the whole server, but by paths. For example per 100 domains.
If the server is running FreeBSD 5.x you may change kern.maxdsiz value in the /boot/loader.conf: