Open MPI logo

Open MPI User's Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Open MPI User's mailing list

Subject: Re: [OMPI users] openib_reg_mr
From: Ralph Castain (rhc_at_[hidden])
Date: 2012-06-15 11:32:49


Hi Marcus

Sounds like you might be running out of IB resources as opposed to main memory - not much we can suggest there other than trying to set queue sizes, which is a complicated option. You might look at "ompi_info --param btl openib" and see if adjusting some of those helps.

Ralph

On Jun 15, 2012, at 9:26 AM, Daniels, Marcus G wrote:

>
> On Jun 15, 2012, at 8:02 AM, Jeff Squyres wrote:
>
>> Were there any clues in /var/log/messages or dmesg?
>>
>
> Thanks. I found a suggestion from Nathan Hjelm to add "options mlx4_core log_mtts_per_seg=X" (where X is 5 in my case).
> Offline suggestions (which also included that) were also add "--mca mpi_leave_pinned 0" to the mpirun line and to double check my locked memory limits.
>
> The only thing I find works reliably is to use "-npernode 32" instead of "-npernode 48". Unfortunately my system has 48 processor node.
> I've got lots of headroom on real memory.
>
> Marcus
>
>
>
> _______________________________________________
> users mailing list
> users_at_[hidden]
> http://www.open-mpi.org/mailman/listinfo.cgi/users