Open MPI logo

Open MPI Development Mailing List Archives

  |   Home   |   Support   |   FAQ   |   all Development mailing list

Subject: [OMPI devel] Flush CQ error on iWARP/Out-of-sync shutdown
From: Jon Mason (jon_at_[hidden])
Date: 2008-05-05 16:13:04


I am seeing some unusual behavior during the shutdown phase of ompi at the end of my testcase. While running a IMB pingpong test over the rdmacm on openib, I get cq flush errors on my iWARP adapters.

This error is happening because the remote node is still polling the endpoint while the other one shutdown. This occurs because iWARP puts the qps in error state when the channel is disconnected (IB does not do this). Since the cq is still being polled when the event is received on the remote node, ompi thinks it hit an error and kills the run. Since this is expected behavior on iWARP, this is not really an error case.

There is a larger question regarding why the remote node is still polling the hca and not shutting down, but my immediate question is if it is an acceptable fix to simply disregard this "error" if it is an iWARP adapter.

Opinions?

Thanks,
Jon

The patch would look something like this:

===================================================================
--- ompi/mca/btl/openib/btl_openib_component.c (revision 18362)
+++ ompi/mca/btl/openib/btl_openib_component.c (working copy)
@@ -2062,6 +2062,11 @@
     if(endpoint && endpoint->endpoint_proc && endpoint->endpoint_proc->proc_ompi)
         remote_proc = endpoint->endpoint_proc->proc_ompi;

+ if (wc->status == IBV_WC_WR_FLUSH_ERR &&
+ IBV_TRANSPORT_IWARP == hca->ib_dev->transport_type) {
+ return;
+ }
+
     if(wc->status != IBV_WC_WR_FLUSH_ERR || !flush_err_printed[cq]++) {
         BTL_PEER_ERROR(remote_proc, ("error polling %s with status %s "
                     "status number %d for wr_id %llu opcode %d qp_idx %d",