Commit Graph
35 Commits
Author SHA1 Message Date
Michael Brown 410566cef7 [hermon] Minimise reset time
Check for reset completion by waiting for the device to respond to PCI
configuration cycles, as documented in the Programmer's Reference
Manual.  On the original ConnectX HCA, this reduces the time spent on
reset from 1000ms down to 1ms.

Signed-off-by: Michael Brown <mcb30@ipxe.org>
2021-02-01 22:29:30 +00:00
Christian Iversen 8b07c88df8 [hermon] Add support for port management event
Inspired by Flexboot, the function hermon_event_port_mgmnt_change() is
added to handle the HERMON_EV_PORT_MGMNT_CHANGE event type, which
updates the Infiniband subsystem.

Signed-off-by: Christian Iversen <ci@iversenit.dk>
Modified-by: Michael Brown <mcb30@ipxe.org>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
2021-02-01 11:44:54 +00:00
Michael Brown e62c3e3513 [hermon] Use reset value suitable for ConnectX-3
The programming documentation states that the reset magic value is
"0x00000001 (Big Endian)", and the current code matches this by using
the value 0x01000000 for the implicitly little-endian writel().

Inspection of the FlexBoot source code reveals an exciting variety of
reset values, some suggestive of confusion around endianness.

Experimentation suggests that the value 0x01000001 works reliably
across a wide range of hardware.

Debugged-by: Christian Iversen <ci@iversenit.dk>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
2021-02-01 01:53:15 +00:00
Michael Brown def46cf344 [hermon] Limit link poll frequency in DOWN state
Some older versions of the hardware (and/or firmware) do not report an
event when an Infiniband link reaches the INIT state.  The driver
works around this missing event by calling ib_smc_update() on each
event queue poll while the link is in the DOWN state.

Commit 6cb12ee ("[hermon] Increase polling rate for command
completions") addressed this by speeding up the time taken to issue
each command invoked by ib_smc_update().  Experimentation shows that
the impact is still significant: for example, in a situation where an
unplugged port is opened, the throughput on the other port can be
reduced by over 99%.

Fix by throttling the rate at which link polling is attempted.

Debugged-by: Christian Iversen <ci@iversenit.dk>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
2021-01-31 23:29:45 +00:00
Christian Iversen 43d72d0087 [hermon] Perform clean MPT unmap on device shutdown
This change is ported from Flexboot sources.  When stopping a Hermon
device, perform hermon_unmap_mpt() which runs HERMON_HCR_HW2SW_MPT to
bring the Memory Protection Table (MPT) back to software control.

Signed-off-by: Christian Iversen <ci@iversenit.dk>
Modified-by: Michael Brown <mcb30@ipxe.org>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
2021-01-29 00:46:53 +00:00
Christian Iversen 699b9f1d1b [hermon] Use Ethernet MAC as eIPoIB local EMAC
The eIPoIB local Ethernet MAC is currently constructed from the port
GUID.  Given a base GUID/MAC value of N, Mellanox seems to populate:

  Node GUID:   N + 0
  Port 1 GUID: N + 1
  Port 2 GUID: N + 2

and

  Port 1 MAC:  N + 0
  Port 2 MAC:  N + 1

This causes a duplicate local MAC address when port 1 is configured as
Infiniband and port 2 as Ethernet, since both will derive their MAC
address as (N + 1).

Fix by using the port's Ethernet MAC as the eIPoIB local EMAC.  This
is a behavioural change that could potentially break configurations
that rely on the local EMAC value, such as a DHCP server relying on
the chaddr field for DHCP reservations.

Signed-off-by: Christian Iversen <ci@iversenit.dk>
Modified-by: Michael Brown <mcb30@ipxe.org>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
2021-01-29 00:13:46 +00:00
Christian Iversen 7c40227e18 [hermon] Increase command timeout from 2 to 10 seconds
Some commands (particularly in relation to device initialization) can
occasionally take longer than 2 seconds, and the Mellanox documentation
recommends a 10 second timeout.

Signed-off-by: Christian Iversen <ci@iversenit.dk>
2021-01-28 20:55:14 +00:00
Michael Brown e84c917f39 [hermon] Allocate space for GRH on UD queue pairs
The Infiniband specification (volume 1, section 11.4.1.2 "Post Receive
Request") notes that for UD QPs, the GRH will be placed in the first
40 bytes of the receive buffer if present.  (If no GRH is present,
which is normal, then the first 40 bytes of the receive buffer will be
unused.)

Mellanox hardware performs this placement automatically: other headers
will be stripped (and their values returned via the CQE), but the
first 40 bytes of the data buffer will be consumed by the (probably
non-existent) GRH.

This does not fit neatly into iPXE's internal abstraction, which
expects the data buffer to represent just the data payload with the
addresses from the GRH (if present) passed as additional parameters to
ib_complete_recv().

The end result of this discrepancy is that attempts to receive
full-sized 2048-byte IPoIB packets on Mellanox hardware will fail.

Fix by allocating a separate ring buffer to hold the received GRHs.

Signed-off-by: Michael Brown <mcb30@ipxe.org>
2016-03-21 08:18:15 +00:00
Michael Brown c4ee23e7c6 [hermon] Use PCI VPD for non-volatile option storage
Signed-off-by: Michael Brown <mcb30@ipxe.org>
2012-10-25 08:24:00 -07:00
Michael Brown 6dd4ac77e5 [hermon] Ensure hardware is quiescent when no interfaces are open
WinPE has been observed to call PXENV_UNDI_SHUTDOWN but not
PXENV_STOP_UNDI.  This means that Hermon hardware is left partially
active (firmware running and one event queue mapped) when WinPE starts
up, which can cause a Blue Screen of Death.

Fix by ensuring that the hardware is left quiescent (with the firmware
stopped) when no interfaces are open.

Reported-by: Itay Gazit <itayg@mellanox.co.il>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
2011-11-15 04:12:07 +00:00
Michael Brown e809985ca9 [hermon] Add BOFM support
Signed-off-by: Michael Brown <mcb30@ipxe.org>
2011-03-03 02:40:12 +00:00
Itay Gazit 0e878b30fe [hermon] Force link speed to SDR
SDR link comes up much faster than other speeds.

Signed-off-by: Itay Gazit <itaygazit@gmail.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
2011-02-17 01:25:10 +00:00
Itay Gazit 1b84a80442 [hermon] Use circular event queue
Avoid memory leak of untreated events by having circular event queue
operation.

Signed-off-by: Itay Gazit <itaygazit@gmail.com>
Modified-by: Michael Brown <mcb30@ipxe.org>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
2011-02-17 01:25:09 +00:00
Michael Brown dc462e8b3b [hermon] Fix incorrectly-padded sense_port structure
Signed-off-by: Michael Brown <mcb30@ipxe.org>
2010-12-24 01:10:26 +00:00
Michael Brown 5a981cff8e [hermon] Add support for dual-protocol devices
Originally-implemented-by: Itay Gazit <itaygazit@gmail.com>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
2010-09-21 02:22:01 +01:00
Itay Gazit ed0ea7cfc2 [hermon] Add infrastructure for Ethernet devices
Add PRM structures to support Hermon Ethernet devices.

Signed-off-by: Itay Gazit <itaygazit@gmail.com>
Modified-by: Michael Brown <mcb30@ipxe.org>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
2010-09-21 02:19:13 +01:00
Michael Brown f62c433d66 [hermon] Use SET_PORT to set port parameters
Unlike Arbel, port parameters must be applied via a separate call to
SET_PORT, rather than as parameters to INIT_PORT.

Signed-off-by: Michael Brown <mcb30@ipxe.org>
2010-09-21 02:15:24 +01:00
Michael Brown 0d6b1d98fa [hermon] Tidy up ICM allocations
Signed-off-by: Michael Brown <mcb30@ipxe.org>
2010-09-19 00:04:26 +01:00
Michael Brown e4ed060983 [hermon] Map all event types to our event queue
Only port state change events are currently mapped to our event queue,
since those are the only events we are prepared to handle.  This
ignores a potentially useful source of diagnostic information in the
case of unexpected failures.

Fix by mapping all events to the event queue; a build with debugging
enabled will therefore at least dump the raw content of the unexpected
events.

Signed-off-by: Michael Brown <mcb30@ipxe.org>
2010-09-16 22:06:20 +01:00
Michael Brown 46c7f99c66 [hermon] Use correct alignment for doorbell records
Doorbell records are currently embedded within the completion queue
and receive work queue strucures, which are allocated using zalloc()
and so have an alignment guarantee of only sizeof(void*), i.e. four
bytes.  This is sufficient for the receive work queue, but not for the
completion queue, which requires an alignment guarantee of eight
bytes.

Though not guaranteed, it so happens that zalloc() will always return
a pointer that is exactly four bytes above a sixteen-byte boundary.
The completion queue doorbell record is therefore always misaligned,
and the value passed to the hardware via SW2HW_CQ is actually always
pointing to the page_offset value within the MTT descriptor (which
directly precedes the inline doorbell record).  Provided that the page
offset is greater than 0x100, this looks to the hardware like an
update_ci value of greater than 0x010000 (taking into account
endianness differences), and so the hardware will happily deliver more
than 0x010000 completions before stopping.  Hence this problem is
rarely observable.

Fix by allocating the doorbell records separately and using the
correct alignment constraints.

Signed-off-by: Michael Brown <mcb30@ipxe.org>
2010-09-12 22:46:01 +01:00
Michael Brown 42f451e070 [hermon] Improve debugging output and facilities
Improve the utility of debugging messages by including the relevant
port number, queue number (QPN, CQN, EQN), work queue entry (WQE)
number, and physical addresses wherever applicable.

Add hermon_dump_cqctx() for dumping a completion queue context, and
hermon_fill_nop_send_wqe() for inserting NOPs into send work queues.

Signed-off-by: Michael Brown <mcb30@ipxe.org>
2010-09-12 22:35:13 +01:00
Michael Brown 8406115834 [build] Rename gPXE to iPXE
Access to the gpxe.org and etherboot.org domains and associated
resources has been revoked by the registrant of the domain.  Work
around this problem by renaming project from gPXE to iPXE, and
updating URLs to match.

Also update README, LOG and COPYRIGHTS to remove obsolete information.

Signed-off-by: Michael Brown <mcb30@ipxe.org>
2010-04-19 23:43:39 +01:00
Michael Brown 9f7141a1ce [hermon] Reset device during probe()
Some systems will retry their boot sequence in the event of a boot
failure.  On these systems, the second and subsequent boot attempts
will fail to initialise the Hermon HCA.

Fix by resetting the HCA during probe().  This incurs a one-second
cost, but there seems to be no viable alternative.

Originally-fixed-by: Itay Gazit <itaygazit@gmail.com>
2009-10-14 02:11:16 +01:00
Michael Brown 0b1222f233 [hermon] Randomise the high-order bits of queue pair numbers
The Infiniband Communication Manager will refuse to establish a
connection if it believes the connection is already established.
There is no immediately obvious way to ask it to tear down the
existing connection and replace it; to issue a DREP we would need to
know the local and remote communication IDs used for the previous
connection setup.

We can work around this by randomising the high-order bits of the
queue pair number; these have no significance to the hardware, but are
sufficient to convince the IB CM that this is a different connection.
2009-08-10 22:19:39 +01:00
Michael Brown 18bcdfb1cc [hermon] Allow for multiple calls to ib_modify_qp() 2009-08-08 23:49:59 +01:00
Michael Brown ad66465b3c [hermon] Add support for multiple ports and detecting non-IB ports
Originally-fixed-by: Itay Gazit <itaygazit@gmail.com>
2009-08-06 01:55:38 +01:00
Michael Brown bb2cf3c8d7 [hermon] Add support for RC queue pairs 2009-07-17 23:06:35 +01:00
Michael Brown cd5a21359c [hermon] Allow software GMA to receive packets destined for QP1
The Linux IB Communication Manager will always send MADs to QP1,
rather than back to the originating QP.  On Hermon, QP1 is by default
handled by the embedded firmware.  We can change this, but the cost is
that we have to handle both QP0 and QP1 (i.e. we have to provide SMA
as well as GMA service in software), and we have to use MLX queues
rather than standard UD queues (i.e. we have to construct the UD
datagrams by hand).

There doesn't seem to be any viable way around this situation, ugly
though it is.
2009-07-17 23:06:35 +01:00
Michael Brown c44a193d0d [legal] Add a selection of FILE_LICENCE declarations
Add FILE_LICENCE declarations to almost all files that make up the
various standard builds of gPXE.
2009-05-18 08:33:25 +01:00
Michael Brown c9fb012d4f [Infiniband] Add multiport support for Arbel cards 2008-04-22 01:37:00 +01:00
Michael Brown 35a5836677 [Infiniband] Move event-queue process from driver to Infiniband core 2008-04-21 13:23:11 +01:00
Michael Brown e55bab3ce3 [Hermon] Fix event queue doorbells.
Event queue doorbells must use UAR pages 0-127 depending on event queue
number; other doorbells must use pages 128+ (and we choose to use page
128).
2008-04-18 05:33:39 +01:00
Michael Brown a176a24ac0 [Infiniband] Add preliminary multiple port support for Hermon cards
Infiniband devices no longer block waiting for link-up in
register_ibdev().

Hermon driver needs to create an event queue and poll for link-up events.

Infiniband core needs to reread MAD parameters when link state changes.

IPoIB needs to cope with Infiniband link parameters being only partially
available at probe and open time.
2008-04-18 02:50:48 +01:00
Michael Brown bc3f0a25c6 [Infiniband] Add preliminary support for multi-port devices.
Arbel and Hermon cards both have multiple ports.  Add the
infrastructure required to register each port as a separate IB
device.  Don't yet register more than one port, since registration
will currently fail unless a valid link is detected.

Use ib_*_{set,get}_{drv,owner}data wrappers to access driver- and
owner-private data on Infiniband structures.
2008-02-27 13:59:56 +00:00
Michael Brown 396dc6e797 [Hermon] Add driver for Mellanox Hermon (ConnectX) cards. 2008-02-23 10:51:38 +00:00