Jump to content

Dell Force10 S60 Switch

From Pulsed Media Wiki


Pulsed Media runs its access layer on a mix of Arista 7050-series switches and Dell Force10 S60 units in its own datacenters in Finland. In 2026 one of those S60s lost a group of four consecutive odd-numbered copper ports: ports 17, 19 and 21 dropped from 1 Gbps to 100 Mbps in the same SNMP polling cycle, and two days later those three and port 23 all lost link in the same poll. Moving the same four cables to ports on a sibling S60 brought every link straight back at 1 Gbps, with at most one input error between them. The servers were alive. The switch ports were dead.

The S60 is a 1U gigabit access switch running Dell's FTOS, built around a 1.25 GB packet buffer, which Dell called the largest in the industry. That buffer is the S60's main strength. It also has two failure patterns that anyone running a used unit should know: copper ports that fail in clusters, and a power supply and fan arrangement that is easy to misread over SNMP. This page covers both, with the commands and OIDs that separate a switch fault from a cable fault or a powered-off server.

Specifications

Attribute Value
Form factor 1U fixed-configuration access switch
Copper ports 44 × 10/100/1000BASE-T (RJ45)
Fibre ports 4 × GbE SFP
Expansion 2 high-speed slots, each taking a 2-port 10 GbE SFP+ uplink module, a 2-port 12 Gbps stacking module or a 1-port 24 Gbps stacking module; up to four 10G uplinks, stacks of up to 12 units
Fans Dual hot-swappable fan modules
Power Ships with one AC or DC supply; runs on a single supply; redundant hot-swap supply supported
Second bay Holds either a second supply or a fan-only module
Supply design Each supply module has an integrated fan
Power draw 175 W nominal, 225 W maximum
Ambient operating range 0 to 50 °C
Internal temperature limit Not published by Dell
Rack clearance 12.7 cm front and rear
Packet buffer 1.25 GB
Airflow Normal (I/O panel to PSU side) or reversed (PSU side to I/O panel)
Chassis MIB F10-S-SERIES-CHASSIS-MIB

Sources: Dell Force10 S60 data sheet and the Dell installation guide.

Dell's installation guide for the S60 says: "Two PSUs are required for full redundancy, but the system will operate with a single PSU. You can equip one of the PSU bays with a fan-only module." A standard unit therefore carries one supply and one fan-only module, and that second bay causes most SNMP misreadings (see Diagnostics).

Airflow direction is a hard constraint. Fan and supply modules come in normal and reversed versions, and if the directions are mismatched the switch powers itself down within one minute. Confirm the airflow of any replacement module before it goes in.

Why the deep buffer matters

An access switch in a hosting rack aggregates: 40 or more server ports send into one or two uplinks. Traffic from many users arrives in bursts, and when several ports burst toward the same uplink at once the switch has to hold the excess until the uplink can send it. A shallow-buffer switch cannot, so it drops packets, every affected TCP flow backs off, and the uplink sits partly idle even though demand exceeds its capacity. A deep buffer absorbs those bursts, so the aggregate can actually fill the uplink.

The difference is large. At 10 Gb/s, the S60's 1.25 GB buffer holds about one second of line-rate traffic; a shallow-buffer switch with 4 MB, like those in the replacement table below, holds about 3 ms. The cost of a deep buffer is queueing delay while it is full, which hurts bulk transfer far less than dropped packets do. In Pulsed Media's experience, a typical low-buffer top-of-rack switch in this role carries only 25 to 40% of its uplink speed with many users before loss sets in, below the standard Pulsed Media runs to. A deep-buffer switch can take 40+ busy ports down to one uplink and run that uplink to full speed.

Known failure modes

Copper ports failing in groups

The four-port failure described above followed a clear sequence. Ports 17, 19 and 21 fell from 1 Gbps to 100 Mbps between one poll and the next. Two days later those three and port 23 lost carrier, again between one poll and the next. Moved unchanged to a sibling S60, all four cables linked at 1 Gbps immediately. That single test cleared the cables and the far-end devices and put the fault inside the switch.

Two more ports on the same unit failed in different ways, which shows what the degradation looks like before a port dies outright.

One port began accumulating input errors, and packet loss grew with frame size: 26% at 200 bytes, 73% at 1472 bytes. A longer frame carries more bits, so at a given bit-error rate more long frames fail their checksum and get dropped. Congestion and software faults do not scale with frame size; physical-layer faults do. An MTU mismatch does depend on size, but it shows as a cliff at one size with clean delivery below it. The port then lost carrier completely. A new cable did not restore it. The same host, moved to a different S60, linked at 1 Gbps with zero errors.

Another port would only hold 100 Mbps. The server's Intel NIC logged "Link Speed was downgraded by SmartSpeed". Gigabit copper (1000BASE-T) needs all four pairs; 100BASE-TX needs two. When gigabit negotiation keeps failing, Intel NICs fall back to 100 Mbps and report it as a SmartSpeed downgrade. The NIC writes the log line, but the failing pair can be at either end of the cable. The same server and cable on a different S60 negotiated 1 Gbps cleanly.

Across the whole unit, 13 ports showed nonzero input error counters. Sibling S60 units doing the same job showed between 0 and 3 each.

Four consecutive odd-numbered ports losing link in the same poll points at shared hardware rather than four independent cable faults. On access switches, one physical-layer (PHY) chip commonly serves a block of neighbouring ports, so a failing PHY would present exactly like this. That is an inference, not a documented fact: Dell does not publish the S60's front-panel PHY part or its port-to-chip mapping. A S60 boot log posted on Dell's community forum mentions a Broadcom BCM54980, an 8-port PHY, but that line refers to eTSEC1, the CPU's own boot and management Ethernet interface, so it says nothing about the front-panel ports.

No Dell advisory or firmware fix for S60 port failures was found. A 2019 post on Dell's community forum asks whether a used S60 has about six dead ports, three of them in a row. The thread was never resolved, so the cause is unknown, but the pattern resembles what Pulsed Media saw.

One false positive looks identical in the logs. A server NIC that is powered off or asleep often keeps a low-speed link up (10 or 100 Mbps) for Wake-on-LAN, then drops it completely when standby power goes. A rack or PDU losing power, or a batch of servers shut down together, therefore produces several ports dropping to 100 Mbps and then losing link in the same poll. Simultaneous port drops prove that something happened at one moment. They do not prove which end it happened to. The test that settles it is the one used above: move one affected cable, with its device, to a known-good port on another switch.

Fans, filters and heat

Each supply module contains its own fan, so power and cooling are coupled: removing a supply also removes the airflow it provided. Module hot-swap works only when a second supply is installed and running. With redundant supplies in place, a fan module can be removed without interrupting traffic. On a single-supply unit, the standard configuration, replacing either the supply or the fan module is a service window, not a hot-swap. A replacement with the wrong airflow direction triggers the one-minute shutdown whether or not the swap was hot.

The switch logs two messages worth alerting on. A failed or missing fan tray:

%CHMGR-2-FAN_TRAY_BAD: Major alarm: fan tray 0 in unit 0 is missing or down

And the warning that precedes a thermal shutdown:

%CHMGR-2-TEMP_SHUTDOWN_WARN: WARNING! Unit 0 temperature is ...

Check the air filters. Dell's installation guide calls for checking the fan air filters every six months, and on the S60 this is not optional housekeeping: the filters do clog, and a clogged filter starves the fans on a switch that otherwise runs untouched for years. Rising fan speed is the early sign. The filters fit the reverse-airflow (-R) fan and supply modules, one per fan, and pop out without tools. Dell is explicit that they cannot be cleaned and reused, only replaced; the part it names is the Qualtek 09150-M/30, 40 mm filter media at 30 PPI. Pulsed Media has seen S60 filters clogged. Its datacenter air is clean enough that it runs its S60s with the filters removed; in a dusty room, keep them fitted and replace them on that six-month schedule.

Temperature readings across Pulsed Media's S60 units lined up with cooling redundancy. The two hottest units, both at 76 °C on the internal sensor, were exactly the two with fan tray 2 reporting down. The two coolest, at 55 and 58 °C, were the two running both supplies. The unit that lost its port group was one of the two hot ones. That heat caused the port failures is not proven. Dell gives only the 0 to 50 °C ambient range and no limit for the internal sensor, so 76 °C cannot be called out of spec. As a general rule of electronics reliability, electrolytic capacitor life roughly halves for every 10 °C rise, which makes a missing fan tray a cheap and obvious thing to fix first.

Power supply failures

A standard S60 ships with a single supply, so when it fails the switch goes dark along with every server behind it. The failure shows up in three ways:

  • the switch shuts itself down or restarts without an operator action, and show system gives an unplanned Last Restart Reason;
  • the front PWR LED turns yellow, which Dell defines as the supply operating outside its expected parameters (a yellow PSU0 or PSU1 LED instead means one of that module's fans is out of range);
  • a bay holding a real supply reports type ac (1) with status down (2), or show environment pem shows it down, while the supply LED is off;
  • after a power event the switch will not start at all.

Pulsed Media has seen an S60 repeatedly shut itself down, with a supply or thermal fault suspected but not confirmed, and another unit whose installed second supply reports down while the first carries the load. Both public reports found start with a power event, and neither was resolved. In a Dell community thread about an S60n, a surge left one supply bay showing absent; a brand-new supply in that bay still showed absent with no LEDs, which puts that fault in the chassis rather than in the module. In a 2019 Dell community thread, an S60 would not boot after a power outage: fans at 100%, the power LED blinking green, and no output on the console or USB port.

No teardown or failure analysis of the S60 supply itself has been published. The general pattern for switch-mode supplies of this age is well documented: electrolytic capacitors are the most common cause of failure, the electrolyte dries out over years of heat, and most fail silently rather than visibly (SMPS troubleshooting guide). An S60 supply that runs hot for years, in a chassis with a missing fan tray or clogged filters, is the textbook case.

S60 supplies are not 460 W Delta units sold as Dell M129P or Y004D. Y004D is a 2360 W supply for the PowerEdge M1000e blade enclosure, and no S60 listing for M129P was found. The S60 uses its own modules, S60-PWR-AC and S60-PWR-AC-R, the "-R" marking reversed airflow (one such module carries the label part number 0NDJM9). Check the airflow marking on the module itself, not the listing title.


Diagnostics

Symptom Likely causes Test Fix
Several ports drop to 100 Mbps or lose link in the same poll Shared PHY failing, or far-end servers losing standby power Move one cable and its device to a known-good port on another switch Links up elsewhere: switch fault, move the links. Stays down: far-end power
Port stuck at 100 Mbps, NIC logs a SmartSpeed downgrade Failing pair in the cable or in either port show interfaces gigabitethernet 0/N phy; move the cable to another switch Gigabit elsewhere: retire the port
Input errors climbing, loss grows with frame size Physical-layer fault in the cable or switch port Ping at several payload sizes; watch the per-poll error delta, not the total; TDR with the far end shut Replace the cable if TDR shows a pair fault; otherwise move the link
No carrier, new cable does not help Dead switch port Same cable and device on another switch Retire the port
Fan tray reports down, FAN_TRAY_BAD in syslog Failed or missing fan module show environment fan; check the air filter Replace the fan module with matching airflow
Second PSU bay reports down, type 0 Fan-only module in that bay (normal) Check chSysPowerSupplyType No action
PSU bay reports down, type ac (1) Failed or unpowered supply show environment pem; check the feed and seating; try a known-good module Replace or repair the supply; if a known-good module also reads down or absent, the fault is in the chassis
High internal temperature, TEMP_SHUTDOWN_WARN Missing fan tray or supply, clogged filter show environment thermal-sensor; inspect the filters; rising fan speed Replace clogged filters; restore fan and supply redundancy
Uptime suspiciously short on an old switch 32-bit sysUpTime wrapped at about 497 days show system, Last Restart Reason No restart recorded: the counter wrapped

CLI

show environment all
show environment fan
show environment pem
show environment stack-unit
show environment thermal-sensor
show interfaces gigabitethernet 0/N phy
show system

pem is the power entry module, the supply bays. The phy form shows the autonegotiation result and what the link partner advertised, which is the starting point for a port stuck at 100 Mbps. show system includes a Last Restart Reason field, which separates an administrative reboot from an unplanned shutdown.

TDR cable test

The S60 configuration guide for FTOS 8.3.3.8 documents a time-domain reflectometry test that checks each of the four pairs for opens and shorts:

tdr-cable-test gigabitethernet 0/N
show tdr gigabitethernet 0/N

The test is intrusive. Do not run it on a link passing traffic, shut the far-end port first, do not start it from both ends, and keep the interface under test enabled. It appears in the configuration guide but not in the CLI references, so check tdr-cable-test ? on the running image before relying on it. TDR is built to find cable faults. A clean result does not prove the port electronics are healthy, so moving the link to another switch remains the deciding test.

SNMP

Power supply, fan and temperature state are in the F10-S-SERIES-CHASSIS-MIB:

chSysPowerSupplyOperStatus  .1.3.6.1.4.1.6027.3.10.1.2.3.1.2   1=up 2=down 3=absent
chSysPowerSupplyType        .1.3.6.1.4.1.6027.3.10.1.2.3.1.3   1=ac 2=dc
chSysFanTrayOperStatus      .1.3.6.1.4.1.6027.3.10.1.2.4.1.2   1=up 2=down 3=absent
chStackUnitTemp             .1.3.6.1.4.1.6027.3.10.1.2.2.1.14  Gauge32, no unit in the MIB

The MIB states no unit for the temperature value; Pulsed Media's monitoring reads it as degrees Celsius, which is how the figures on this page are given.

Pulsed Media's monitoring auto-discovery found only the temperature sensor on this platform, with no supply or fan state. A switch showing as monitored does not mean its supplies and fans are. Compare what the monitoring system discovered against the OIDs above and poll them directly if they are missing. Alert on ports that go dark and stay dark, not only on ports that flap.

Reading the supply table correctly avoids two opposite mistakes. A second bay reporting down (2) with type 0, which is neither ac nor dc, is most likely the fan-only module Dell describes, so on a single-supply S60 it is a normal reading. A bay reporting type ac (1) and status down (2) is a real supply that has failed or lost its feed.

Uptime counter wrap

The standard sysUpTime object is a 32-bit count of hundredths of a second, so it wraps to zero after about 497.1 days. Access switches often run far longer than that. A Pulsed Media S60 that had been up for 532 days returned a raw sysUpTime of about 35 days. Before concluding that an old switch rebooted recently, check Last Restart Reason in show system: a real reboot records one, a counter wrap does not.

Repair and replacement

Dead or degraded ports

Move the affected links off the failing ports, preferably to another switch rather than to neighbouring ports, since a failing PHY may take more ports with it. Confirm with a known-live device before writing a port off, and use TDR to rule out the cable.

Fan tray down

Replace the fan module with one of matching airflow. With a redundant supply running, this is a hot-swap; on a single-supply unit, plan a service window.

Reversing a fan module's airflow

A fan module of the wrong airflow direction does not have to be scrapped or reordered. Dell sells each direction as a separate part and its installation guide does not describe a conversion, but Pulsed Media frequently converts them in house. The fans are 40 mm units (Dell's replacement filter for them is 40 mm media); a supply module carries three of its own.

  1. Work on a spare module on the bench where possible. If the installed module has to come out, keep its slot empty for as little time as possible: with a slot empty the switch powers down once it passes its internal temperature limit, and Dell recommends having the tray back within five minutes. To convert the installed module itself, book a service window.
  2. Open the module and turn each fan around so it blows the other way.
  3. Move the module's airflow jumper to the matching position, so the switch reads the module as the new direction. A module whose fans were flipped but whose jumper was not moved reports the wrong direction.
  4. Mark the current direction on the outside of the module with a marker, so the next technician can see what it is without opening it, then fit the module.

The converted module must end up matching every other module in the switch, including the supplies, whose fans are built in and whose direction is set by the part they shipped as. A mismatch shuts the switch down within a minute, so check the supply's airflow marking before choosing the direction to convert to.

Failed power supply

Confirm the supply is the fault before replacing anything:

  1. Check the feed first: the power cord, the PDU outlet and the supply's own switch or LED. A supply with no input reports down exactly like a dead one.
  2. Reseat the module, unplugging its power cord before pulling it and plugging the cord back in only after it is seated. With a second supply running this is a hot-swap; on a single-supply unit it takes the switch down, so plan a window.
  3. Fit a known-good module of matching airflow (see the part numbers below). If it comes up, the old module is the fault.
  4. If a known-good module also reads down or absent, stop replacing supplies: the fault is in the chassis side of the power path, and the switch needs replacing.

Never leave a bay empty. Dell's guide requires an operating fan unit in both slots, as part of a supply or as a fan module, or the switch shuts down when it passes its internal temperature limit, and it recommends replacing a fan module within five minutes. Have the replacement in hand before pulling anything. When a single-supply unit needs a supply or fan replaced anyway, fit a second supply at the same time. Every later fan or supply swap then becomes a hot-swap, and a single supply failure no longer takes the switch and every server behind it offline. Keep a tested spare of the right airflow on the shelf.

Repairing the supply module

Dell's position is that a failed supply must be replaced completely and has no field-serviceable components, so repairing one is outside Dell support and at your own risk. A failed module can often be brought back by replacing its electrolytic capacitors. This is mains-voltage work. The supply's large primary capacitor holds a dangerous charge after the module is unplugged. Do it only if you are competent with mains electronics, and never on a module still in a running switch.

  1. Remove the module and leave it unplugged. Discharge the primary capacitor through a resistor of a few kilohms and a few watts on insulated probes, never with a screwdriver, and confirm 0 V with a meter before touching anything.
  2. Open the module and inspect it: domed or bulging capacitor tops, leaked electrolyte, discoloured board around hot parts, burnt resistors, a blown fuse. A blown fuse means something on the primary side has shorted; a new fuse alone will blow again.
  3. Measure the ESR of every electrolytic capacitor, including the small ones near the control circuitry, not only the large ones. Many dried-out capacitors look normal. Compare each reading with a known-good part of the same value and replace any that read high.
  4. Replace capacitors with new parts of the same capacitance and the same or higher voltage rating, rated 105 °C and low-ESR, in the same diameter and lead spacing. Take the values from the parts on the board, not from a list.
  5. Clean any leaked electrolyte off the board with isopropanol before fitting the new parts. Replace the module's fan if it is worn or noisy; it cools the supply itself.
  6. Test the repaired module in a non-production S60 first. It is ready when show environment pem reports it up and its fan runs, then it can go back on the shelf as the tested spare.

Part numbers

Order by airflow direction. Dell calls the S60's I/O panel the rear and the supply panel the front, so its "rear to front" modules are the normal-airflow ones. show hardware lists the part and serial number of every installed module, which is the quickest way to check what a running switch has.

Module Normal airflow (I/O panel to PSU) Reversed airflow (PSU to I/O panel)
AC supply Dell 331-5225 (S60-PWR-AC) Dell 331-5226 (S60-PWR-AC-R; label: Force10 752-00594-01, Dell P/N 0NDJM9)
DC supply Dell 331-5227 Dell 331-5228
Fan module Dell 331-5231 Dell 331-5232
Fan filter none Qualtek 09150-M/30, one per fan

Dell order codes are from the S60 data sheet; the Force10 and Dell part numbers for the reversed AC supply are from a module label. Resellers list other numbers for the same modules inconsistently, so match the airflow marking on the module itself.

Whole unit degrading

An S60 that has lost a port group and is also running hot on one fan tray is past port-by-port triage. Move its links and replace it, and keep the buffer class when you do. Common 48-port gigabit switches can be shallow-buffer designs: the Dell S3048-ON has 4 MB of packet buffer, roughly one three-hundredth of the S60's, and the 7010T-48, which Arista named as the replacement for its deep-buffer 7048T-A, also has 4 MB. Either would bring back the aggregation limit described above.

Switch Copper ports Uplinks Packet buffer Status
Dell Force10 S60 44 × 1G RJ45 (+ 4 × GbE SFP) up to 4 × 10G SFP+ 1.25 GB —
Arista 7020TR-48 48 × 100M/1G RJ45 6 × 1/10G SFP+ up to 3 GB End of sale June 2025; Arista TAC support to June 2030
Arista 7280TR3-40C6 40 × 10GBASE-T RJ45 (36 usable at 1G/10G) 6 × 100G QSFP 2 GB Current model
Arista 7280TR-48C6 48 × 1G/10G RJ45 6 × 100G QSFP 4 GB End of sale December 2023; software support ends December 2026
Arista 7048T-A 48 × 100M/1G RJ45 4 × 10G SFP+ 768 MB End of sale 2016; support ended 2019
Dell S3048-ON 48 × 1G RJ45 4 × 10G SFP+ 4 MB —

The closest step up is the Arista 7020TR-48: the same class as the S60, 48 gigabit copper ports with 10G SFP+ uplinks, and up to 3 GB of buffer on a virtual-output-queue design, up to about 2.4 times the S60's. It is end of sale but sold used by resellers, and Arista's support for it runs to June 2030. For a new switch, Arista names the 7280TR3-40C6 as the replacement for both the 7020TR-48 and the 7280TR-48C6: 2 GB of buffer, still more than the S60, with 10GBASE-T copper ports that run at 1G for existing servers and 100G uplinks. Deep-buffer switches with SFP+ ports instead of RJ45 need fibre, DAC or transceivers on every server link.

None of the deep-buffer switches above runs copper at 2.5 or 5 Gb/s. The 7020TR-48 is 100M/1G, the 7280TR-48C6 is 100M/1G/10G, and Arista's data sheet lists the 7280TR3-40C6 copper ports as 1G/10G only, so a server with a 2.5G network card would link at 1G there. The switches that do run 2.5G and 5G copper (multi-gigabit, IEEE 802.3bz) are campus access designs that keep their buffer on the switch chip: 8 or 16 MB on the Arista CCS-720XP, 8 MB per chip on the Extreme 5520, and 16 to 64 MB across the Cisco Catalyst 9300 range, where Cisco describes the larger buffers of its higher-scale models as "deep buffer technology". Juniper's EX4400 data sheet and hardware guide give no buffer figure at all. Even 64 MB holds about 50 ms at 10 Gb/s, against about one second on the S60. None of the switches checked here combines the two, so a server that needs deep-buffer aggregation runs at 1G or 10G.

Manufacturers rarely publish packet buffer sizes. Jim Warner's switch buffer list at UC Santa Cruz, and the community-maintained Packet Buffer Reference that continues it, collect the figures from data sheets, vendor papers and published tests, and both warn that many of them are estimates. Both put the S60 and the Arista 7048 in the deep-buffer 1U class.

At Pulsed Media

Pulsed Media's access layer mixes Arista 7050-series switches and Dell Force10 S60 units, as documented on Pulsed Media Datacenters. The failures on this page come from that equipment and were diagnosed with the commands and OIDs above. The four-port group was proven dead by moving the same cables to a sibling S60, where every link came up at 1 Gbps. The error-counter comparison, 13 ports on the failing unit against 0 to 3 on its siblings, is what flagged the unit as a whole rather than one bad port.

Pulsed Media operates the datacenters and the switches in them, so every step of this diagnosis, from the SNMP counters to the cable moves, ran on its own equipment. Server connections on that network run from 1 Gbps to 20 Gbps depending on product tier, and Pulsed Media seedbox plans are hosted on it.

See also