Wayside Fiber, Part 2: How It Talks

This is Part 2 of Wayside Fiber, a three-part series. Part 1 built and proved the physical plant. This part makes it talk, and finds the new ways it can fail.

The dead link

Part 1 ended somewhere strange: a fifty-mile fiber plant, proven to a fraction of a decibel, with a green link LED at every location, over which nothing could say anything to anything. That is worth sitting with, because it marks the boundary between two disciplines. Below the LED, problems are physics: loss, reflectance, water, glass. Above it, problems are decisions: who may talk, to whom, in what order, and how long anyone waits before assuming the worst. Every failure in this part happens on a physically perfect network.

Inventory before architecture

The temptation is to start drawing subnets. Resist it, because the network's structure should fall out of a question that has nothing to do with IP: how long can each device tolerate silence?

Take the census of the corridor. It sorts into three tolerance classes:

• Vital. The interlocking controllers at Arden, Bowman, and Chester, and the office gateway that carries vital traffic between them. These devices exchange safety-critical state, and their tolerance for disruption is measured in seconds, enforced by fail-safe: silence beyond a timer is treated as danger, and the railroad stops. A comms failure here is not an outage, it is an operational event with trains braking.
• Essential non-vital. The crossing monitor at Arden, Det 381's reporting, event recorders, SCADA for power switches and house alarms. Silence is tolerable for minutes: nothing unsafe happens, but the railroad is flying blind on health and diagnostics, and somebody should be getting paged.
• Support. CCTV, VoIP phones, the maintainer's laptop port. Tolerance: hours to days. Annoying, ticketed, not paged at 3 a.m.

The middle tier earns its separate existence. It is tempting to lump everything non-vital together, but a crossing monitor that stops reporting is not a camera that stops streaming: the monitor's silence has regulatory weight, feeds mandated response procedures, and shortens the time the railroad has to catch a crossing malfunction before the public does. "Non-vital" spans two very different kinds of consequence, and the network should know the difference.

Three tolerances, three blast radii, and that is the entire argument for segregation, made from evidence rather than habit: devices with second-scale, fail-safe consequences must not share fate with devices whose failure tolerance is "whenever." A camera firmware bug that floods the wire with multicast is an inconvenience on a support network and a train-stopping event on a shared one. The physical plant already made this cut once, in glass, when tube 1 went to vital alone. The logical plant now has to make the same cut in configuration, and unlike glass, configuration can be wrong by Tuesday.

The flat-namespace trap

Here is how it goes wrong on a real railroad, slowly and then suddenly. The corridor gets commissioned with one subnet, a single flat /24, because there are only eleven devices and a deadline. It works. It works next year, with twenty devices. It works for a decade, at sixty devices, each address handed out in the order equipment arrived, recorded (sometimes) in a spreadsheet column.

Then the wall: address 254 gets assigned, or, more likely, long before that, the ambiguity becomes lethal. Nobody can say what .87 is without tracing a cable. Free addresses are only "probably free." And the fix everyone proposes, renumbering, turns out to be nearly impossible, because renumbering vital equipment is not an IT change. Vital device addresses are burned into peer tables in safety-validated configurations. Those configurations were tested when placed in service; touch them and the tests come back, with a signal engineer, at night, under traffic, per location, with FRA paperwork. Renumbering a corridor of vital equipment costs more than the network did. The flat namespace was never cheap; it was a loan, and this is the balloon payment.

Twelve years in, under two regimes. Both cost the same on day one.

Designing the schema

So the schema gets designed once, on paper, before the first address is assigned, and the design principle is: an address should mean something to a human being reading it cold. Space costs nothing (this is all private space; there are 65,000 addresses in one /16), so spend space to buy meaning:

Four octets, four questions answered before anyone opens a spreadsheet.

• First octet: 10. Private space, never routed beyond the railroad.
• Second octet: the corridor. This corridor is 128; the next line the railroad builds gets its own.
• Third octet: the location, stepping by ten in milepost order: Waverly is 10, Arden 20, Int 187 is 30, Bowman 40, Det 381 is 50, Chester 60. The gaps are deliberate. When a location gets added at MP 44 someday, it becomes 55, and the schema absorbs it without renumbering a soul. Det 381, the location that did not exist in Part 1's original design, slotted into a gap exactly this way.
• Fourth octet: the role, in reserved ranges: .1-.15 network equipment, .16-.31 vital, .32-.63 essential non-vital, .64-.127 support, and .128-.254 assigned to nothing, reserved for roles nobody has invented yet.

So 10.128.40.17 reads, with no documentation at hand: this corridor, CP Bowman, vital range, controller A. A maintainer seeing it in a log at 3 a.m. knows what it is and how careful to be. Space is free; discipline is not, and the discipline is the ranges: the day someone parks a camera at .30 "just for now," the schema starts down the same road as the flat /24.

VLANs: the same cut, one layer up

The tolerance classes become VLANs: 110 vital, 120 essential, 130 support, plus a management VLAN for the switches themselves. Every switch port on the corridor is assigned, tagged, and documented, and untagged traffic goes nowhere: the native VLAN is an unused, unrouted parking lot, so a device plugged into the wrong jack gets silence instead of a seat on someone else's network.

What crosses between VLANs matters more than what lives inside them. Support and essential traffic route through the office like any enterprise network. VLAN 110 is different: nothing routes into it except through the vital gateway at Waverly, one inspected, deliberately narrow doorway. The interlocking controllers speak their vital protocol to each other and to the office, and that is the complete list of conversations. No laptop reaches a vital controller directly, ever; even the signal engineer's maintenance access goes through the gateway, logged.

The same corridor as Part 1. The plant did not change; we added a layer of decisions on top of it.

The convergence race

Now the centerpiece, and the deepest way the logical plant can betray the vital one. It involves no misconfiguration at all: every device below is working exactly as designed.

The setup: the six switches form the ring from Part 1. The interlocking controllers exchange vital state continuously, roughly once a second, and each runs a stale-data timer: if fresh data has not arrived within 2.0 seconds, the processor does the only safe thing it can do with unknown information. It treats silence as danger, drops the affected functions to their most restrictive state, and the signals go to stop. This is exactly the fail-safe principle from relay practice: de-energize to danger. It is not negotiable, and you would not want it to be.

The other clock: when the ring breaks, say a fiber pair fails at Int 187, the switches must detect the break and re-route traffic the long way around. Rapid Spanning Tree, the standard enterprise answer, re-learns the topology in a few seconds on a ring this size: a perfectly respectable performance by enterprise standards, where a few seconds of loss means a paused video call.

Watch both clocks at once:

The race. RSTP loses it three seconds after the railroad has already stopped.

At t=0 the fiber breaks. At t=2.0 the vital timers expire and three interlockings drop to fail-safe. At roughly t=5 RSTP finishes reconverging and the network is, by every metric it knows, healthy again. The network healed successfully. The railroad stopped anyway, and stayed stopped for the minutes it takes vital systems to re-establish sessions and for signals to clear back up, with a train on the main dumping its air somewhere around MP 20.

The obvious question is why nobody simply lengthens the vital timer to outlast RSTP. Because the timer is not a network setting; it is a safety parameter. During the stale-data window the processor is acting on old information, and the width of that window is an input to the hazard analysis: how far reality is permitted to drift from the processor's picture of it before the system must assume the worst. Widening it to accommodate a slow network means widening it for every failure, including the ones the timer exists to catch. The safety layer's numbers are fixed by analysis; the network gets to be exactly as good as they demand, not the other way around.

This is why railroads and utilities run fast ring protocols, proprietary vendor rings or ERPS-class standards, purpose-built for a known ring topology and specced to heal in under 50 milliseconds. Fifty milliseconds is not a luxury number; it is "faster than the vital timer by a factor of forty," chosen so that the physical event completes its repair before the safety layer ever notices. The general lesson: every timer in the vital stack must be slower than the sum of everything below it. Nobody gets to configure the network and the signal system in separate rooms and hope.

Rules that are not negotiable

Three of them, each earned somewhere expensive:

No DHCP anywhere near vital. A vital device's address is part of its safety configuration: static, recorded, and tested in place. Dynamic assignment means an address can change for reasons no signal engineer authorized, and it puts a protocol server in the vital path. Static everywhere on 110 and 120, and every assignment written down at the moment it is made.

The record is part of the install. An undocumented static address is a landmine with a delay fuze (the next section steps on one). The assignment record is not paperwork after the work; it is the work.

One time source, arranged in a hierarchy. A GPS clock at Waverly, every switch and device on the corridor disciplined to it via NTP. Not for convenience: when something goes wrong on a railroad, the reconstruction lives or dies on whether the event recorder at Arden, the interlocking log at Bowman, and the office log agree on when things happened. Regulators expect a coherent timeline; a corridor with three opinions about what time it is cannot produce one.

The duplicate address

Now the failure the series introduction planted. The event recorder at Chester dies. A replacement is staged on the bench at the office, and the tech, sensibly, loads it from a saved config rather than typing forty parameters. The saved config is Arden's. The recorder ships to Chester, gets racked, links up, and begins answering as 10.128.20.34, an address that is alive and well ninety minutes away at CP Arden.

What follows is two days of high-quality confusion. Both recorders answer ARP for the same address; every switch along the ring flips its tables toward whichever spoke last. The office poller reaches Arden's recorder, then Chester's, then neither. From the outside it looks exactly like a flaky physical path: intermittent reachability, no pattern, worse under load. A tech shoots the Waverly-Arden section with an OTDR and finds, of course, nothing, because Part 1's plant is blameless. The trace gets a second opinion. Someone requisitions replacement optics.

The tell, when someone finally sees it, is embarrassingly small: the ARP entry for .34 alternating between two MAC addresses. Total cost of the incident: an address that was never checked against the record at the moment of staging. The schema said .34 belongs to Arden's ranges; the record said .34 was in service; the bench had neither open. Every rule in the previous section exists because of afternoons like this.

Who owns the record

Which raises the real question: what record? The industry answer is IPAM, IP address management, and the honest answer on most railroads is a spreadsheet, maintained by one person, whose name everyone knows. When it is current, it is because that person is disciplined. When they retire, the corridor's address plan retires with them.

It does not need to be sophisticated. A version-controlled table that is required reading before any address is assigned and required writing after beats an enterprise IPAM platform that the signal gang is not allowed to log into. What it cannot be is optional, and it cannot be plural: two spreadsheets are the same as none.

The seam

Notice who has appeared in this part: switch configs and VLANs, which look like IT; vital timers and fail-safe drops, which are signal engineering; an address plan, which is both. On most railroads these are different departments with different training, different change processes, and different definitions of the word "urgent." IT owns addressing everywhere else in the company, and is good at it. Signals owns everything vital, and must. Both positions are defensible, which is what makes the boundary dangerous: the convergence race and the duplicate address both live exactly on it, in territory each side reasonably assumes the other is watching. There is rarely a written process at that seam.

The physical plant fails by backhoe, the logical plant by assumption. What keeps both honest for thirty years is neither cable nor configuration but institutions: records, money, and process. That is Part 3: Who Keeps It True.