Wayside Fiber, Part 2: How It Talks

Part 2 of Wayside Fiber. Part 1 built and proved the physical plant; this part makes it talk.

The dead link

Part 1 ended somewhere strange: a fifty-mile plant proven to a fraction of a decibel, a green link LED at every location, and nothing able to say anything to anything. The LED marks the boundary between two disciplines. Below it, problems are physics: loss, reflectance, water. Above it, problems are decisions: who may talk, to whom, and how long anyone waits before assuming the worst. Every failure in this part happens on a physically perfect network.

Inventory before architecture

The temptation is to start drawing subnets. Resist it: the structure should fall out of a question that has nothing to do with IP: how long can each device tolerate silence? The census sorts into three classes:

• Vital. The interlocking controllers at Arden, Bowman, and Chester, and the office vital gateway through which the dispatcher's control reaches them. The controllers exchange safety-critical state with each other directly, peer to peer; the gateway is the office's doorway into that conversation, not a relay the field depends on. Tolerance for disruption is measured in seconds, enforced by fail-safe: silence beyond a timer is treated as danger, and the railroad stops. A comms failure here is not an outage, it is an operational event with trains braking.
• Essential non-vital. The crossing monitor at Arden, Det 381's reporting, event recorders, SCADA for power switches and house alarms. Silence is tolerable for minutes: nothing unsafe happens, but the railroad is flying blind on health and diagnostics, and somebody should be getting paged.
• Support. CCTV, VoIP phones, the maintainer's laptop port. Tolerance: hours to days. Annoying, ticketed, not paged at 3 a.m.

The middle tier earns its existence: a crossing monitor that stops reporting is not a camera that stops streaming. The monitor's silence has regulatory weight and shortens the time to catch a crossing malfunction before the public does. "Non-vital" spans two kinds of consequence, and the network should know the difference.

Three tolerances, three blast radii: that is the whole argument for segregation, from evidence rather than habit. A camera firmware bug flooding the wire with multicast is an inconvenience on a support network and a train-stopping event on a shared one. The physical plant made this cut once, in glass, when tube 1 went to vital alone. The logical plant must make the same cut in configuration, and configuration can be wrong by Tuesday.

The flat-namespace trap

Here is how it goes wrong, slowly and then suddenly. The corridor gets commissioned on a single flat /24, because there are eleven devices and a deadline. It works. It works next year at twenty devices, and for a decade at sixty, each address handed out in arrival order, recorded (sometimes) in a spreadsheet column.

Then the wall: long before address 254 is assigned, the ambiguity becomes lethal. Nobody can say what .87 is without tracing a cable; free addresses are only "probably free." And renumbering, the fix everyone proposes, is nearly impossible, because renumbering vital equipment is not an IT change. Vital addresses are burned into peer tables in safety-validated configurations; touch them and the tests come back, with a signal engineer, at night, under traffic, with FRA paperwork. Renumbering a corridor of vital equipment costs more than the network did. The flat namespace was a loan, and this is the balloon payment.

Twelve years in, under two regimes. Both cost the same on day one.

Designing the schema

So the schema is designed once, before the first address is assigned, on one principle: an address should mean something to a human reading it cold. Space costs nothing in a private /16; spend it to buy meaning:

Four octets, four questions answered before anyone opens a spreadsheet.

• First octet: 10. Private space, never routed beyond the railroad.
• Second octet: the corridor. This corridor is 128; the next line the railroad builds gets its own.
• Third octet: the location, stepping by ten in milepost order: Waverly is 10, Arden 20, Int 187 is 30, Bowman 40, Det 381 is 50, Chester 60. The gaps are deliberate. When a location gets added at MP 44 someday, it becomes 55, and the schema absorbs it without renumbering a soul. Det 381, the location that did not exist in Part 1's original design, slotted into a gap exactly this way.
• Fourth octet: the role, in reserved ranges: .1-.15 network equipment, .16-.31 vital, .32-.63 essential non-vital, .64-.127 support, and .128-.254 assigned to nothing, reserved for roles nobody has invented yet.

So 10.128.40.17 reads, with no documentation at hand: this corridor, CP Bowman, vital range, controller A. A maintainer seeing it in a log at 3 a.m. knows what it is and how careful to be. Space is free; discipline is not, and the discipline is the ranges: the day someone parks a camera at .30 "just for now," the schema starts down the same road as the flat /24.

One clarification: the third octet is a human convention, not a routing boundary. Each VLAN carries one corridor-wide subnet inside 10.128.0.0/16, so vital controllers at Arden and Chester are neighbors with no router between them: that is what lets interlockings peer directly, and why a duplicate address can collide across ninety minutes of railroad. The location octet buys legibility; isolation is the VLANs' job, deliberately not the mask's.

VLANs: the same cut, one layer up

The tolerance classes become VLANs: 110 vital, 120 essential, 130 support, plus a management VLAN for the switches themselves. Every switch port on the corridor is assigned, tagged, and documented, and untagged traffic goes nowhere: the native VLAN is an unused, unrouted parking lot, so a device plugged into the wrong jack gets silence instead of a seat on someone else's network.

What crosses between VLANs matters more than what lives inside them. Support and essential traffic route through the office like any enterprise network. VLAN 110 is different: nothing enters except through the vital gateway at Waverly, one inspected, deliberately narrow doorway, built as a redundant active/standby pair; a corridor that just paid for a diverse-route cable does not leave one power supply as the equal of a backhoe. And its failure costs the right thing: the interlockings peer directly, so losing both gateways loses dispatcher control while the field keeps protecting itself. Degraded, loudly alarmed, safe. No laptop reaches a vital controller directly, ever; even maintenance access goes through the gateway, logged.

The same corridor as Part 1. The plant did not change; we added a layer of decisions on top of it.

The convergence race

Now the centerpiece, and the deepest way the logical plant can betray the vital one. It involves no misconfiguration at all: every device below is working exactly as designed.

The setup: the five signal-house switches form the ring from Part 1 (Det 381 hangs off its own pair; never ring material). The interlockings exchange vital state about once a second, and each runs a stale-data timer: no fresh data within 2.0 seconds and the processor does the only safe thing possible with unknown information. Silence is danger, affected functions drop to most restrictive, signals go to stop. Relay practice's fail-safe principle, de-energize to danger: not negotiable, and you would not want it to be.

The other clock: when the ring breaks, say a fiber pair fails at Int 187, the switches must detect the break and re-route traffic the long way around. Rapid Spanning Tree, the standard enterprise answer, usually does this fast: a cleanly detected link-down converges in well under a second. Its worst cases, an indirect failure, a lost BPDU, a unidirectional fault, run to several seconds, and nothing in the protocol bounds which case you get. By enterprise standards that is perfectly respectable, because the worst case costs a paused video call.

Watch both clocks at once:

The race. RSTP loses it three seconds after the railroad has already stopped.

The timeline draws RSTP's bad day, and the bad day is the only day a safety argument is allowed to consider. At t=0 the fiber breaks. At t=2.0 the vital timers expire and three interlockings drop to fail-safe. At roughly t=5 RSTP finishes reconverging and the network is, by every metric it knows, healthy again. The network healed successfully. The railroad stopped anyway, and stayed stopped for the minutes it takes vital systems to re-establish sessions and for signals to clear back up, with a train on the main dumping its air somewhere around MP 20. And here is the honest version of the disqualifier: RSTP's typical day would win this race. It loses on the certification question, because no one can put a bounded number on its worst case, and a safety function cannot be certified against "usually."

Why not lengthen the timer to outlast RSTP? Because it is a safety parameter, not a network setting. The stale-data window is an input to the hazard analysis: how far reality may drift from the processor's picture before the system assumes the worst. Widen it for a slow network and you widen it for every failure, including the ones it exists to catch. The safety layer's numbers are fixed by analysis; the network gets to be exactly as good as they demand, not the other way around.

This is why railroads and utilities run fast ring protocols, proprietary vendor rings or ERPS-class standards, purpose-built for a known ring topology and specced to heal in under 50 milliseconds, with a bound you can put in a safety case. Fifty milliseconds is not a luxury number; it is "faster than the vital timer by a factor of forty," chosen so that the physical event completes its repair before the safety layer ever notices. (The newest practice goes one step further: IEC 62439-3 PRP and HSR duplicate every frame across both paths and heal in zero milliseconds, and they are increasingly common in rail and substation work. Against a 2.0-second timer, ERPS remains a perfectly sound choice.) The general lesson: every timer in the vital stack must be slower than the sum of everything below it. Nobody gets to configure the network and the signal system in separate rooms and hope.

One honesty note: redundancy protects the path, not the endpoints. If Arden's switch dies, the ring heals flawlessly and Arden is still gone: its interlocking times out and that control point stops. One switch per house is reasonable (the interlocking it serves is single too), but it is an accepted residual risk, and the safety case should say so out loud.

Rules that are not negotiable

Three of them, each earned somewhere expensive:

No DHCP anywhere near vital. A vital address is safety configuration: static, recorded, tested in place. Dynamic assignment lets an address change without a signal engineer's authorization and puts a protocol server in the vital path. Static on 110 and 120, written down the moment it is assigned.

The record is part of the install. An undocumented static is a landmine with a delay fuze; the next section steps on one. The record is not paperwork after the work; it is the work.

One time source, arranged in a hierarchy. A GNSS-disciplined clock at Waverly, multi-constellation, with days of oscillator holdover (antenna failures and jamming are no longer exotic) and its health alarmed: a dead reference nobody notices produces exactly the drifting clocks the rule exists to prevent. Everything disciplines to it via NTP. Not for convenience: accident reconstruction lives or dies on whether the recorder at Arden, the log at Bowman, and the office agree on when things happened. A corridor with three opinions about the time cannot produce the coherent timeline regulators expect.

The duplicate address

Now the failure the series introduction planted. The event recorder at Chester dies. A replacement is staged on the bench at the office, and the tech, sensibly, loads it from a saved config rather than typing forty parameters. The saved config is Arden's. The recorder ships to Chester, gets racked, links up, and begins answering as 10.128.20.34, an address that is alive and well ninety minutes away at CP Arden.

Two days of high-quality confusion follow. Both recorders answer ARP for the same address; switch tables flip toward whichever spoke last; the office poller reaches Arden's recorder, then Chester's, then neither. It looks exactly like a flaky physical path: intermittent, patternless, worse under load. A tech shoots the Waverly-Arden section and finds nothing, because Part 1's plant is blameless. The trace gets a second opinion. Someone requisitions optics.

The tell, when someone finally sees it, is small: the ARP entry for .34 alternating between two MACs. Root cause: an address never checked against the record at staging. The schema said .34 was Arden's; the record said it was in service; the bench had neither open. Every rule above exists because of afternoons like this.

Who owns the record

Which raises the question: what record? The industry answer is IPAM; the honest answer on most railroads is a spreadsheet maintained by one person whose name everyone knows. When they retire, the address plan retires with them.

It does not need to be sophisticated. A version-controlled table that is required reading before any assignment and required writing after beats an enterprise IPAM the signal gang cannot log into. It cannot be optional, and it cannot be plural: two spreadsheets are the same as none.

The seam

Notice who appeared in this part: switch configs and VLANs, which look like IT; vital timers and fail-safe drops, which are signal engineering; an address plan, which is both. On most railroads these are different departments with different training and different definitions of "urgent." IT owns addressing everywhere else, and is good at it. Signals owns everything vital, and must. Both positions are defensible, which is what makes the boundary dangerous: the convergence race and the duplicate address both live exactly there, in territory each side assumes the other is watching. There is rarely a written process at that seam.

The physical plant fails by backhoe, the logical plant by assumption. What keeps both honest for thirty years is institutions: records, money, process. That is Part 3: Who Keeps It True.