When the lights go out.

Grace Ewura-Esi:

You're listening to Traceroute, a podcast about the inner workings of our digital world. I'm Grace Ewura-Esi.

Shweta Saraf:

And I'm Shweta Saraf.

Grace Ewura-Esi:

Shweta, I'm so excited that we're talking about this today. What happens when the lights go out is, well, provocative for anyone in technology, but especially someone like you, with the experience that you have with network modernization and what it means to be on the cutting digital edge. And so what I want to know is inside of the work you do, what's been the biggest light out event and how did you react? What was the engineering fix that had to be done in the dark?

Shweta Saraf:

Ooh. It really takes me back to the dark ages, and we always prepare for this, right? We have disaster recovery planned as part of our, uh, regular operations.

But even before I worked at Equinix, we had lost power to part of the data center, and we had to send someone physically out there in the dark, starting to hook up the right wires, bring the backup power back, and then we could do the work of restoring the software stack, which took 20 minutes. But you know, those were the longest 20 minutes of my life because it was truly testing our faith in our cohesion as a team, our communication as a team, but also the robustness and reliability of the systems we built.

Grace Ewura-Esi:

I understand that we work in an industry where things falling down can result in things falling apart, but I think it's okay for things to fall apart. I think that mistakes are where creation is born. And me personally, I think it's okay to have a little imperfection in the system, and that in of itself is why humanity should always maintain itself inside of the hardware so that we can really understand that it's okay for systems to be flawed just like we are.

Shweta Saraf:

Oftentimes, I'm working into the weeds and tinkering on the switches and the servers and so on. What I'm looking forward to is really taking a step back and imagining what if the things we rely on do not exist. Even if it's a brief moment, how can we use our technologically focused minds to dig us out of that situation and how would we react and what do we learn from it, right?

Grace Ewura-Esi:

Yeah. I think that that's actually the perfect backdrop for today's story. And instead of me telling it, I'm going to be passing off the mic to our producer John Taylor. Hey John.

John Taylor:

Hey Grace. Great to be here with you and Shweta. I've been doing a lot of thinking about this myself, especially this idea of the humanity and the hardware. What happens if you try to remove one from the other? What if you look down the stack and try to separate them? What does that mean? What happens if it's gone? There are, unfortunately, some tragic examples of what occurs when that happens. But, just as importantly, there are some redemptive stories as well, and the one that comes to mind the most is what happened on September 20th, 2017: Hurricane Maria. If you’ve never been in a Category 4 hurricane, it’s nearly impossible to describe what it’s like. The wind is a violent cacophony, with sustained winds up to 155 miles per hour. The rain hits so hard, it’s like being shot with a nail gun. And the storm surge can be up to a 9-foot wall of destruction, capable of wiping away anything in its path. Hurricane Maria had been building up this energy in the Atlantic for days, and when it finally made landfall in Puerto Rico, it only took a few hours before the power went out across the island. And then with the power out, the internet went down. Then cell towers fell and most cell phone service went out, followed by most landlines. TV stations went off the air and just like that, Puerto Rico was basically plunged back into the dark ages. But one piece of technology survived. One crucial piece of hardware infrastructure remained in the City of San Juan that allowed one woman and her team of volunteers to not only help restore civilization but really hope itself, to the island of Puerto Rico. Now, you might have a guess as to that one incredible piece of hardware so resilient it withstood the storm of the century. But hold that thought for a moment. If we really want to understand how to create technology that lasts, we need to take a little side trek first. A trek to the middle of nowhere. This is Moncure Pittsboro Road, a mile or so southwest of Pittsboro, North Carolina. Pittsboro is a small town, but it’s only about 15 miles from Chapel Hill. However, the entire stretch of road between Route one and sixty-four has really spotty service. In fact, it’s so hard to make a cell phone call on this road, I’m not even here. I mean, how could I be? There’s no cell service, no internet, no way for your modern, digital reporter to get the job done. And it’s in this kind of environment where a young graduate from the University of Washington, is experimenting with a new way to make complicated infrastructure simpler, more resilient, and more accessible.

Matt Johnson:

My name's Matt Johnson, and I'm currently working on communication networks and rural cell phone networks.

John Taylor:

I first heard about Matt when I read one of his papers about hardware obsolescence online. Matt has this interesting take on hardware, and it goes something like this.

Matt Johnson:

Really tight levels of integration, which lead to high performance often actually lead to making it more difficult to repair.

John Taylor:

So think of a cell phone. It gives us access to the entirety of human knowledge as well as worldwide communication in a box the size of a flat sardine can. Maybe you can get someone to fix the screen or change the battery, but opening that little thing up is unleashing a Pandora's box of transistor nightmares. So you just don't do it.

Matt Johnson:

Whenever you have a network system, you now are involving multiple components or multiple things that aren't under your direct control as the operator. And so definitely, it makes it harder to fix something when now you need to physically go to multiple locations or even interact with equipment that's owned by different entities.

John Taylor:

So now we know what doesn't lead to hardware longevity, items that are too small and integrated and things that are too massive and networked. Lesson learned, right? No.

Matt Johnson:

There's a great story about being able to stand on printers that you could just use as a step stool in the '90s. And people realize that, hey, these printers don't actually need to be that strong. They're way over-engineered. And so we can make the printers cheaper, lighter, and better in all of these other dimensions. So I think some of that comes in too where if we're really optimizing for cost, we are designing systems to perform the function they're designed for and not necessarily valuing reliability or durability as much as maybe was done in the past, either on purpose or just because we didn't know well enough to design systems right up to that limit.

John Taylor:

So we're making tech that can't be repaired and is optimized for cost. I think we're about to address the 800 pound server rack in the room.

Audio:

For the sake of profit, we build great factories to produce goods, and in so doing, help to build the industrial might of the nation.

John Taylor:

Look, it's no secret that we purposely build technology that's not meant to be repaired. It's meant to be used until it breaks and then we just toss it out or it's meant to be used until the next model comes along, a newer cooler model that gives us all kinds of FOMO. Is this a habit we can break from, a mindset that we can break from? Well, like I said, Matt's got a plan. Tell you, why don't we go back to Pittsboro for a moment. So, here we are, back on the roadside of our semi-imaginary MONCURE PITTSBORO ROAD, a few miles south of Pittsboro, North Carolina. I have the dreaded “No Service” message on my phone, which means I and all these farms and houses around me aren’t getting cell service. But good news: I see Matt Johnson coming this way in a pickup, and there’s something quite extraordinary in the bed of his truck. It’s a cell tower.

Matt Johnson:

The high level goal is to take cellular networking infrastructure, which is usually very centralized and looking at if we can break that infrastructure apart so that people in small remote areas could look at deploying their own small cell phone networks.

John Taylor:

Fascinating. I don't want to get too far into the weeds, but how does that work? How would you deploy your own small local cell phone network?

Matt Johnson:

Yeah, that's a great question. Cell phone networks in particular, have a history of using tightly licensed spectrum, so it's very difficult to get legal permission to use the parts of the radio spectrum that you need to operate. But all of that's actually changing and that there are these new paradigms about secondary use that allow normal people to actually deploy this cell phone networking hardware without needing a nation scale spectrum license from the FCC. So you can deploy these femtocells, which is just a small cell tower, that can provide coverage for the area around your house, and they hook into this CBRS system and get access to this spectrum kind of on demand via an API. And that's changing who could think about operating a cell phone network, particularly in one of these kind of disaster emergency situations?

John Taylor:

So, in other words, Matt and I are going to take this little cell tower over to a farmhouse and the two of us are going to install it on the roof. Then we'll meet with the owner of the house and over the course of a couple of beers, we'll explain exactly how to maintain the tower. By the end of the following day, we'll have taught the owner how to service the tower just in case it falls off the roof during a storm. And just like that, the community within a one square mile radius of this farmhouse has their own cell network. Right? Definitely cool. But perhaps the thing that stuck with me the most in my conversation with Matt was this concept that tech longevity isn't simply a matter of creating hardware that doesn't have tightly integrated systems. Just as importantly, there's an agency component.

Matt Johnson:

When you're talking about resiliency, this local agency piece is actually really important and I think is potentially as or more important than the technology itself. Feeling like you can attempt to fix something is necessary in order to attempt to fix it. And when you're talking about resiliency in the face of a disaster, it will take a long time for external resources to come in. And so if you're able to fix something locally and you have the agency and the tools to do that, then that will happen much more quickly.

Grace Ewura-Esi:

I love that concept. I love that idea of a community deciding these are the three people who are going to learn how to repair it. Here are the two people who are going to maybe fix any software updates or anything that it needs. Here are the two or three people who are going to clean it, just make sure our little cell tower stays clean, free of debris up and working, and then all 60 of us are going to benefit from this localized internet. And it makes me wonder if there are opportunities to do that in different ways across the stack. Can we actually break down component pieces like connectivity and allow people to opt in ways that make them feel safe and connected?

Shweta Saraf:

Yeah. I like the idea of it. I would love to live in a village like that. But I want to say it's hard, right? Because going back to what Grace was saying, there's so much years of innovation, infrastructure that went into allowing for disaggregation of this piece of technology and putting it into people's hand. But at the end of the day, it's hard. I think of we are at a point where internet has been around for a few decades now, but we haven't seen the commoditization of network. And that could be an important way of saying the infrastructure is there, the pipes are there, but the big companies don't get to decide who is on it or who's not. There is a network as a service kind of concept.

John Taylor:

Not only do I love this idea of a cell tower that you can fix on your own, but this concept that if it breaks down, you have the agency to fix it, and this gives you a whole new perspective on the hardware around you. Now you start looking at these things as fixable, like fixable for the average person. And Grace, it makes me wonder if maybe that sense of agency could spread to other larger, more complex hardware systems.

Grace Ewura-Esi:

I think that is a... How would I want to frame that? I think that that is a very privileged thought. I'm always asking myself, "What are the assumptions we made as we got here?" And we've made the assumption that everyone wants to use a computer and we have acted as if this is a contractual agreement, that everyone is very clear about what the harm and benefit associated with every element is. Is this the agreement we actually want to make?

Shweta Saraf:

When you are always willing to take from the world, from the technology, from the resources, from the people around you, what are you willing to give back? And how do we sort of create that technology which outlasts any of us or our future generation?

John Taylor:

Right. And I think that's a really important point. I think that when we talk about how to create technology that lasts, we're finding these two recurring themes, simplicity and agency. And you have some cultures that are way ahead on the agency piece, because they've been fixing their tech for generations. And if you look at technology from that reference point, now you're talking about legacy.

Shweta Saraf:

When you think about a legacy system, somehow in hardware or in technology, there's always like end of life. People are always planning for what is that lifespan of hardware where it produces the most optimal result and then it goes into this hardware junkyard, and how can you solve for that? And maybe the truth is you can't, you can't solve it beyond a certain extent and you have to acknowledge that the next thing then you think about is recycle or upcycle.

Grace Ewura-Esi:

I love technology. I wouldn't work in it if I didn't. I love the internet, I love the power of connection that this infrastructure gives us. But have we really considered what the trade-off is, and is everyone really in agreement with what we get from what we take?

John Taylor:

Is everyone really in agreement with what we get from what we take? This is such a great question. And for me, the answer is no, I'm not in agreement. For example, I don't think a laptop computer that lasts only three years is worth what we ask for in return from our society, our culture, and our resources. And I don't think anyone should agree to that. But if I'm understanding the bigger point here, it's that nobody asked. And perhaps, that's the biggest privilege of all. So using that same example, if I want my laptop to last longer than three years, we need to be thinking ahead of those three years. We need to be thinking in terms of legacy and we need to look at how we incorporate longevity into our thinking. So how far ahead should we be thinking, in order to substantially change how we view hardware longevity? Is it 10 years, 50 years? Well, according to the executive director of the Long Now Foundation, we should be looking at something closer to 10,000 years.

Alexander Rose:

I'm Alexander Rose and I'm the executive director at the Long Now Foundation, and I've been working for quite some time in building 10,000 year clock.

John Taylor:

Alexander Rose is a Founding Partner of the robotics company Inertia Labs. In the mid-90s, Alexander and a group of fellow technologists were witnessing what he calls Silicon Valley’s obsession with only doing things that could be done quickly. And it occurred to the group that there were many things that needed to work on a slower timescale than the direction that business and policy were headed. According to them, if civilization was ever going to solve the really big problems, we’d need to start thinking long-term. Like really long term, like generations. So the Long Now Foundation started all sorts of projects to promote this kind of thinking. They have a project called Long Bets where people can bet about things that might happen of scientific or social consequence in the future. They have a project to preserve language for thousands of years on micro etched metal discs. And then, there's The Clock of the Long Now.

Alexander Rose:

Danny Hillis, who had been building some of the fastest supercomputers in the world, at his company Thinking Machine, decided that maybe a kind of an icon to long-term thinking would be an interesting corrective or a way to tell stories about people taking the long-term more seriously. And so he decided to build in a way the slowest computer in the world, which became this 10,000-year clock project.

John Taylor:

Very interesting. So it's not just from a technology perspective, but rather all social and societal changes as well.

Alexander Rose:

Yeah, I think we need to be able to have a civilization wide conversation about any of these problems that are large worldwide problems and they're going to be solved over generations. If we can't allow ourselves to take on projects of that scale, then it's very difficult to take that kind of thing on and for that to be part of the conversation.

John Taylor:

So, this is about the coolest thing I’ve ever heard. And better still, one of the parts for the prototype for The Clock of The Long Now is displayed here in San Francisco, where I live. So I decided to convince our executive producer to come down to the city to check this thing out with me. And they were like, "An afternoon in San Francisco. Yeah, works for me." Hey Matt, how you doing?

Mathr de Leon:

Is this it?

John Taylor:

This is it, this is Fort Mason.

Mathr de Leon:

Awesome.

John Taylor:

So we met on one beautiful sunny afternoon at Fort Mason, which you can think of as like a mall with a twist. It was a huge sort of military installation. And just down this first row are a big line of shops and bars, and the one bar that we're going to is called The Interval. The Interval may actually be the quintessential hipster San Francisco coffee shop and bar. Stepping inside, the place looks like a library from some steampunk mad scientist's 1890s Victorian home. It's two stories tall with a spiral staircase up the middle. The walls are lined with bookshelves with various arts and artifacts in the spaces in between. But the feature you can't help but literally run into the moment you walk into this place is this, well-

Mathr de Leon:

Is this it right here?

John Taylor:

This is it right here. Again, this is a prototype of what's called the Orrery. Now, what we're looking at here is probably eight or nine feet tall. On the top of this is a series of intersecting circles and spheres, inside of which are models of the planets. You can see that these planets are fixed in rings that represent their orbits and the relative position of those orbits to each other. So that sits on top of a huge mechanical device, probably four feet tall by three feet wide. You'll see there's like six different sets of circular gears on the bottom. Each one of those represents the orbit of one of the planets represented here. Now, all of this may sound a bit complicated and overblown for a clock, but The Clock of the Long Now is no ordinary timekeeping device. It's a clock that's supposed to last 10,000 years and the finished hardware will be a bit bigger than the Orrery here at The Interval. In fact, it'll be built into the side of a mountain in the middle of the desert.

Alexander Rose:

We use the temperature difference from day to night that's kind of harvested at the top of this mountain to expand material that then gives us this both a mechanical signal that synchronizes the clock to solar noon, and then also gives the clock enough power to keep it knowing the time, like the pendulum and the escapement and a timekeeper moving along. So the clock always knows what time it is.

John Taylor:

They also realize that any metal to metal contact could have a welding effect after 10,000 years. So they use ceramic bearings developed for the space program in all the movable parts. But the success of The Clock of the Long Now isn't simply about finding the right engineering solutions, it's about building an institution around the clock. And that requires integrating a human component within the clock itself.

Alexander Rose:

How do you create a design language within that machine that compels people so they know where to interact with it and know where not to? And does the experience work for somebody 1,000 years from now? And of course, we can't know that for sure, but we can look back through time and see some of the things that we continue to think are beautiful and have continued to value and have cues as to how they're supposed to be interacted with.

John Taylor:

As Einstein said, imagination is more important than knowledge. So imagine this for a moment. It’s the year 2043, and the Clock of The Long Now has been functional and operating in this huge desert mountain for the last ten years. Occasionally, adventurers make the trek to see the clock, and each one takes turns reading the instructions and cranking the mechanism that resets the clock to today’s time. People love it. It’s a very unique and interactive experience. But now, imagine it’s four thousand years in the future. And like a Pharaoh’s sarcophagus, the Clock of The Long Now has suddenly been unearthed and rediscovered - a strange, mechanical thing lost in the desert for millennia. This structure is huge - 500 feet tall - and its marine-grade steel construction gives off a vibe that perhaps this thing should not be touched. There are dials on the structure with models of the planets, and in front of us is a tablet, with what appears to be ancient writing that’s indiscernible and mostly worn away. But in front of that tablet is a smaller mechanism that looks much more inviting. It’s made from a warm, bronze material, and its edges are soft and rounded. It speaks to us with a design language: a universal language that says, “this part would fit the palm of your hand.” And when we get up the courage to put our palm on that handle and crank it, bells sound and gears turn and this magnificent machine comes to life again. And the movement of it - the turning of those planet models and the echo of clicking gears - it speaks to us in that universal language of design and says, “hey, this thing is a clock.” And this is the biggest issue facing The Clock of the Long Now, facing any technology that we want to have last for a long time. It's not if the levers stop moving or the gears fused together. It's more like-

Alexander Rose:

How do you build a 10,000 year institution? No one's done that before. How do you make the experience compelling and interesting and storytelling worthy that it can become hopefully some kind of myth and legend in and of itself, even if it doesn't last that long?

John Taylor:

Stories. The success of The Clock of the Long Now doesn't depend exclusively on simplicity or resilience or reliability or even agency. It relies on this whole different kind of legacy, these stories that are created around the technology and then passed down through the generations.

Grace Ewura-Esi:

What a cool idea. And I think that what's really fascinating about this is it still has a really human component. And I think that inside of good technology, what we're finding is the answer to long-lasting 10,000 year tech are the humans, not the systems and not the machines.

Shweta Saraf:

I want to say that I 100% agree with the human connection and the human component to this. And I want to preface by saying, I'm much more of a futurist, right? I live for tomorrow. My ideas are more like what's going to happen next? But what I've come to acknowledge and learn is in order to plan for your future, you need to learn from your past. You can't just ignore that, right? And to me, what else is 10,000 years old and has still lasted us?

Grace Ewura-Esi:

Like what actually are we saying inside of the technology we're building today? And who is listening to those stories, internalizing those myths? Who is going to sing the song of technology today, tomorrow? John Taylor: And that makes me wonder, what technologies do we have right now that already have their own mythology? What hardware have we found to be so worthy of song and in our singing, have created longevity for that tech? And I thought about it for a while until the answer interrupted this recording session. This is my life. Yeah, hold on one second. For over a century, cars have captured our imagination. Ask any car owner and they'll give you a car story, everything ranging from adventures on the open road to my personal history of losing my virginity in the back of a 1972 Chevy G20 Sportvan. Now maybe that's TMI, but all good mythologies need sex, just ask the Greeks. And if oral history isn’t enough, one of the highest-grossing movie franchises of all time is a series of animated films about cars called…Cars. In fact, one of the top 10 highest grossing is a series called The Fast & The Furious, and if you’ve been living under a rock for the past two decades, they’re not movies about data centers. We’ve written books about cars, created TV shows about cars, and sang countless songs about cars. There’s even a band called The Cars, although you might argue they never wrote an actual song about cars. But Gary Newman did. He had a hit in 1979 with a song about cars called, Cars. It starts with engineering that's functional and elegant and ingenious. If we love what the hardware does and how it works and what it adds to our quality of life will not only maintain that technology, we will make it part of our communal story, our shared pathology, because the right technology is that important to us because there are some technologies we depend on for our very lives. And this takes us back to where our story began. Puerto Rico, September 20th, 2017: the day Hurricane Maria struck. The piece of tech that survived the storm had all these elements we discussed: compared to newer technology, you could say its hardware seems simple, the owner had the agency to repair it, and its mythology began over a hundred years ago. What was this critical piece of infrastructure? The AM radio transmitter. Sitting in the darkness as the winds pounded against her home, local journalist, Sandra Rodríguez, was glued to her radio, listening to one of the last stations on the air.

Sandra Rodríguez Cotto:

They were talking for about 24 hours straight, guessing a lot of things because they don't have any telephone lines to communicate to the government or to the authorities. And they were talking and talking and talking, and I knew that we're tired. So that night I said, "I'm going to go there. I'm going to prepare a bag of sandwiches and some coffee and I'm going to brought"... Because I thought they didn't have any food there. I went there just to help to see if they needed me to write the headlines or help them do research. And they said, "We don't have internet, we cannot even talk. We don't have any voice. Can you just talk?" And I'm like, "What am I going to say? I don't have the internet. I don't know what's going on."

John Taylor:

So they stick her in front of the mic and she just starts talking. And without internet access, she had no way of knowing what the community even needed during the crisis. So she sent out a call on the air for people to come to the radio station and share their experiences. Within hours, 70 people showed up. Broadcasters from other radio and TV stations, newspaper reporters, artists, politicians, psychiatrists, psychologists, and even doctors. And just like that, WAPA became a 24-hour news station.

Sandra Rodríguez Cotto:

The owner of the station basically said, "Sandra, do whatever you want because we have to keep this on the air. This is an emergency and that's what the radio is for. We need to save lives."

John Taylor:

Over the next couple of days, Sandra developed a system. Anyone could come to the station, stand in line, and then tell volunteer reporters the information they wanted to convey on the air. So for example, the presidents of all the major telecom networks came to the station and asked if employees could come in to help get the networks back online. But more importantly, people who had loved ones who were missing could come in and drop off names. And if that person was alive and heard their name on the air, they could come to the station to let their friends and family know that they were safe.

Sandra Rodríguez Cotto:

Some of the politicians would show up and start breaking in tears describing all the death, all the destruction that they faced. It was really emotional for all of us because we had never seen the impact like that. People that were completely lost and... Oh my God, I'm sorry that I'm getting emotional here. But it was hard. It's been five years and sometimes I don't want to open that door because I remember so many things that were really tough. But at the same time, it was a story of hope. Technology was a key role in saving lives at that moment.

John Taylor:

And how was all of this possible? Because the station owner knew how to repair the station, as well as other stations around the island. He had that agency. And I wonder if this is something developers and engineers think about.

Grace Ewura-Esi:

You know what my real question is? What happens if an extinction level event occurs today? Will our descendants know who we were based on the technology we have now? Software is not a long-lasting artifact because it is built upon a framework that has not yet become long-lasting. I think my big takeaway is that there are a lot of people working on this, and I think sometimes when I'm in my bubble of thoughts and I think about things like infrastructure, legacy, it can feel like there aren't enough people who are pondering our mortality and how to make these systems resilient. But then you share the stories of people coming from all disciplines, all fields, all walks of life, and honestly doing their own resilience practice and it makes me hopeful. I think that we, as a species, will figure it out before it's too late. And I do think that we will have a legacy that in 10,000 years will still exist.

John Taylor:

I think one of the problems we face when applying these longevity principles to huge pieces of infrastructure like data centers and server racks is that someone might say, first of all, they’re not simple, like Matt Johnson’s community cell networks are attempting to be. And they could also say there's no mythology around them, right? Like, we don't sing songs about data centers. But if you look down the stack for the humanity and the hardware and you find that the hardware doesn't work anymore, then what are you left with? You're left with humanity. Because what we ask from our technology is the same thing we ask from our human relationships, value me, be there for me, be reliable, take care of me, as opposed to say, buy me, use me, then toss me aside when you're done with me. And maybe it comes down to this idea that we will take care of data centers and increase longevity for servers the way the station owner in Puerto Rico took care of his AM radio transmitter. Because this is the way that we now spread our mythologies. Hardware is the tool, infrastructure is the tool we now use to preserve our mythologies and pass them along. If we want our story to carry on through time, we have to be worthy of stories. Like a clock that lasts 10,000 years… or a woman who helps save her community on that day when the lights go out.

Grace Ewura-Esi:

Traceroute is a podcast from Equinix and Stories Bureau. This episode was produced by John Taylor, with help from Tim Balint and Cat Bagsic. It was edited by Joshua Ramsey and mixed by Jeremy Tuttle, with additional editing and sound design by Mathr de Leon. Our theme song was composed by Ty Gibbons. You can check us out on Twitter at Origins underscore dev — that’s D-E-V. And type Origins dot dev into your browser for even more stories about the human layer of the stack. We’ll leave these links and more, including an episode transcript, down in the show notes. If you enjoyed this story, please share it wherever you hang out online and consider leaving a five-star rating on Apple and Spotify, because it really helps other people find the show. Traceroute will be back in two weeks. Until then, I’m Grace Ewura-Esi. Thanks for listening.