[Wiggins]: For me, thinking about that word longevity, I tend to think about what is long, I guess. What is a span of time that counts as long and of course it depends a lot on what you're talking about. So if you're talking about all of society, culture, humanity, then perhaps you’re talking in the thousands of years, hundreds or thousands of years. So for example, the long now is a phrase that we use to the local-first paper, we tend to use it internally on our team to refer to longer-term. Thinking this comes from the Long Now Foundation, and the long now clock project, which is sort of this idea to build a clock that can keep time for 10,000 years and it's basically an art project, but it's designed to inspire us to think longer term, and that can obviously connect to things about climate change or human culture. And since humans are naturally inclined to think probably pretty short term, actually really short term: We think about the day that's ahead or the week that's ahead of us, maybe at most the year that's ahead of us, we don't tend to think in tens or hundreds or thousands of years. That's maybe a society level thing. I think for individuals and thinking about, you know, software as it impacts our lives as individuals, I think a human lifespan is actually is a pretty good chunk of time to compare something to. Wiggins and McGranaghan, Metamuse Episode 49, Software Longevity, 05:06, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=8Ny-UxkNStWnvjGA1EnO-A
Data and rolling it over to newer versions
[…] another thing would be preserving history, having access to the past. I think this is especially important with software, it's very easy to lose. that both in the sense of the software itself and the data that it was manipulating. McGranaghan, Metamuse Episode 49, Software Longevity, 08:14 , https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=NrliJcWGQsWYY4NkOqTnYA
[McGranaghan]: I think there's a dimension of longevity which is, and I am going to make up a word here, the rollover ability to buy or get a new version of something that rolls over your previous state. So, for example, maybe you re-bind a book and roll over the pages or you can put a new engine and transmission in a truck and you roll over the truck frame. But some stuff can't be rolled-over: classically, the black-box data, just some opaque binary format. So I think that ability to roll over is an important dimension even if it’s separate from a single instance having longevity. [Wiggins]: Information has this particular property that exists on a substrate or a medium of something physical - atoms - but in the end, the information is not the physical thing. So the book rebinding example, you can either rebind the same book or in some cases just transcribe it completely. And as long as it's a correct and accurate copy, older books are an incredible artifact, maybe they have margin, something about how the pages were made or whatever does carry some history, there were the artifacts and objects of their own right, but ultimately the contents, the words on the page are probably what we care about. So, on one hand information technology has actually been getting worse in the sense of the durability of the underlying thing: Stone tablets were very durable. Papyrus and later paper were less. So then we go into the digital world and it's actually vastly less. Everything from CD-Rs to cloud storage, the failure rate is pretty high but because it is so easy cheap to copy those bits from one place to another to replicate it. You potentially have the ability to roll it forward as far as it's worthwhile to do. McGranaghan, Metamuse Episode 49, Software Longevity, 09:05, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=zDGVEtNTR_KHIINVW-HrKQ
[McGranaghan]: Another thing that I'm very big on is really focusing on data, often, especially in this world of rolling forward over multiple generations, what you really want to keep as the data. This is more true for productivity-like stuff than games for example, but data is very important. And, if you want to roll forward data, you either need to adopt an existing open data format or you need to do your own, which can be done, but it's a lot of work. I think that programs or program lineages, if you call that, embrace this idea of really focusing on the data so that that can always be ejected and rolled forward into the next generation of software. [...] One of my favorite examples is SQLite. If you use SQLite to write even your semi-custom data format, if you write it into SQLite database, you're absolutely gonna be able to read that in 30 years, even if you do nothing yourself. Whereas, if you roll your own format on disk, there's a very good chance that it will at some point become totally lost. [Wiggins]: Yeah, SQLite is a favorite example […] of well-made simple software that does one thing and does it well. They have a great page though about their long-term support where they list off some techniques, like, testing and making their database files cross-platform and disaster planning and not embracing hot new technologies with too much gusto. McGranaghan & Wiggins, Metamuse Episode 49, Software Longevity, 27:05, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=7WHCqDsQRaK-HaTeS4HQVA
[McGranaghan]: […] that circles back to this idea of data. I see people typically design and implement programs […] we tend to focus on the interface and it behavior because that's what users are demanding because that's what the short-term success of our business depends on. But the reality is that interface, including its implementation, is going to almost certainly fully churn at least once potentially several times over the course of software. But what's not going away is the data. And so, one thing that I think is helpful for longevity of software is really focusing on the underlying data model. Not only in the sense of how it's stored, which is mostly what we've been talking about in the podcast so far in terms of text files or SQLite, but also what is the data model itself. Like if you were to draw out the SQL tables, or equivalent, draw the boxes and put labels on them, it sounds really basic but it's a step that I feel like people often skip over and then it leads to all kinds of longevity headaches down the road. Future-proofing I think is a word we sometimes use in general for talking about helping things be longer-lived, but certainly I think that's most notable in the data model, precisely for the reason you said: it's just much less changeable and harder to change and heavier weight to change. McGranaghan & Wiggins, Metamuse Episode 49, Software Longevity, 31:20, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=2S81RLe9Q4mA36etUZN7QA
[Wiggins] The version thing is also making me think of, I think it’s the image file format, maybe tiff is the one I'm thinking of, but this approach has been used in a lot of different places, and I feel like you're doing a version of this in the local-first sync infrastructure we’re working on right now. And so you can add new chunks to a stream of data or to a file that older versions of the software can load, recognize they don't know this chunk and they can skip past it and just try to interpret the rest of it the best they can. Maybe this is something the web browsers do pretty well which is, you load a newer page in an older web browser, if there's some not supported features it doesn't completely break, it just does its best to render it. […] It's not that it just gives up the moment it sees something it doesn't understand. [McGranaghan]: And this gets into the peculiar challenges of longevity with distributed system software. We've been talking about longevity in terms of supporting the past, but with distributed systems you gotta support the future because the future arrives unevenly. So that's the sort of challenge you deal with in your working on distributed systems like the data synchronization layer from Muse. [Wiggins]: As usual Ink & Switch has some good research on this subject with the data lenses in the Cambria project. One of the things there is about data migrations, but it's assuming you need to translate both ways. You need to translate back in time the older versions and forward in time to newer versions, but in fact, it's actually more complex than that, because there may be just a branching tree of systems that all understand that the data in slightly different ways. Wiggins & McGranaghan, Metamuse Episode 49, Software Longevity, 34:55, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=A2dedJMzR4KhKIgM8COa2g
[Wiggins]: And the other small thing, it's sort of obvious in some ways, it's kind of the Lindy Effect of file formats. So, Lindy effect here being sort of the idea that something that’s been around a long time probably will be around a long time. So, I'm always a big fan of those flat file formats, PNG, JPEG, PDF, plain text, because they've proven themselves to be durable over the long term and they don't always do everything you need but where you can, it's really nice to just go down to that simple common format that is understood by many applications both now and in the past and potentially will be in the future. Wiggins & McGranaghan, Metamuse Episode 49, Software Longevity, 36:41, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=ozNX-91xRhKag6u_febOKw
[Wiggins]: Now, some have argued that perhaps part of the challenge here is the rapid pace of change of that underlying system of those APIs. I think it's interesting here to contrast the Apple versus Microsoft approach. So I think that Microsoft with DOS and later Windows was really focused on backwards compatibility quite in an impressive level. I'm not exactly sure when the cut-offs are, but I think you could run DOS stuff natively on windows till well into the Windows lineage, and much later versions of Windows could run much earlier versions. But that's part of, I think, what also created the relative instability and unreliability of that line of operating systems, which is when you need to support every conceivable thing that's ever existed in the past and put it all together in the same box, and things get messy pretty fast. Apple with the iOS world has gone very much the other route which is they basically are releasing new stuff, a new OS update, once a year and if app developers don't keep up-to-date you pretty quickly just fall out of the App Store. […] You have this very demanding operating system that requires developers be doing active maintenance, or else it [your app] tends to go away. [McGranaghan]: While I don't doubt that there was some contribution in the Windows situation to instability from needing to support all versions forever, I think what really gets you is when you need to do that, and things that are really end-user-facing, like visual in particular, because the strategies that you typically use for providing stability are not viable in that world. So, if you look at things like Unix-style server, operating systems, many programming languages, these are things that can have backwards compatibility for a long time, certainly decades. But the way they do that is they just never ever, ever ever break any APIs. And if you want to do something new, either too bad, or call it "thing 2” and just keep “thing 1” around forever. […] you can't really do that with a windowing system, for example, or a user interface paradigm or how things look on the screen, right? You just got to pick something and do it, and the old thing needs to go. And so I think it does become very hard to have that culture of very long-lived durability that you see in those systems-y programs. [Wiggins]: Yeah, maybe that points to the fundamental trade-off there, which is durability in many cases means stability, which means less change. But change is how we get things that are new and better and the world technology and computers in the internet is so at the very beginning, there's so much unexplored space, and it would be a shame if we just said, okay, well we've been working on making these computers here for a few decades now, this is pretty much how they are, probably it is probably good for the next several hundred years, let's just hold it there. That's not what we want to do. We want to keep changing and improving. So yeah, there just is a trade-off between those two. [McGranaghan]: Yeah, I think there's a very real tension and I agree want to keep exploring. I do think that in adopting this attitude of exploring and taking risks, we are going to find some things that in retrospect were not good choices. The intuition here is, I think we're projecting these very dynamic system benefits over lives that we have historically associated with durable goods. So we get the shiny new computer-enabled, software-enabled thing, and it works great the first year, and we think: things like this, before we had computers, they worked for 20 or 30 years, therefore, this thing is going to work for 20 or 30 years, it's gonna be awesome. And we've kind of projected that out for, and I think in many cases that's not going to be true. And so we are going to have to confront the downside of that tradeoff. [Wiggins]: And do you think the downside there is mainly economic? You buy the smart refrigerator seems great for the first year, some cloud service goes offline suddenly and your refrigerator doesn't work anymore and you got to throw it out and get a new one, or do you think there's something beyond the economic side? [McGranaghan]: Unfortunately I think there are several potential very serious downsides. Just to give you a couple, one that we're already seeing is this issue of repairability or even modifiability. Again, if you look at these durable goods categories like tractors, which is very prominent example right now, traditionally people could service and maintain and change and modify and improve their own tractors. But as tractors increasingly become like iPhones, you know, these black-box computers in many ways, or highly computerized hardware, Potentially you can’t service them yourself, you can't repair them yourself, you can't modify them yourself and that has all kinds of second order effects. There's also this whole thing about surveillance basically that I think we're sleeping on a little bit, but we do need to go down that rabbit hole today. [Wiggins]: On the economic side, I think there's all sort of the incentive which is basically Apple and other phone manufacturers have done very very well by always creating that shiny new thing that comes out every year or two. And you think: I want to upgrade. It’s even often built into cell phone contracts and things, people want to get a new phone very frequently. And I think that's paired with sometimes the operating system upgrades, or maybe the visual refreshes, it's almost more like fashion, the fashion industry found this very clever way to get around the fact their IP isn't really protectable, which is they just make new fashion trends the totally change for a couple of years. So then you like need to go out and get a new wardrobe. It's not the clothes wore out necessarily. It's that you want to be up-to-date and you look behind the times when you’re strolling around in your bell bottoms and those have been out of fashion for three years or whatever it is. And so I think there's an element of this kind of a fashion or desire to get the new thing that has driven a lot of revenue for these companies. I don't mean to imply that that some kind of a cackling in the back layer, they're using this opportunity to improve and genuinely make their products better: Each version of the iPhone has that better camera, that better battery life, the bigger brighter screen, it is genuinely better, but it's also the companies are very incentivized for a high pace of change that has you buying new products all the time, as opposed to something longer-term. Wiggins and McGranaghan, Metamuse Episode 49, Software Longevity, 18:33 , https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=euUyYhr6SWC4C0KFtLmMHQ
Software environment and dependencies
[Wiggins]: Even thinking of a binary artifact, you've got a compiled executable, you can download it, you don't need to the dependencies, you don't need the compiler chain and I think there's the feeling that “shouldn’t that work forever?”. But of course as you said it sits within this context of the system: if it's accessing files on your file system, it's calling out to APIs on the network, it's accessing APIs in the operating system. Even something like it just makes assumptions about what kind of hardware exists, the keyboard and mouse has being standard input devices that exist and then you go onto a touch device and those aren't there. So over time, those fundamental assumptions and apis and the system changes around it and I think that's gotten dramatically more so with the internet, where you might call out to even something as simple as a static website, that has a Google Maps embed, well, now a big portion of that website is dependent on this exact integration to the service and that service is still online. So I think that the one-off binary build, shouldn’t-it-work-forever does make some sense in, for example, games, which don't have as many hooks (let's say just like a single player game, not some complicated cooperative thing) as it tends to have very simple inputs and outputs, displays things on the screen, maybe it needs to save a high-score file to the disc, as opposed to productivity software where all those little integration points: you drag and drop and how do you open a thing in the browser and all the different apis that are used to be a good citizen on the system, and those change all the time - they should with an operating system that's growing and improving and evolving - but then that means the apps have to keep up with that. And if they don't, they fall out of date. And eventually either become somewhat irrelevant or more commonly just stop working. [McGranaghan]: Yeah, I think this idea of the complexity of the surface area and the APIs is very important. You mentioned games: I think we actually had it much easier back in the good old days of games where we had keyboard input, mouse input, you draw to a pixel buffer, and that's it. The issue with games now, in addition to stuff around networking and multiplayer is that the graphics pipeline is incredibly complex, and unless you end up emulating earlier versions of graphics pipeline there's little chance this stuff rolls forward, whereas you could plausibly roll forward writing into a pixel bus offering. You kind of go the other way - you can emulate a pixel buffer with our advanced graphics APIs today, but there’s no chance of going the other direction. I think this generalizes to API complexity broadly and it's one of the reasons why it's so hard for software to persist Wiggins and McGranaghan, Metamuse Episode 49, Software Longevity, 13:12, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=ST_DRl-YQ3W4nxTMYYOD3w
[Wiggins]: Now another one from games that maybe is your roll-forward idea, I really like what id Software does, which is they take their classics - Doom, Quake and so on - And after I don't know if they're 20, 30 years old, something like that, they open source of them, and that creates this interesting effect where maybe it's less that people are playing the original, although I'm sure that happens, but then it becomes fun to just port it to weird places. Like you've seen Quake that's been ported to be 100% ASCII. I also saw a Twitter thread recently where someone managed to open one of these digital pregnancy tests, realize that it had a reasonably good processor and display, and decided to get Doom running on it. […] So in a way that has turned this over to remix culture and the internet and so it gets rolled forward and become something that maybe people will experience not in its original form necessarily, but becomes woven into the cultural tapestry let’s say through these fun little projects. [McGranaghan]: I think this also speaks to the importance of that API environment they were talking about earlier being specced or at least speccable for the software to be rolled forward or actively preserved. […] Old-timey games, that's a pretty speccable environment, like needing to be able to compile C, and relatively straightforward input and output system, and even if the games weren't designed specifically with a really clean interface there you can basically back it out after the the fact, and then do the porting. Books and paintings, by the way, are examples of analog things that are very speccable: they map down to text streams and bitmaps respectfully. But stuff like modern software is much more difficulty being specced, and so it becomes harder to do that sort of rolling forward act of preservation. Wiggins and McGranaghan, Metamuse Episode 49, Software Longevity, 16:40, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=JYEnhTUbSPCutemfh9wI-A
One thing we've alluded to is idea of narrow-defined APIs, because your software has to run in something, and the broader, the more complex, the more ill-specified that environment is, the harder it's going to be to do this job of keeping the software running. And this would include things like taking on few dependencies, minimizing weird binary dependencies that are hard to compile, limiting the APIs that you access, things like that. McGranaghan & Wiggins, Metamuse Episode 49, Software Longevity, 26:39, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=-2-SDcZXSUaI6iFWTO7eDg
The constituent parts of software longevity. Software as a special class of information
[Wiggins]: If we come to software as a special class of information, and it is information but it's highly dynamic, and I think that points to some of the particular challenges with it in […] software itself and its ability to be long-lived. And I think some of this is cultural, the tech industry is dynamic. It thrives on change, that's sort of the nature of it and many things that are good about software and the internet and the tech world come from that willingness to embrace the new and almost an endless seeking of novelty. But the downside is, you get something that is not just an upgrade treadmill but it just seems like it can be very much the case that the lifetime of product or of data or any particular piece of software can be measured in years, and only a small number of years, which is comparatively just a very, very short time span. […] But I feel there's more to it than just cultural, the tech world doesn't do the buy-it-for-life / built-to-last thing. I feel like there's something inherent to the dynamic-information nature of software, that makes it hard for it to be long-lived. [McGranaghan]: Yes, I think that's true. There's a lot of things that need to go right for software to work, and you can see it, in fact, as something that's strictly harder than preserving simple textual information for the following reason: You have at least the source code, then you need to preserve information about the dependencies. You potentially have the compiled source and then you have the whole runtime around it, very broadly defined as the computer and the APIs, maybe even the data. And this is just a very broad, multi-dimensional, and many cases, ill-specified state space: if you don't get your thing, exactly right in that state space, it doesn't work at all. Which, by the way, is something that's a little bit unique to digital systems versus classic analog systems. If your wrench is Rusty, you know, doesn't work as well, but it's still basically works as a wrench. Whereas, if you have one bit off in your software it just crashes - you're out of luck. It's very high-dimensional and it's very sensitive to errors in preserving the environment. Wiggins and McGranaghan, Metamuse Episode 49, Software Longevity, 10:52, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=u0GQfL1US5ujSnD0yl1Jsw
[Wiggins]: First I have to ask what it means to you to talk about something being long-lived or having longevity, whether its software or a plane or something else. [McGranaghan]: I think it's the software being able to serve its stated purpose, which sounds very straightforward, but has several important constituent parts: - First of all, you actually need to software, you know, sometimes we just lose it. - It needs to run and needs to run correctly and - it needs to have access to the relevant data. - It needs to have access to relevant pieces of the outside world in terms of APIs and - it needs the appropriate substrate to run on an interact with. - And it's probably other pieces that we could come up with. McGranaghan and Wiggins, Metamuse Episode 49, Software Longevity, 04:20, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=ca3gfTLARQ2EExDeBeQ5yA
[Wiggins]: Even thinking of a binary artifact, you've got a compiled executable, you can download it, you don't need to the dependencies, you don't need the compiler chain and I think there's the feeling that “shouldn’t that work forever?”. But of course as you said it sits within this context of the system: if it's accessing files on your file system, it's calling out to APIs on the network, it's accessing APIs in the operating system. Even something like it just makes assumptions about what kind of hardware exists, the keyboard and mouse has being standard input devices that exist and then you go onto a touch device and those aren't there. So over time, those fundamental assumptions and apis and the system changes around it and I think that's gotten dramatically more so with the internet, where you might call out to even something as simple as a static website, that has a Google Maps embed, well, now a big portion of that website is dependent on this exact integration to the service and that service is still online. So I think that the one-off binary build, shouldn’t-it-work-forever does make some sense in, for example, games, which don't have as many hooks (let's say just like a single player game, not some complicated cooperative thing) as it tends to have very simple inputs and outputs, displays things on the screen, maybe it needs to save a high-score file to the disc, as opposed to productivity software where all those little integration points: you drag and drop and how do you open a thing in the browser and all the different apis that are used to be a good citizen on the system, and those change all the time - they should with an operating system that's growing and improving and evolving - but then that means the apps have to keep up with that. And if they don't, they fall out of date. And eventually either become somewhat irrelevant or more commonly just stop working. [McGranaghan]: Yeah, I think this idea of the complexity of the surface area and the APIs is very important. You mentioned games: I think we actually had it much easier back in the good old days of games where we had keyboard input, mouse input, you draw to a pixel buffer, and that's it. The issue with games now, in addition to stuff around networking and multiplayer is that the graphics pipeline is incredibly complex, and unless you end up emulating earlier versions of graphics pipeline there's little chance this stuff rolls forward, whereas you could plausibly roll forward writing into a pixel bus offering. You kind of go the other way - you can emulate a pixel buffer with our advanced graphics APIs today, but there’s no chance of going the other direction. I think this generalizes to API complexity broadly and it's one of the reasons why it's so hard for software to persist Wiggins and McGranaghan, Metamuse Episode 49, Software Longevity, 13:12, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=ST_DRl-YQ3W4nxTMYYOD3w
[Wiggins] The version thing is also making me think of, I think it’s the image file format, maybe tiff is the one I'm thinking of, but this approach has been used in a lot of different places, and I feel like you're doing a version of this in the local-first sync infrastructure we’re working on right now. And so you can add new chunks to a stream of data or to a file that older versions of the software can load, recognize they don't know this chunk and they can skip past it and just try to interpret the rest of it the best they can. Maybe this is something the web browsers do pretty well which is, you load a newer page in an older web browser, if there's some not supported features it doesn't completely break, it just does its best to render it. […] It's not that it just gives up the moment it sees something it doesn't understand. [McGranaghan]: And this gets into the peculiar challenges of longevity with distributed system software. We've been talking about longevity in terms of supporting the past, but with distributed systems you gotta support the future because the future arrives unevenly. So that's the sort of challenge you deal with in your working on distributed systems like the data synchronization layer from Muse. [Wiggins]: As usual Ink & Switch has some good research on this subject with the data lenses in the Cambria project. One of the things there is about data migrations, but it's assuming you need to translate both ways. You need to translate back in time the older versions and forward in time to newer versions, but in fact, it's actually more complex than that, because there may be just a branching tree of systems that all understand that the data in slightly different ways. Wiggins & McGranaghan, Metamuse Episode 49, Software Longevity, 34:55, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=A2dedJMzR4KhKIgM8COa2g
Preserving games as a form of cultural history preservation
[Wiggins]: Now another one from games that maybe is your roll-forward idea, I really like what id Software does, which is they take their classics - Doom, Quake and so on - And after I don't know if they're 20, 30 years old, something like that, they open source of them, and that creates this interesting effect where maybe it's less that people are playing the original, although I'm sure that happens, but then it becomes fun to just port it to weird places. Like you've seen Quake that's been ported to be 100% ASCII. I also saw a Twitter thread recently where someone managed to open one of these digital pregnancy tests, realize that it had a reasonably good processor and display, and decided to get Doom running on it. […] So in a way that has turned this over to remix culture and the internet and so it gets rolled forward and become something that maybe people will experience not in its original form necessarily, but becomes woven into the cultural tapestry let’s say through these fun little projects. [McGranaghan]: I think this also speaks to the importance of that API environment they were talking about earlier being specced or at least speccable for the software to be rolled forward or actively preserved. […] Old-timey games, that's a pretty speccable environment, like needing to be able to compile C, and relatively straightforward input and output system, and even if the games weren't designed specifically with a really clean interface there you can basically back it out after the the fact, and then do the porting. Books and paintings, by the way, are examples of analog things that are very speccable: they map down to text streams and bitmaps respectfully. But stuff like modern software is much more difficulty being specced, and so it becomes harder to do that sort of rolling forward act of preservation. Wiggins and McGranaghan, Metamuse Episode 49, Software Longevity, 16:40, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=JYEnhTUbSPCutemfh9wI-A
[Wiggins]: Games also make me think of […] emulators and it turns out that games seem to be something that people do want to preserve as a cultural history, but in a way, the best way to do that is not try to get the original hardware but to have these emulators like MAME for example, and it's really impressive: you can play almost every arcade cabinet video game from the 80s, just in your browser with this kind of JavaScript emulators that will load up the ROMs, which are basically the way they distributed binaries in those days and play them, not quite as they were intended because the hardware is different, certainly displays are different, but certainly we do preserve some of that legacy. Wiggins and McGranaghan, Metamuse Episode 49, Software Longevity, 15:54, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=ohDy4dTAThWNh92mQP1ENA
Techniques for achieving software longevity
[McGranaghan]: You gotta sit down every week for 30 years and make sure the software doesn't break, and that's sort of a necessary but not sufficient condition. [Wiggins]: And the implication there is just maintenance, right? And that maybe ties into some of our discussions about software supported by subscriptions versus one-off payments, which is really is an ongoing effort, important ongoing maintenance effort, rather than a one and done. McGranaghan & Wiggins, Metamuse Episode 49, Software Longevity, 26:05, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=-2-SDcZXSUaI6iFWTO7eDg
One thing we've alluded to is idea of narrow-defined APIs, because your software has to run in something, and the broader, the more complex, the more ill-specified that environment is, the harder it's going to be to do this job of keeping the software running. And this would include things like taking on few dependencies, minimizing weird binary dependencies that are hard to compile, limiting the APIs that you access, things like that. McGranaghan & Wiggins, Metamuse Episode 49, Software Longevity, 26:39, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=-2-SDcZXSUaI6iFWTO7eDg
[McGranaghan]: Another thing that I'm very big on is really focusing on data, often, especially in this world of rolling forward over multiple generations, what you really want to keep as the data. This is more true for productivity-like stuff than games for example, but data is very important. And, if you want to roll forward data, you either need to adopt an existing open data format or you need to do your own, which can be done, but it's a lot of work. I think that programs or program lineages, if you call that, embrace this idea of really focusing on the data so that that can always be ejected and rolled forward into the next generation of software. [...] One of my favorite examples is SQLite. If you use SQLite to write even your semi-custom data format, if you write it into SQLite database, you're absolutely gonna be able to read that in 30 years, even if you do nothing yourself. Whereas, if you roll your own format on disk, there's a very good chance that it will at some point become totally lost. [Wiggins]: Yeah, SQLite is a favorite example […] of well-made simple software that does one thing and does it well. They have a great page though about their long-term support where they list off some techniques, like, testing and making their database files cross-platform and disaster planning and not embracing hot new technologies with too much gusto. McGranaghan & Wiggins, Metamuse Episode 49, Software Longevity, 27:05, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=7WHCqDsQRaK-HaTeS4HQVA
[Wiggins] […] one of the things that points to that to me feels like maybe the most fundamental is to just think about longevity in the first place and just care about it in the first place. Part of the reason I thought this topic would be an interesting one.I think, again, tech industry skews (?) young, I think also had the same gap in my thinking as a younger person, which was I just didn't have the longer experience to be able to see what it is like for software that I've created that's in production for three years, five years, ten years, and longer and what happens over time. It's a part of what I like with the SQLite long-term support page, they basically lead with a statement that our goal is to support this until the year 2050 and so, will they achieve that? hard to know… But just by making that statement, calling it out as a value for themselves and thinking about it actively, maybe this comes back to the long-now clock idea, which is, they may or may not succeed but certainly one good way to not succeed is to not think about it in the first place. And so this makes it a first-class concern. Wiggins & McGranaghan, Metamuse Episode 49, Software Longevity, 30:14, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=UT2qfBJAQWWjr5QsUGkunw
[McGranaghan]: […] that circles back to this idea of data. I see people typically design and implement programs […] we tend to focus on the interface and it behavior because that's what users are demanding because that's what the short-term success of our business depends on. But the reality is that interface, including its implementation, is going to almost certainly fully churn at least once potentially several times over the course of software. But what's not going away is the data. And so, one thing that I think is helpful for longevity of software is really focusing on the underlying data model. Not only in the sense of how it's stored, which is mostly what we've been talking about in the podcast so far in terms of text files or SQLite, but also what is the data model itself. Like if you were to draw out the SQL tables, or equivalent, draw the boxes and put labels on them, it sounds really basic but it's a step that I feel like people often skip over and then it leads to all kinds of longevity headaches down the road. Future-proofing I think is a word we sometimes use in general for talking about helping things be longer-lived, but certainly I think that's most notable in the data model, precisely for the reason you said: it's just much less changeable and harder to change and heavier weight to change. McGranaghan & Wiggins, Metamuse Episode 49, Software Longevity, 31:20, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=2S81RLe9Q4mA36etUZN7QA
[Wiggins] The version thing is also making me think of, I think it’s the image file format, maybe tiff is the one I'm thinking of, but this approach has been used in a lot of different places, and I feel like you're doing a version of this in the local-first sync infrastructure we’re working on right now. And so you can add new chunks to a stream of data or to a file that older versions of the software can load, recognize they don't know this chunk and they can skip past it and just try to interpret the rest of it the best they can. Maybe this is something the web browsers do pretty well which is, you load a newer page in an older web browser, if there's some not supported features it doesn't completely break, it just does its best to render it. […] It's not that it just gives up the moment it sees something it doesn't understand. [McGranaghan]: And this gets into the peculiar challenges of longevity with distributed system software. We've been talking about longevity in terms of supporting the past, but with distributed systems you gotta support the future because the future arrives unevenly. So that's the sort of challenge you deal with in your working on distributed systems like the data synchronization layer from Muse. [Wiggins]: As usual Ink & Switch has some good research on this subject with the data lenses in the Cambria project. One of the things there is about data migrations, but it's assuming you need to translate both ways. You need to translate back in time the older versions and forward in time to newer versions, but in fact, it's actually more complex than that, because there may be just a branching tree of systems that all understand that the data in slightly different ways. Wiggins & McGranaghan, Metamuse Episode 49, Software Longevity, 34:55, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=A2dedJMzR4KhKIgM8COa2g
[Wiggins]: […] For archival purposes I have put a lot of effort in the past to preserving things I've created, whether it's a essay, or back when I used to make music, […] usually I find the kind of flattened artifact, read-only, is really kind of what I want in the long term because I do go back to reference those things or look at them for inspiration or just for take a walk down memory lane, but I'm very rarely want to edit, even if I could, and so in that sense just it's actually superior to flatten out to an image or a video or something. [McGranaghan]: Yeah, it's a good point, in many ways, it's a benefit, not a downside, especially when you have this habit of building up an archive that's all on these handful of formats. Everything is basically text, PNG, PDF, maybe video, MP3, you can browse it all together and you can do it very quickly, basically, in Finder or the equivalent, whereas you can imagine, if for every picture you want to check out you had to open up Adobe Photoshop or whatever, you’d wait 2 minutes, so it's nice to have that very lightweight, archival copy. Wiggins & McGranaghan, Metamuse Episode 49, Software Longevity, 38:21, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=ttBllaQQTES8dUyfX6kGMA
[Wiggins]: And the other small thing, it's sort of obvious in some ways, it's kind of the Lindy Effect of file formats. So, Lindy effect here being sort of the idea that something that’s been around a long time probably will be around a long time. So, I'm always a big fan of those flat file formats, PNG, JPEG, PDF, plain text, because they've proven themselves to be durable over the long term and they don't always do everything you need but where you can, it's really nice to just go down to that simple common format that is understood by many applications both now and in the past and potentially will be in the future. Wiggins & McGranaghan, Metamuse Episode 49, Software Longevity, 36:41, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=ozNX-91xRhKag6u_febOKw
In the fullness of time, the files you create are more important than the tools you use to create them. Apps are ephemeral, but your files have a chance to last. The ancient temples of Egypt contain hieroglyphs that were chiseled in stone thousands of years ago. The ideas hieroglyphs convey are more important than the type of chisel that was used to carve them. The world is filled with ideas from generations past, transmitted through many mediums, from clay tablets to manuscripts, paintings, sculptures, and tapestries. These artifacts are objects that you can touch, hold, own, store, preserve, and look at. To read something written on paper all you need is eyeballs. Today, we are creating innumerable digital artifacts, but most of these artifacts are out of our control. They are stored on servers, in databases, gated behind an internet connection, and login to a cloud service. Even the files on your hard drive use proprietary formats that make them incompatible with older systems and other tools. […] If you want your writing to still be readable on a computer from the 2060s or 2160s, it’s important that your notes can be read on a computer from the 1960s. You should want the files you create to be durable, not only for posterity, but also for your future self. You never know when you might want to go back to something you created years or decades ago. Don’t lock your data into a format you can’t retrieve. These days I write using an app I help make called Obsidian, but it’s a delusion to think it will last forever. The app will eventually become obsolete. It’s the plain text files I create that are designed to last. Who knows if anyone will want to read them besides me, but future me is enough of an audience to make it worthwhile. Steph Ango, File over App, https://stephango.com/file-over-app
[Wiggins] […] one of the things that points to that to me feels like maybe the most fundamental is to just think about longevity in the first place and just care about it in the first place. Part of the reason I thought this topic would be an interesting one.I think, again, tech industry skews (?) young, I think also had the same gap in my thinking as a younger person, which was I just didn't have the longer experience to be able to see what it is like for software that I've created that's in production for three years, five years, ten years, and longer and what happens over time. It's a part of what I like with the SQLite long-term support page, they basically lead with a statement that our goal is to support this until the year 2050 and so, will they achieve that? hard to know… But just by making that statement, calling it out as a value for themselves and thinking about it actively, maybe this comes back to the long-now clock idea, which is, they may or may not succeed but certainly one good way to not succeed is to not think about it in the first place. And so this makes it a first-class concern. Wiggins & McGranaghan, Metamuse Episode 49, Software Longevity, 30:14, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=UT2qfBJAQWWjr5QsUGkunw
Local-first and software longevity
An important aspect of data ownership is that you can continue accessing the data for a long time in the future. When you do some work with local-first software, your work should continue to be accessible indefinitely, even after the company that produced the software is gone. “Old-fashioned” apps continue to work forever, as long as you have a copy of the data and some way of running the software. Even if the software author goes bust, you can continue running the last released version of the software. Even if the operating system and the computer it runs on become obsolete, you can still run the software in a virtual machine or emulator. As storage media evolve over the decades, you can copy your files to new storage media and continue to access them. On the other hand, cloud apps depend on the service continuing to be available: if the service is unavailable, you cannot use the software, and you can no longer access your data created with that software. This means you are betting that the creators of the software will continue supporting it for a long time — at least as long as you care about the data. Although there does not seem to be a great danger of Google shutting down Google Docs anytime soon, popular products do sometimes get shut down or lose data, so we know to be careful. And even with long-lived software there is the risk that the pricing or features change in a way you don’t like, and with a cloud app, continuing to use the old version is not an option — you will be upgraded whether you like it or not. Local-first software enables greater longevity because your data, and the software that is needed to read and modify your data, are all stored locally on your computer. We believe this is important not just for your own sake, but also for future historians who will want to read the documents we create today. Without longevity of our data, we risk creating what Vint Cerf calls a "digital Dark Age." Some file formats (such as plain text, JPEG, and PDF) are so ubiquitous that they will probably be readable for centuries to come. The US Library of Congress also recommends XML, JSON, or SQLite as archival formats for datasets. However, in order to read less common file formats and to preserve interactivity, you need to be able to run the original software (if necessary, in a virtual machine or emulator). Local-first software enables this. Martin Kleppmann, Adam Wiggins, Peter van Hardenberg, and Mark McGranaghan. Local-first software: you own your data, in spite of the cloud. 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!), October 2019, pages 154–178. doi:10.1145/3359591.3359737
In the fullness of time, the files you create are more important than the tools you use to create them. Apps are ephemeral, but your files have a chance to last. The ancient temples of Egypt contain hieroglyphs that were chiseled in stone thousands of years ago. The ideas hieroglyphs convey are more important than the type of chisel that was used to carve them. The world is filled with ideas from generations past, transmitted through many mediums, from clay tablets to manuscripts, paintings, sculptures, and tapestries. These artifacts are objects that you can touch, hold, own, store, preserve, and look at. To read something written on paper all you need is eyeballs. Today, we are creating innumerable digital artifacts, but most of these artifacts are out of our control. They are stored on servers, in databases, gated behind an internet connection, and login to a cloud service. Even the files on your hard drive use proprietary formats that make them incompatible with older systems and other tools. […] If you want your writing to still be readable on a computer from the 2060s or 2160s, it’s important that your notes can be read on a computer from the 1960s. You should want the files you create to be durable, not only for posterity, but also for your future self. You never know when you might want to go back to something you created years or decades ago. Don’t lock your data into a format you can’t retrieve. These days I write using an app I help make called Obsidian, but it’s a delusion to think it will last forever. The app will eventually become obsolete. It’s the plain text files I create that are designed to last. Who knows if anyone will want to read them besides me, but future me is enough of an audience to make it worthwhile. Steph Ango, File over App, https://stephango.com/file-over-app
In the fullness of time, the files you create are more important than the tools you use to create them. Apps are ephemeral, but your files have a chance to last. The ancient temples of Egypt contain hieroglyphs that were chiseled in stone thousands of years ago. The ideas hieroglyphs convey are more important than the type of chisel that was used to carve them. The world is filled with ideas from generations past, transmitted through many mediums, from clay tablets to manuscripts, paintings, sculptures, and tapestries. These artifacts are objects that you can touch, hold, own, store, preserve, and look at. To read something written on paper all you need is eyeballs. Today, we are creating innumerable digital artifacts, but most of these artifacts are out of our control. They are stored on servers, in databases, gated behind an internet connection, and login to a cloud service. Even the files on your hard drive use proprietary formats that make them incompatible with older systems and other tools. […] If you want your writing to still be readable on a computer from the 2060s or 2160s, it’s important that your notes can be read on a computer from the 1960s. You should want the files you create to be durable, not only for posterity, but also for your future self. You never know when you might want to go back to something you created years or decades ago. Don’t lock your data into a format you can’t retrieve. These days I write using an app I help make called Obsidian, but it’s a delusion to think it will last forever. The app will eventually become obsolete. It’s the plain text files I create that are designed to last. Who knows if anyone will want to read them besides me, but future me is enough of an audience to make it worthwhile. Steph Ango, File over App, https://stephango.com/file-over-app