[…] another thing would be preserving history, having access to the past. I think this is especially important with software, it's very easy to lose. that both in the sense of the software itself and the data that it was manipulating.
McGranaghan, Metamuse Episode 49, Software Longevity, 08:14 , https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=NrliJcWGQsWYY4NkOqTnYA
[McGranaghan]: I think there's a dimension of longevity which is, and I am going to make up a word here, the rollover ability to buy or get a new version of something that rolls over your previous state.
So, for example, maybe you re-bind a book and roll over the pages or you can put a new engine and transmission in a truck and you roll over the truck frame. But some stuff can't be rolled-over: classically, the black-box data, just some opaque binary format.
So I think that ability to roll over is an important dimension even if it’s separate from a single instance having longevity.
[Wiggins]: Information has this particular property that exists on a substrate or a medium of something physical - atoms - but in the end, the information is not the physical thing. So the book rebinding example, you can either rebind the same book or in some cases just transcribe it completely. And as long as it's a correct and accurate copy, older books are an incredible artifact, maybe they have margin, something about how the pages were made or whatever does carry some history, there were the artifacts and objects of their own right, but ultimately the contents, the words on the page are probably what we care about.
So, on one hand information technology has actually been getting worse in the sense of the durability of the underlying thing: Stone tablets were very durable. Papyrus and later paper were less. So then we go into the digital world and it's actually vastly less. Everything from CD-Rs to cloud storage, the failure rate is pretty high but because it is so easy cheap to copy those bits from one place to another to replicate it. You potentially have the ability to roll it forward as far as it's worthwhile to do.
McGranaghan, Metamuse Episode 49, Software Longevity, 09:05, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=zDGVEtNTR_KHIINVW-HrKQ
[McGranaghan]: Another thing that I'm very big on is really focusing on data, often, especially in this world of rolling forward over multiple generations, what you really want to keep as the data.
This is more true for productivity-like stuff than games for example, but data is very important. And, if you want to roll forward data, you either need to adopt an existing open data format or you need to do your own, which can be done, but it's a lot of work. I think that programs or program lineages, if you call that, embrace this idea of really focusing on the data so that that can always be ejected and rolled forward into the next generation of software.
[...]
One of my favorite examples is SQLite. If you use SQLite to write even your semi-custom data format, if you write it into SQLite database, you're absolutely gonna be able to read that in 30 years, even if you do nothing yourself. Whereas, if you roll your own format on disk, there's a very good chance that it will at some point become totally lost.
[Wiggins]: Yeah, SQLite is a favorite example […] of well-made simple software that does one thing and does it well. They have a great page though about their long-term support where they list off some techniques, like, testing and making their database files cross-platform and disaster planning and not embracing hot new technologies with too much gusto.
McGranaghan & Wiggins, Metamuse Episode 49, Software Longevity, 27:05, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=7WHCqDsQRaK-HaTeS4HQVA
[McGranaghan]: […] that circles back to this idea of data. I see people typically design and implement programs […] we tend to focus on the interface and it behavior because that's what users are demanding because that's what the short-term success of our business depends on.
But the reality is that interface, including its implementation, is going to almost certainly fully churn at least once potentially several times over the course of software. But what's not going away is the data. And so, one thing that I think is helpful for longevity of software is really focusing on the underlying data model. Not only in the sense of how it's stored, which is mostly what we've been talking about in the podcast so far in terms of text files or SQLite, but also what is the data model itself. Like if you were to draw out the SQL tables, or equivalent, draw the boxes and put labels on them, it sounds really basic but it's a step that I feel like people often skip over and then it leads to all kinds of longevity headaches down the road.
Future-proofing I think is a word we sometimes use in general for talking about helping things be longer-lived, but certainly I think that's most notable in the data model, precisely for the reason you said: it's just much less changeable and harder to change and heavier weight to change.
McGranaghan & Wiggins, Metamuse Episode 49, Software Longevity, 31:20, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=2S81RLe9Q4mA36etUZN7QA
[Wiggins] The version thing is also making me think of, I think it’s the image file format, maybe tiff is the one I'm thinking of, but this approach has been used in a lot of different places, and I feel like you're doing a version of this in the local-first sync infrastructure we’re working on right now.
And so you can add new chunks to a stream of data or to a file that older versions of the software can load, recognize they don't know this chunk and they can skip past it and just try to interpret the rest of it the best they can. Maybe this is something the web browsers do pretty well which is, you load a newer page in an older web browser, if there's some not supported features it doesn't completely break, it just does its best to render it. […] It's not that it just gives up the moment it sees something it doesn't understand.
[McGranaghan]: And this gets into the peculiar challenges of longevity with distributed system software. We've been talking about longevity in terms of supporting the past, but with distributed systems you gotta support the future because the future arrives unevenly. So that's the sort of challenge you deal with in your working on distributed systems like the data synchronization layer from Muse.
[Wiggins]: As usual Ink & Switch has some good research on this subject with the data lenses in the Cambria project. One of the things there is about data migrations, but it's assuming you need to translate both ways. You need to translate back in time the older versions and forward in time to newer versions, but in fact, it's actually more complex than that, because there may be just a branching tree of systems that all understand that the data in slightly different ways.
Wiggins & McGranaghan, Metamuse Episode 49, Software Longevity, 34:55, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=A2dedJMzR4KhKIgM8COa2g
[Wiggins]: And the other small thing, it's sort of obvious in some ways, it's kind of the Lindy Effect of file formats. So, Lindy effect here being sort of the idea that something that’s been around a long time probably will be around a long time.
So, I'm always a big fan of those flat file formats, PNG, JPEG, PDF, plain text, because they've proven themselves to be durable over the long term and they don't always do everything you need but where you can, it's really nice to just go down to that simple common format that is understood by many applications both now and in the past and potentially will be in the future.
Wiggins & McGranaghan, Metamuse Episode 49, Software Longevity, 36:41, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=ozNX-91xRhKag6u_febOKw