Principles and characteristics of creating tools fostering creativity and collaboration
Interoperability
Colllaborating across tools can mean several different things: - It can mean like you are saying you have a given project and at different stages of the project you want to be able to use different tools. That's one case where an open format would help. - You might want to collaborate with other people and they might want to use their own tools, which is different from yours. It's another case. - And also there's this element of time where overtime software tools tend to atrophy companies come and go, you know, platforms changed but you at least want your data and to be able to carry that with you in some sort of archive, at least. So there are many cases. McGranaghan, Metamuse ep.30 Computers and Creativity with Molly Mielke 31:28, https://open.spotify.com/episode/5HoFWZzo4PAzEhF5Tn3WbD?si=AmfN5kUhQK669_VhAK9e7w
Absence of interoperability and its relationship with capitalism and prevailing social structures
I think this is a prime example of the political economy issue. It's very easy to say we should have x. And even if x is relatively easy to do, which is not, in this case, there's still this huge issue of the “we should” that's quite the weasel phrase, right? If we were to accomplish this, we will need a bunch of companies or individual developers to temporarily make more work for themselves. Lower the profitability, make their products worse for their customers in the short term to get to some kind of global maximum. It’s a case where the coordination problem is really important. McGranaghan, Metamuse ep.30 Computers and Creativity with Molly Mielke 32:05, https://open.spotify.com/episode/5HoFWZzo4PAzEhF5Tn3WbD?si=AmfN5kUhQK669_VhAK9e7w
I think that the lack of interoperability or standardization between digital tools today, really, it means that all work created within a tool is confined to that tool and to me that seems very clearly antithetical to creativity and specifically the collaborative aspect of creativity. I think that there's so much to be said for tools. Amplifying the power of our brains and really taking over the mechanical aspects of human thought and limiting creation to a single piece of software's capabilities is just kind of crazy if you have step back and think about it. And I just think that standardization and having tools talk to each other, which is fundamentally change the tide of how we use them and introduced in more collaborators, and really just expand the projects constraints beyond any one tool. Mielke, Metamuse ep.30 Computers and Creativity with Molly Mielke 30:00, https://open.spotify.com/episode/5HoFWZzo4PAzEhF5Tn3WbD?si=KRW6ed30T1KEm4G8fa8FSA
Data silos and lack of interoperability can very well be a contradiction of capitalism when we think of the political economy of social media.
Computer visionaries and pioneers of the 60s and 70s
New frontiers exploration needs a group of people doing heavy knowledge work with a new technology that enables it. Doug Engelbart, https://youtu.be/LpIYdyOXqkk?t=2789
“We also see this in the million startups that have reinvented the spreadsheet except constrained it to perform a particular set of things and realize: no, really, people use spreadsheets because they are unconstrained”. https://youtu.be/07j2AXC9BH8?si=aekq9ikZ-xjlaeS-&t=1037
Our Collective IQ is boosted by concurrently developing, integrating and applying knowledge. Doug Engelbart, https://www.youtube.com/watch?v=LpIYdyOXqkk&t=880s
The sense of a lost vision of computer pioneers of the 60s and 70s. Dynamics of digital media production
[…] there was these amazing visionaries who saw the potential for computers and creativity, sort of laid out a vision way back in what seems like just the stone age's to us the 1960s and 1970s and then in some ways we lost our way. And we ended up with, I don't know, social media and kind of lockdown appliance like smartphones. And in fact, there's this glorious world of Smalltalk, Dynabook and Mother of all Demos style stuff that we still need to build, or we haven't built or something like that. Wiggins, Metamuse ep.30 Computers and Creativity with Molly Mielke 14:30, https://open.spotify.com/episode/5HoFWZzo4PAzEhF5Tn3WbD?si=AmfN5kUhQK669_VhAK9e7w
[…] let me bring the positive side, which is I totally agree that they did lay out a vision for computing. That is grounding in a world where we do seem stuck in social media consumer not exactly sure what; everything is oriented around commerce, and again things are all good: commerce entertainment, these things are fine, I consume these. But the reason I got interested in computers that are very young ages seeing their potential for creativity, and unlocking the noblest parts of the human spirit. Wiggins, Metamuse ep.30 Computers and Creativity with Molly Mielke 23:25, https://open.spotify.com/episode/5HoFWZzo4PAzEhF5Tn3WbD?si=AmfN5kUhQK669_VhAK9e7w
And it's a good reminder, to go back and look at some of this history, maybe, especially because these folks didn't have any of that prior stuff computers were still so new. Particularly personal computing was essentially, you know, they were in the process of inventing it thinking what can people do if they had access to computers with graphics and networking and all the things that nowadays we take for granted but they dreamed something very different from the world we have today and that can be very grounding to look back at that and take a bit of a blank slate from where we are today. So yeah, I personally take a lot of inspiration from all their work as well. Wiggins, Metamuse ep.30 Computers and Creativity with Molly Mielke 24:05, https://open.spotify.com/episode/5HoFWZzo4PAzEhF5Tn3WbD?si=AmfN5kUhQK669_VhAK9e7w
I also think that they were operating in an environment that was largely kind of independent from the actual business environment and like, the technology sector as we see it today. Mielke, Metamuse ep.30 Computers and Creativity with Molly Mielke 16:04, https://open.spotify.com/episode/5HoFWZzo4PAzEhF5Tn3WbD?si=5lptp_mFReqInuu1Bjsbnw
Both Mielke and McGranaghan acknowledge that it’s the political economy and the business needs that prevented computer pioneers’ vision to be materialized. But, especially McGranaghan remains uncritical of capitalism (quite the opposite basically). I think the latter is a general sense I have, not directly expressed in this partcular excerpt.
[…] a lot of this study analysis has been at the level of the tools. So it's like what's on the screen, how do you program it, what's the user interface even. But there’s an entire complex system around how software is developed and used and, like you were saying, I think the reason that the vision for the software and the reality the software don't line up is because we haven't understood or at least accounted for how that ecosystem works. Sometimes I call this the political economy of software development, there's weird path dependence, there's economic incentives; you got to understand all of that if you are going to understand how we came to where we are now. And on the flip side, if you want to predict and guide the future in that direction, you need to become a sort of of political economist of software, and get not only the interfaces in the code, but also the funding and the incentives and legal stuff and all that. McGranaghan, Metamuse ep.30 Computers and Creativity with Molly Mielke 18:57, https://open.spotify.com/episode/5HoFWZzo4PAzEhF5Tn3WbD?si=hDV9jmWRQPSyXg1tvFtVeQ
Important open problems for knowledge work primarily not tackled due to economic reasons
[McGranaghan]: Isn’t the state of the art [for rendering equations in the browser] generating PNG and sticking it in the web page? [Slim]: Not exactly but almost. So MathML, which is an XML dialect or mathematical markup language was this effort to build XML style syntax for typesetting mathematics naturally. It is only implemented in Firefox, so that's really unfortunate. So in terms of the state of the art, there are basically two libraries that you can use to typeset mathematics: there's MathJax and KaTex. MathJax supports all valid LaTex […]. The problem is that MathJax is very slow. So if you ever go on MathOverflow or another related StackExchange and you see all of these answers with weird gaps, and then as you watch before you the page starting to load all of the rendered equations bumping everything down one level at a time, that's MathJax in action. And oftentimes it is doing what you're describing where it is outputting an SVG or a PNG or something like that. And it's reflowing the page with every equation. Then you have KaTex, which was a library developed at Khan Academy where they realize that MathJax’s performance was not satisfactory for their exercises. KaTex supports a much more limited subset of all of LaTex syntax, but it does it all using CSS and it doesn't reflow the page for every equation, it’s basically instant rendering. And basically, it means that your options if you want to render math are: - Only target Firefox - Use a limited subset of math that’s supported by KaTex - Constrain yourself to extremely slow, dozens-of-reflow, full-expressive-power rendering to inline PNGs. And that's just not a great situation to be in and we haven't even gotten to the question of how people write math. So I would say that people underestimate how open this problem space is. Just take a moment of silence to like recognize the gravity of the situation. Slim Lim, Metamuse Episode 48 Rich Text with Slim Lim, 52:05, https://open.spotify.com/episode/51u2e3Ak6E4yMfqmA0NOdm?si=kKrmc0JWRdey8N1s1OXC9A
[Slim] And that goes back to something that Adam and I were chatting about earlier, which is that a lot of people gripe and complain, that LaTex is the best we have, and, you know, I'm one of them. But it really is the case that LaTex was this monumental effort by really a few people, and amount of effort that would be considered really impressive if I were to try to do the same thing but better today. Not a lot of people have spare time to do this all-in-one text formatting and packaging document representation project even though it would have a huge impact on the way people write and publish these kinds of documents. And so, in many ways, we're sort of just bottlenecked on the fact that it's hard to do incremental improvements to this particular area. We really depend on these software monoliths to keep us afloat. […] [Wiggins] And yeah, I agree. It's a monumental effort, but I mean mathematics is so important and foundational and so much of human endeavor that certainly seems like one worth investing in, although perhaps hard to reap a profit from and that makes it harder to put concentrated capital behind it. Slim Lim & Wiggins, Metamuse Episode 48 Rich Text with Slim Lim, 48:30, https://open.spotify.com/episode/51u2e3Ak6E4yMfqmA0NOdm?si=BHebRCxjQVSETJlrxi1McQ
Now that we have all these wonderful devices, it should be the goal of society to put them in the service of truth and learning. And this is the way I propose. Not through obscure forms of "information retrieval;" not through newly oppressive forms of "computer-assisted instruction;" and not through a purported science of "artificial intelligence“ that will create new personalisms to irk us. All these obstructive oddities, I think, have developed as separate ideals because of the grand preposterosity of Professionalism that has created a world-wide cult of mutual incomprehensibility and disconnected special goals. Now we need to get everybody together again. We want to go back to the roots of our civilization - the ability, which we once had, for everybody who could read to be able to read everything. We must once again become a community of common access to a shared heritage. This was of course what Vannevar Bush said in 1945 in an article everybody cites but nobody reads. Ted Nelson, Computer Lib/Dream Machines, Hypertext
This all has a certain vagueness if you do not understand how bound you are today by paper— the problems of finding it , sorting it, looking things up. (If you write, that is, write a lot, you know all too well how intractable is paper, what a damned nuisance.) With a system like Engelbart's, now, whatever is written is instantly there. Whatever you want to look up Is instantly there, simultaneously interconnected to everything else l t should be connected to - source materials, footnotes, comments and so on. A document is completed the moment it is written: no human being has to retype i t . (It need not be typed on paper at all ,if it’s just for the workshop members: a printout is only needed if it has to go to someone outside the system). In many ways, Engelbart's system is a prototype for the world of the future, I hope. ALL HANDLING OF PAPER IS ELIMINATED. Whatever you write, you write on the screens with keyboard and pointer. (No more backs of envelopes, yellow pads, file cards, typewriters.) Whatever you transmit to fellow users of the system you simply 'release' - no physical paper changes hands. The group has also worked out some remarkable techniques for collaborative endeavor. Two people - say, one in California and one in New York - can work together through their screens, plus a phone link; it's as if they were side-by-side at a magic table. Each sees on his screen what the other sees; each controls a moving dot (or "cursor") that shows where he's pointing. The effect is somewhere between a blackboard and a desk; both may call up documents, point things out in them, change them, and anything else two people might do when working on something together. Basically the system Is a large-scale setup for the storage, bringing forth, viewing and revision of documents and connections among them. Ted Nelson, Computer Lib/Dream Machines, Doug Engelbart and “The Augmentation of Intellect”
[…] human action, or even human thought, can only take place through some kind of material medium and therefore can’t be understood without taking the qualities of that medium into account. Hence the importance in Turner’s analysis of the notion of material media of circulation. The media have qualities in and of themselves. For all the (often quite legitimate) criticisms of Jack Goody’s dichotomies between orality and literacy, for example, it is simply obvious that technologies of writing allow for possibilities that do not exist in speech (and equally, vice versa). Graeber, Toward an Anthropological Theory of Value, p.83
The open problem of writing and rendering math using inline equations
Back in the days of WYSIWYG web editors like Dreamweaver and Microsoft FrontPage you would often end up with this problem, where you try to do any edit in the WYSIWYG side and then you look at the generated HTML and is ridiculous: There's like 16 nested, empty span tags and no one would ever be able to maintain that. And my worry is basically that when you automatically create markup for something that has a very complex graphical representation, it's really like one way: maybe it will help you produce a compiled output, but it doesn't actually help you go back in edit and tweak the representation later. I think we really need to get to some kind of good intermediate representation that allows you to flexibly go both ways. Slim Lim, Metamuse Episode 48 Rich Text with Slim Lim, 47:35, https://open.spotify.com/episode/51u2e3Ak6E4yMfqmA0NOdm?si=c8WSrsIUTvqlq4tfmT1CRA
[Slim] And that goes back to something that Adam and I were chatting about earlier, which is that a lot of people gripe and complain, that LaTex is the best we have, and, you know, I'm one of them. But it really is the case that LaTex was this monumental effort by really a few people, and amount of effort that would be considered really impressive if I were to try to do the same thing but better today. Not a lot of people have spare time to do this all-in-one text formatting and packaging document representation project even though it would have a huge impact on the way people write and publish these kinds of documents. And so, in many ways, we're sort of just bottlenecked on the fact that it's hard to do incremental improvements to this particular area. We really depend on these software monoliths to keep us afloat. […] [Wiggins] And yeah, I agree. It's a monumental effort, but I mean mathematics is so important and foundational and so much of human endeavor that certainly seems like one worth investing in, although perhaps hard to reap a profit from and that makes it harder to put concentrated capital behind it. Slim Lim & Wiggins, Metamuse Episode 48 Rich Text with Slim Lim, 48:30, https://open.spotify.com/episode/51u2e3Ak6E4yMfqmA0NOdm?si=BHebRCxjQVSETJlrxi1McQ
[McGranaghan]: Isn’t the state of the art [for rendering equations in the browser] generating PNG and sticking it in the web page? [Slim]: Not exactly but almost. So MathML, which is an XML dialect or mathematical markup language was this effort to build XML style syntax for typesetting mathematics naturally. It is only implemented in Firefox, so that's really unfortunate. So in terms of the state of the art, there are basically two libraries that you can use to typeset mathematics: there's MathJax and KaTex. MathJax supports all valid LaTex […]. The problem is that MathJax is very slow. So if you ever go on MathOverflow or another related StackExchange and you see all of these answers with weird gaps, and then as you watch before you the page starting to load all of the rendered equations bumping everything down one level at a time, that's MathJax in action. And oftentimes it is doing what you're describing where it is outputting an SVG or a PNG or something like that. And it's reflowing the page with every equation. Then you have KaTex, which was a library developed at Khan Academy where they realize that MathJax’s performance was not satisfactory for their exercises. KaTex supports a much more limited subset of all of LaTex syntax, but it does it all using CSS and it doesn't reflow the page for every equation, it’s basically instant rendering. And basically, it means that your options if you want to render math are: - Only target Firefox - Use a limited subset of math that’s supported by KaTex - Constrain yourself to extremely slow, dozens-of-reflow, full-expressive-power rendering to inline PNGs. And that's just not a great situation to be in and we haven't even gotten to the question of how people write math. So I would say that people underestimate how open this problem space is. Just take a moment of silence to like recognize the gravity of the situation. Slim Lim, Metamuse Episode 48 Rich Text with Slim Lim, 52:05, https://open.spotify.com/episode/51u2e3Ak6E4yMfqmA0NOdm?si=kKrmc0JWRdey8N1s1OXC9A
Digital tools and human-computer interaction
Spreadsheets
[Slim] But there's like a separate totally untapped underexplored area of spreadsheets which is that it basically this canvas right? Spreadsheets capture everything that people like about table based layouts in HTML with none of the stigma associated with it. You can create these really complex interfaces that basically just do data manipulation and put things be like: “okay, I'm gonna like copy this data and bring it over closer to where I'm working now so I can reference it more easily”. It's basically just this grid, right? And that's totally unstructured. It doesn't correspond to any kind of relational format but it's also a really powerful computation paradigm. [McGranaghan] Yeah, totally. I think people really love to be able to click somewhere and put stuff there and a lot of spreadsheet use is just that they just want to click there and put text or put a color and there's no formulas at all. Slim Lim & McGranaghan, Metamuse Episode 48 Rich Text with Slim Lim, 1:03:40, https://open.spotify.com/episode/51u2e3Ak6E4yMfqmA0NOdm?si=6pmti8IPQ4itIRFN0iG2RA
Hypertext
By "hypertext" I mean non-sequential writing. Ordinary writing is sequential for two reasons. First, it grew out of speech and speech-making, which have to be sequential; and second, because books are not convenient to read except in a sequence. But the structures of ideas are not sequential. They tie together every whichway. And when we write, we are always trying to tie things together in non-sequential ways. The footnote is a break from sequence; but it cannot really be extended (though some, like Will Cuppy, have toyed with the technique). […] writers do better if they don't have to write in sequence (but may create multiple structures, branches and alternatives), and readers do better if they don't have to read in sequence, but may establish impressions, jump around, and try different pathways until they find the ones they want to study most closely. [...] in my view, a new day is dawning. Computer storage and screen display mean that we no longer have to have things in sequence; totally arbitrary structures are possible, and I think that after we've tried them enough people will see how desirable they are. Ted Nelson, Computer Lib/Dream Machines, Hypertext
Paper moulders. Microfilm is inconvenient. In the best libraries it takes at least minutes to get a particular thing. But as to linking them together - footnoting Aeschylus with Marcus Aurelius, linking genetic data to 15th-century accounts of Indian tribes - well, you can only do it on paper by writing something new that ties them together. Isn't that ridiculous? When you could do it all electronically in seconds? Ted Nelson, Computer Lib/Dream Machines, Hypertext
Now that we have all these wonderful devices, it should be the goal of society to put them in the service of truth and learning. And this is the way I propose. Not through obscure forms of "information retrieval;" not through newly oppressive forms of "computer-assisted instruction;" and not through a purported science of "artificial intelligence“ that will create new personalisms to irk us. All these obstructive oddities, I think, have developed as separate ideals because of the grand preposterosity of Professionalism that has created a world-wide cult of mutual incomprehensibility and disconnected special goals. Now we need to get everybody together again. We want to go back to the roots of our civilization - the ability, which we once had, for everybody who could read to be able to read everything. We must once again become a community of common access to a shared heritage. This was of course what Vannevar Bush said in 1945 in an article everybody cites but nobody reads. Ted Nelson, Computer Lib/Dream Machines, Hypertext
The hypertext solution in many ways obviates some of these other approaches, and in addition retains and puts back together the great traditions of literature and scholarship, traditions based on the fact that dividing things up arbitrarily just generally doesn't work. Ted Nelson, Computer Lib/Dream Machines, Hypertext
In an important sense there are no "subjects" at all; there is only all knowledge since the cross- connections among the myriad topics of this world simply cannot be divided up neatly. Hypertext at last offers the possibility of representing and exploring it all without carving it destructively. Ted Nelson, Computer Lib/Dream Machines, Everything is Deeply Intertwingled
I believe computer screens can make people happier, smarter, and better able to cope with the copious problems of tomorrow. But only if we do right, right now. WHY? The computer's capability for branching among events, controlling exterior devices, controlling outside events, and mediating in all other events, makes possible a new era of media. Until now, the mechanical properties of external objects determined what they were to us and how we used them. But henceforth this is arbitrary. The recognition of that arbitrariness, and re-consideration among broader and more general alternatives, awaits us. All the previous units and mechanisms of learning, scholarship, arts, transaction and confirmation, and even self-reminder, were based in various ways upon physical objects - the properties of paper, carbon paper, files, books and bookshelves. To read from paper you must move the physical object in front of you. Its contents cannot be made to slide, fold, shrink, become transparent, or get larger. But all this Is now changing, and suddenly. The computer display screen does all these things if desired, to the same markings we have previously handled on paper. The computer display screen is going to become universal very fast; this is guaranteed by the suddenly rising cost of paper. And we will use them for everything. [...] Our screen displays will be alive with animation in their separate segments of activity, and will respond to our actions as if alive physically too. Ted Nelson, Computer Lib/Dream Machines, Fantics
Harnessing Complexity
Our greatest problems involve thinking and the visualization of complexity. By "Thinkertoy" I mean, first of all, a system to help people think. ('Toy' means it should be easy and fun to use.) This is the same general ides for which Engelbart, for instance, uses the term "augmentation of intellect." But a Thinkertoy is something quite specific: I define it as a computer display system that helps you envision complex alternatives. The process of envisioning complex alternatives is by no means the only important form of human thought; but it is essential to making decisions, designing, planning, writing, weighing alternate theories, considering alternate forms of legislation, doing scholarly research, and so on. it is also complicated enough that, in solving it, we may solve simpler problems as well. Ted Nelson, Computer Lib/Dream Machines, Thinkertoys
Software Longevity
[Wiggins]: For me, thinking about that word longevity, I tend to think about what is long, I guess. What is a span of time that counts as long and of course it depends a lot on what you're talking about. So if you're talking about all of society, culture, humanity, then perhaps you’re talking in the thousands of years, hundreds or thousands of years. So for example, the long now is a phrase that we use to the local-first paper, we tend to use it internally on our team to refer to longer-term. Thinking this comes from the Long Now Foundation, and the long now clock project, which is sort of this idea to build a clock that can keep time for 10,000 years and it's basically an art project, but it's designed to inspire us to think longer term, and that can obviously connect to things about climate change or human culture. And since humans are naturally inclined to think probably pretty short term, actually really short term: We think about the day that's ahead or the week that's ahead of us, maybe at most the year that's ahead of us, we don't tend to think in tens or hundreds or thousands of years. That's maybe a society level thing. I think for individuals and thinking about, you know, software as it impacts our lives as individuals, I think a human lifespan is actually is a pretty good chunk of time to compare something to. Wiggins and McGranaghan, Metamuse Episode 49, Software Longevity, 05:06, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=8Ny-UxkNStWnvjGA1EnO-A
Data and rolling it over to newer versions
[…] another thing would be preserving history, having access to the past. I think this is especially important with software, it's very easy to lose. that both in the sense of the software itself and the data that it was manipulating. McGranaghan, Metamuse Episode 49, Software Longevity, 08:14 , https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=NrliJcWGQsWYY4NkOqTnYA
[McGranaghan]: I think there's a dimension of longevity which is, and I am going to make up a word here, the rollover ability to buy or get a new version of something that rolls over your previous state. So, for example, maybe you re-bind a book and roll over the pages or you can put a new engine and transmission in a truck and you roll over the truck frame. But some stuff can't be rolled-over: classically, the black-box data, just some opaque binary format. So I think that ability to roll over is an important dimension even if it’s separate from a single instance having longevity. [Wiggins]: Information has this particular property that exists on a substrate or a medium of something physical - atoms - but in the end, the information is not the physical thing. So the book rebinding example, you can either rebind the same book or in some cases just transcribe it completely. And as long as it's a correct and accurate copy, older books are an incredible artifact, maybe they have margin, something about how the pages were made or whatever does carry some history, there were the artifacts and objects of their own right, but ultimately the contents, the words on the page are probably what we care about. So, on one hand information technology has actually been getting worse in the sense of the durability of the underlying thing: Stone tablets were very durable. Papyrus and later paper were less. So then we go into the digital world and it's actually vastly less. Everything from CD-Rs to cloud storage, the failure rate is pretty high but because it is so easy cheap to copy those bits from one place to another to replicate it. You potentially have the ability to roll it forward as far as it's worthwhile to do. McGranaghan, Metamuse Episode 49, Software Longevity, 09:05, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=zDGVEtNTR_KHIINVW-HrKQ
[McGranaghan]: Another thing that I'm very big on is really focusing on data, often, especially in this world of rolling forward over multiple generations, what you really want to keep as the data. This is more true for productivity-like stuff than games for example, but data is very important. And, if you want to roll forward data, you either need to adopt an existing open data format or you need to do your own, which can be done, but it's a lot of work. I think that programs or program lineages, if you call that, embrace this idea of really focusing on the data so that that can always be ejected and rolled forward into the next generation of software. [...] One of my favorite examples is SQLite. If you use SQLite to write even your semi-custom data format, if you write it into SQLite database, you're absolutely gonna be able to read that in 30 years, even if you do nothing yourself. Whereas, if you roll your own format on disk, there's a very good chance that it will at some point become totally lost. [Wiggins]: Yeah, SQLite is a favorite example […] of well-made simple software that does one thing and does it well. They have a great page though about their long-term support where they list off some techniques, like, testing and making their database files cross-platform and disaster planning and not embracing hot new technologies with too much gusto. McGranaghan & Wiggins, Metamuse Episode 49, Software Longevity, 27:05, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=7WHCqDsQRaK-HaTeS4HQVA
[McGranaghan]: […] that circles back to this idea of data. I see people typically design and implement programs […] we tend to focus on the interface and it behavior because that's what users are demanding because that's what the short-term success of our business depends on. But the reality is that interface, including its implementation, is going to almost certainly fully churn at least once potentially several times over the course of software. But what's not going away is the data. And so, one thing that I think is helpful for longevity of software is really focusing on the underlying data model. Not only in the sense of how it's stored, which is mostly what we've been talking about in the podcast so far in terms of text files or SQLite, but also what is the data model itself. Like if you were to draw out the SQL tables, or equivalent, draw the boxes and put labels on them, it sounds really basic but it's a step that I feel like people often skip over and then it leads to all kinds of longevity headaches down the road. Future-proofing I think is a word we sometimes use in general for talking about helping things be longer-lived, but certainly I think that's most notable in the data model, precisely for the reason you said: it's just much less changeable and harder to change and heavier weight to change. McGranaghan & Wiggins, Metamuse Episode 49, Software Longevity, 31:20, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=2S81RLe9Q4mA36etUZN7QA
[Wiggins] The version thing is also making me think of, I think it’s the image file format, maybe tiff is the one I'm thinking of, but this approach has been used in a lot of different places, and I feel like you're doing a version of this in the local-first sync infrastructure we’re working on right now. And so you can add new chunks to a stream of data or to a file that older versions of the software can load, recognize they don't know this chunk and they can skip past it and just try to interpret the rest of it the best they can. Maybe this is something the web browsers do pretty well which is, you load a newer page in an older web browser, if there's some not supported features it doesn't completely break, it just does its best to render it. […] It's not that it just gives up the moment it sees something it doesn't understand. [McGranaghan]: And this gets into the peculiar challenges of longevity with distributed system software. We've been talking about longevity in terms of supporting the past, but with distributed systems you gotta support the future because the future arrives unevenly. So that's the sort of challenge you deal with in your working on distributed systems like the data synchronization layer from Muse. [Wiggins]: As usual Ink & Switch has some good research on this subject with the data lenses in the Cambria project. One of the things there is about data migrations, but it's assuming you need to translate both ways. You need to translate back in time the older versions and forward in time to newer versions, but in fact, it's actually more complex than that, because there may be just a branching tree of systems that all understand that the data in slightly different ways. Wiggins & McGranaghan, Metamuse Episode 49, Software Longevity, 34:55, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=A2dedJMzR4KhKIgM8COa2g
[Wiggins]: And the other small thing, it's sort of obvious in some ways, it's kind of the Lindy Effect of file formats. So, Lindy effect here being sort of the idea that something that’s been around a long time probably will be around a long time. So, I'm always a big fan of those flat file formats, PNG, JPEG, PDF, plain text, because they've proven themselves to be durable over the long term and they don't always do everything you need but where you can, it's really nice to just go down to that simple common format that is understood by many applications both now and in the past and potentially will be in the future. Wiggins & McGranaghan, Metamuse Episode 49, Software Longevity, 36:41, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=ozNX-91xRhKag6u_febOKw
[Wiggins]: Now, some have argued that perhaps part of the challenge here is the rapid pace of change of that underlying system of those APIs. I think it's interesting here to contrast the Apple versus Microsoft approach. So I think that Microsoft with DOS and later Windows was really focused on backwards compatibility quite in an impressive level. I'm not exactly sure when the cut-offs are, but I think you could run DOS stuff natively on windows till well into the Windows lineage, and much later versions of Windows could run much earlier versions. But that's part of, I think, what also created the relative instability and unreliability of that line of operating systems, which is when you need to support every conceivable thing that's ever existed in the past and put it all together in the same box, and things get messy pretty fast. Apple with the iOS world has gone very much the other route which is they basically are releasing new stuff, a new OS update, once a year and if app developers don't keep up-to-date you pretty quickly just fall out of the App Store. […] You have this very demanding operating system that requires developers be doing active maintenance, or else it [your app] tends to go away. [McGranaghan]: While I don't doubt that there was some contribution in the Windows situation to instability from needing to support all versions forever, I think what really gets you is when you need to do that, and things that are really end-user-facing, like visual in particular, because the strategies that you typically use for providing stability are not viable in that world. So, if you look at things like Unix-style server, operating systems, many programming languages, these are things that can have backwards compatibility for a long time, certainly decades. But the way they do that is they just never ever, ever ever break any APIs. And if you want to do something new, either too bad, or call it "thing 2” and just keep “thing 1” around forever. […] you can't really do that with a windowing system, for example, or a user interface paradigm or how things look on the screen, right? You just got to pick something and do it, and the old thing needs to go. And so I think it does become very hard to have that culture of very long-lived durability that you see in those systems-y programs. [Wiggins]: Yeah, maybe that points to the fundamental trade-off there, which is durability in many cases means stability, which means less change. But change is how we get things that are new and better and the world technology and computers in the internet is so at the very beginning, there's so much unexplored space, and it would be a shame if we just said, okay, well we've been working on making these computers here for a few decades now, this is pretty much how they are, probably it is probably good for the next several hundred years, let's just hold it there. That's not what we want to do. We want to keep changing and improving. So yeah, there just is a trade-off between those two. [McGranaghan]: Yeah, I think there's a very real tension and I agree want to keep exploring. I do think that in adopting this attitude of exploring and taking risks, we are going to find some things that in retrospect were not good choices. The intuition here is, I think we're projecting these very dynamic system benefits over lives that we have historically associated with durable goods. So we get the shiny new computer-enabled, software-enabled thing, and it works great the first year, and we think: things like this, before we had computers, they worked for 20 or 30 years, therefore, this thing is going to work for 20 or 30 years, it's gonna be awesome. And we've kind of projected that out for, and I think in many cases that's not going to be true. And so we are going to have to confront the downside of that tradeoff. [Wiggins]: And do you think the downside there is mainly economic? You buy the smart refrigerator seems great for the first year, some cloud service goes offline suddenly and your refrigerator doesn't work anymore and you got to throw it out and get a new one, or do you think there's something beyond the economic side? [McGranaghan]: Unfortunately I think there are several potential very serious downsides. Just to give you a couple, one that we're already seeing is this issue of repairability or even modifiability. Again, if you look at these durable goods categories like tractors, which is very prominent example right now, traditionally people could service and maintain and change and modify and improve their own tractors. But as tractors increasingly become like iPhones, you know, these black-box computers in many ways, or highly computerized hardware, Potentially you can’t service them yourself, you can't repair them yourself, you can't modify them yourself and that has all kinds of second order effects. There's also this whole thing about surveillance basically that I think we're sleeping on a little bit, but we do need to go down that rabbit hole today. [Wiggins]: On the economic side, I think there's all sort of the incentive which is basically Apple and other phone manufacturers have done very very well by always creating that shiny new thing that comes out every year or two. And you think: I want to upgrade. It’s even often built into cell phone contracts and things, people want to get a new phone very frequently. And I think that's paired with sometimes the operating system upgrades, or maybe the visual refreshes, it's almost more like fashion, the fashion industry found this very clever way to get around the fact their IP isn't really protectable, which is they just make new fashion trends the totally change for a couple of years. So then you like need to go out and get a new wardrobe. It's not the clothes wore out necessarily. It's that you want to be up-to-date and you look behind the times when you’re strolling around in your bell bottoms and those have been out of fashion for three years or whatever it is. And so I think there's an element of this kind of a fashion or desire to get the new thing that has driven a lot of revenue for these companies. I don't mean to imply that that some kind of a cackling in the back layer, they're using this opportunity to improve and genuinely make their products better: Each version of the iPhone has that better camera, that better battery life, the bigger brighter screen, it is genuinely better, but it's also the companies are very incentivized for a high pace of change that has you buying new products all the time, as opposed to something longer-term. Wiggins and McGranaghan, Metamuse Episode 49, Software Longevity, 18:33 , https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=euUyYhr6SWC4C0KFtLmMHQ
Software environment and dependencies
[Wiggins]: Even thinking of a binary artifact, you've got a compiled executable, you can download it, you don't need to the dependencies, you don't need the compiler chain and I think there's the feeling that “shouldn’t that work forever?”. But of course as you said it sits within this context of the system: if it's accessing files on your file system, it's calling out to APIs on the network, it's accessing APIs in the operating system. Even something like it just makes assumptions about what kind of hardware exists, the keyboard and mouse has being standard input devices that exist and then you go onto a touch device and those aren't there. So over time, those fundamental assumptions and apis and the system changes around it and I think that's gotten dramatically more so with the internet, where you might call out to even something as simple as a static website, that has a Google Maps embed, well, now a big portion of that website is dependent on this exact integration to the service and that service is still online. So I think that the one-off binary build, shouldn’t-it-work-forever does make some sense in, for example, games, which don't have as many hooks (let's say just like a single player game, not some complicated cooperative thing) as it tends to have very simple inputs and outputs, displays things on the screen, maybe it needs to save a high-score file to the disc, as opposed to productivity software where all those little integration points: you drag and drop and how do you open a thing in the browser and all the different apis that are used to be a good citizen on the system, and those change all the time - they should with an operating system that's growing and improving and evolving - but then that means the apps have to keep up with that. And if they don't, they fall out of date. And eventually either become somewhat irrelevant or more commonly just stop working. [McGranaghan]: Yeah, I think this idea of the complexity of the surface area and the APIs is very important. You mentioned games: I think we actually had it much easier back in the good old days of games where we had keyboard input, mouse input, you draw to a pixel buffer, and that's it. The issue with games now, in addition to stuff around networking and multiplayer is that the graphics pipeline is incredibly complex, and unless you end up emulating earlier versions of graphics pipeline there's little chance this stuff rolls forward, whereas you could plausibly roll forward writing into a pixel bus offering. You kind of go the other way - you can emulate a pixel buffer with our advanced graphics APIs today, but there’s no chance of going the other direction. I think this generalizes to API complexity broadly and it's one of the reasons why it's so hard for software to persist Wiggins and McGranaghan, Metamuse Episode 49, Software Longevity, 13:12, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=ST_DRl-YQ3W4nxTMYYOD3w
[Wiggins]: Now another one from games that maybe is your roll-forward idea, I really like what id Software does, which is they take their classics - Doom, Quake and so on - And after I don't know if they're 20, 30 years old, something like that, they open source of them, and that creates this interesting effect where maybe it's less that people are playing the original, although I'm sure that happens, but then it becomes fun to just port it to weird places. Like you've seen Quake that's been ported to be 100% ASCII. I also saw a Twitter thread recently where someone managed to open one of these digital pregnancy tests, realize that it had a reasonably good processor and display, and decided to get Doom running on it. […] So in a way that has turned this over to remix culture and the internet and so it gets rolled forward and become something that maybe people will experience not in its original form necessarily, but becomes woven into the cultural tapestry let’s say through these fun little projects. [McGranaghan]: I think this also speaks to the importance of that API environment they were talking about earlier being specced or at least speccable for the software to be rolled forward or actively preserved. […] Old-timey games, that's a pretty speccable environment, like needing to be able to compile C, and relatively straightforward input and output system, and even if the games weren't designed specifically with a really clean interface there you can basically back it out after the the fact, and then do the porting. Books and paintings, by the way, are examples of analog things that are very speccable: they map down to text streams and bitmaps respectfully. But stuff like modern software is much more difficulty being specced, and so it becomes harder to do that sort of rolling forward act of preservation. Wiggins and McGranaghan, Metamuse Episode 49, Software Longevity, 16:40, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=JYEnhTUbSPCutemfh9wI-A
One thing we've alluded to is idea of narrow-defined APIs, because your software has to run in something, and the broader, the more complex, the more ill-specified that environment is, the harder it's going to be to do this job of keeping the software running. And this would include things like taking on few dependencies, minimizing weird binary dependencies that are hard to compile, limiting the APIs that you access, things like that. McGranaghan & Wiggins, Metamuse Episode 49, Software Longevity, 26:39, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=-2-SDcZXSUaI6iFWTO7eDg
The constituent parts of software longevity. Software as a special class of information
[Wiggins]: If we come to software as a special class of information, and it is information but it's highly dynamic, and I think that points to some of the particular challenges with it in […] software itself and its ability to be long-lived. And I think some of this is cultural, the tech industry is dynamic. It thrives on change, that's sort of the nature of it and many things that are good about software and the internet and the tech world come from that willingness to embrace the new and almost an endless seeking of novelty. But the downside is, you get something that is not just an upgrade treadmill but it just seems like it can be very much the case that the lifetime of product or of data or any particular piece of software can be measured in years, and only a small number of years, which is comparatively just a very, very short time span. […] But I feel there's more to it than just cultural, the tech world doesn't do the buy-it-for-life / built-to-last thing. I feel like there's something inherent to the dynamic-information nature of software, that makes it hard for it to be long-lived. [McGranaghan]: Yes, I think that's true. There's a lot of things that need to go right for software to work, and you can see it, in fact, as something that's strictly harder than preserving simple textual information for the following reason: You have at least the source code, then you need to preserve information about the dependencies. You potentially have the compiled source and then you have the whole runtime around it, very broadly defined as the computer and the APIs, maybe even the data. And this is just a very broad, multi-dimensional, and many cases, ill-specified state space: if you don't get your thing, exactly right in that state space, it doesn't work at all. Which, by the way, is something that's a little bit unique to digital systems versus classic analog systems. If your wrench is Rusty, you know, doesn't work as well, but it's still basically works as a wrench. Whereas, if you have one bit off in your software it just crashes - you're out of luck. It's very high-dimensional and it's very sensitive to errors in preserving the environment. Wiggins and McGranaghan, Metamuse Episode 49, Software Longevity, 10:52, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=u0GQfL1US5ujSnD0yl1Jsw
[Wiggins]: First I have to ask what it means to you to talk about something being long-lived or having longevity, whether its software or a plane or something else. [McGranaghan]: I think it's the software being able to serve its stated purpose, which sounds very straightforward, but has several important constituent parts: - First of all, you actually need to software, you know, sometimes we just lose it. - It needs to run and needs to run correctly and - it needs to have access to the relevant data. - It needs to have access to relevant pieces of the outside world in terms of APIs and - it needs the appropriate substrate to run on an interact with. - And it's probably other pieces that we could come up with. McGranaghan and Wiggins, Metamuse Episode 49, Software Longevity, 04:20, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=ca3gfTLARQ2EExDeBeQ5yA
[Wiggins]: Even thinking of a binary artifact, you've got a compiled executable, you can download it, you don't need to the dependencies, you don't need the compiler chain and I think there's the feeling that “shouldn’t that work forever?”. But of course as you said it sits within this context of the system: if it's accessing files on your file system, it's calling out to APIs on the network, it's accessing APIs in the operating system. Even something like it just makes assumptions about what kind of hardware exists, the keyboard and mouse has being standard input devices that exist and then you go onto a touch device and those aren't there. So over time, those fundamental assumptions and apis and the system changes around it and I think that's gotten dramatically more so with the internet, where you might call out to even something as simple as a static website, that has a Google Maps embed, well, now a big portion of that website is dependent on this exact integration to the service and that service is still online. So I think that the one-off binary build, shouldn’t-it-work-forever does make some sense in, for example, games, which don't have as many hooks (let's say just like a single player game, not some complicated cooperative thing) as it tends to have very simple inputs and outputs, displays things on the screen, maybe it needs to save a high-score file to the disc, as opposed to productivity software where all those little integration points: you drag and drop and how do you open a thing in the browser and all the different apis that are used to be a good citizen on the system, and those change all the time - they should with an operating system that's growing and improving and evolving - but then that means the apps have to keep up with that. And if they don't, they fall out of date. And eventually either become somewhat irrelevant or more commonly just stop working. [McGranaghan]: Yeah, I think this idea of the complexity of the surface area and the APIs is very important. You mentioned games: I think we actually had it much easier back in the good old days of games where we had keyboard input, mouse input, you draw to a pixel buffer, and that's it. The issue with games now, in addition to stuff around networking and multiplayer is that the graphics pipeline is incredibly complex, and unless you end up emulating earlier versions of graphics pipeline there's little chance this stuff rolls forward, whereas you could plausibly roll forward writing into a pixel bus offering. You kind of go the other way - you can emulate a pixel buffer with our advanced graphics APIs today, but there’s no chance of going the other direction. I think this generalizes to API complexity broadly and it's one of the reasons why it's so hard for software to persist Wiggins and McGranaghan, Metamuse Episode 49, Software Longevity, 13:12, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=ST_DRl-YQ3W4nxTMYYOD3w
[Wiggins] The version thing is also making me think of, I think it’s the image file format, maybe tiff is the one I'm thinking of, but this approach has been used in a lot of different places, and I feel like you're doing a version of this in the local-first sync infrastructure we’re working on right now. And so you can add new chunks to a stream of data or to a file that older versions of the software can load, recognize they don't know this chunk and they can skip past it and just try to interpret the rest of it the best they can. Maybe this is something the web browsers do pretty well which is, you load a newer page in an older web browser, if there's some not supported features it doesn't completely break, it just does its best to render it. […] It's not that it just gives up the moment it sees something it doesn't understand. [McGranaghan]: And this gets into the peculiar challenges of longevity with distributed system software. We've been talking about longevity in terms of supporting the past, but with distributed systems you gotta support the future because the future arrives unevenly. So that's the sort of challenge you deal with in your working on distributed systems like the data synchronization layer from Muse. [Wiggins]: As usual Ink & Switch has some good research on this subject with the data lenses in the Cambria project. One of the things there is about data migrations, but it's assuming you need to translate both ways. You need to translate back in time the older versions and forward in time to newer versions, but in fact, it's actually more complex than that, because there may be just a branching tree of systems that all understand that the data in slightly different ways. Wiggins & McGranaghan, Metamuse Episode 49, Software Longevity, 34:55, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=A2dedJMzR4KhKIgM8COa2g
Preserving games as a form of cultural history preservation
[Wiggins]: Now another one from games that maybe is your roll-forward idea, I really like what id Software does, which is they take their classics - Doom, Quake and so on - And after I don't know if they're 20, 30 years old, something like that, they open source of them, and that creates this interesting effect where maybe it's less that people are playing the original, although I'm sure that happens, but then it becomes fun to just port it to weird places. Like you've seen Quake that's been ported to be 100% ASCII. I also saw a Twitter thread recently where someone managed to open one of these digital pregnancy tests, realize that it had a reasonably good processor and display, and decided to get Doom running on it. […] So in a way that has turned this over to remix culture and the internet and so it gets rolled forward and become something that maybe people will experience not in its original form necessarily, but becomes woven into the cultural tapestry let’s say through these fun little projects. [McGranaghan]: I think this also speaks to the importance of that API environment they were talking about earlier being specced or at least speccable for the software to be rolled forward or actively preserved. […] Old-timey games, that's a pretty speccable environment, like needing to be able to compile C, and relatively straightforward input and output system, and even if the games weren't designed specifically with a really clean interface there you can basically back it out after the the fact, and then do the porting. Books and paintings, by the way, are examples of analog things that are very speccable: they map down to text streams and bitmaps respectfully. But stuff like modern software is much more difficulty being specced, and so it becomes harder to do that sort of rolling forward act of preservation. Wiggins and McGranaghan, Metamuse Episode 49, Software Longevity, 16:40, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=JYEnhTUbSPCutemfh9wI-A
[Wiggins]: Games also make me think of […] emulators and it turns out that games seem to be something that people do want to preserve as a cultural history, but in a way, the best way to do that is not try to get the original hardware but to have these emulators like MAME for example, and it's really impressive: you can play almost every arcade cabinet video game from the 80s, just in your browser with this kind of JavaScript emulators that will load up the ROMs, which are basically the way they distributed binaries in those days and play them, not quite as they were intended because the hardware is different, certainly displays are different, but certainly we do preserve some of that legacy. Wiggins and McGranaghan, Metamuse Episode 49, Software Longevity, 15:54, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=ohDy4dTAThWNh92mQP1ENA
Techniques for achieving software longevity
[McGranaghan]: You gotta sit down every week for 30 years and make sure the software doesn't break, and that's sort of a necessary but not sufficient condition. [Wiggins]: And the implication there is just maintenance, right? And that maybe ties into some of our discussions about software supported by subscriptions versus one-off payments, which is really is an ongoing effort, important ongoing maintenance effort, rather than a one and done. McGranaghan & Wiggins, Metamuse Episode 49, Software Longevity, 26:05, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=-2-SDcZXSUaI6iFWTO7eDg
One thing we've alluded to is idea of narrow-defined APIs, because your software has to run in something, and the broader, the more complex, the more ill-specified that environment is, the harder it's going to be to do this job of keeping the software running. And this would include things like taking on few dependencies, minimizing weird binary dependencies that are hard to compile, limiting the APIs that you access, things like that. McGranaghan & Wiggins, Metamuse Episode 49, Software Longevity, 26:39, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=-2-SDcZXSUaI6iFWTO7eDg
[McGranaghan]: Another thing that I'm very big on is really focusing on data, often, especially in this world of rolling forward over multiple generations, what you really want to keep as the data. This is more true for productivity-like stuff than games for example, but data is very important. And, if you want to roll forward data, you either need to adopt an existing open data format or you need to do your own, which can be done, but it's a lot of work. I think that programs or program lineages, if you call that, embrace this idea of really focusing on the data so that that can always be ejected and rolled forward into the next generation of software. [...] One of my favorite examples is SQLite. If you use SQLite to write even your semi-custom data format, if you write it into SQLite database, you're absolutely gonna be able to read that in 30 years, even if you do nothing yourself. Whereas, if you roll your own format on disk, there's a very good chance that it will at some point become totally lost. [Wiggins]: Yeah, SQLite is a favorite example […] of well-made simple software that does one thing and does it well. They have a great page though about their long-term support where they list off some techniques, like, testing and making their database files cross-platform and disaster planning and not embracing hot new technologies with too much gusto. McGranaghan & Wiggins, Metamuse Episode 49, Software Longevity, 27:05, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=7WHCqDsQRaK-HaTeS4HQVA
[Wiggins] […] one of the things that points to that to me feels like maybe the most fundamental is to just think about longevity in the first place and just care about it in the first place. Part of the reason I thought this topic would be an interesting one.I think, again, tech industry skews (?) young, I think also had the same gap in my thinking as a younger person, which was I just didn't have the longer experience to be able to see what it is like for software that I've created that's in production for three years, five years, ten years, and longer and what happens over time. It's a part of what I like with the SQLite long-term support page, they basically lead with a statement that our goal is to support this until the year 2050 and so, will they achieve that? hard to know… But just by making that statement, calling it out as a value for themselves and thinking about it actively, maybe this comes back to the long-now clock idea, which is, they may or may not succeed but certainly one good way to not succeed is to not think about it in the first place. And so this makes it a first-class concern. Wiggins & McGranaghan, Metamuse Episode 49, Software Longevity, 30:14, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=UT2qfBJAQWWjr5QsUGkunw
[McGranaghan]: […] that circles back to this idea of data. I see people typically design and implement programs […] we tend to focus on the interface and it behavior because that's what users are demanding because that's what the short-term success of our business depends on. But the reality is that interface, including its implementation, is going to almost certainly fully churn at least once potentially several times over the course of software. But what's not going away is the data. And so, one thing that I think is helpful for longevity of software is really focusing on the underlying data model. Not only in the sense of how it's stored, which is mostly what we've been talking about in the podcast so far in terms of text files or SQLite, but also what is the data model itself. Like if you were to draw out the SQL tables, or equivalent, draw the boxes and put labels on them, it sounds really basic but it's a step that I feel like people often skip over and then it leads to all kinds of longevity headaches down the road. Future-proofing I think is a word we sometimes use in general for talking about helping things be longer-lived, but certainly I think that's most notable in the data model, precisely for the reason you said: it's just much less changeable and harder to change and heavier weight to change. McGranaghan & Wiggins, Metamuse Episode 49, Software Longevity, 31:20, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=2S81RLe9Q4mA36etUZN7QA
[Wiggins] The version thing is also making me think of, I think it’s the image file format, maybe tiff is the one I'm thinking of, but this approach has been used in a lot of different places, and I feel like you're doing a version of this in the local-first sync infrastructure we’re working on right now. And so you can add new chunks to a stream of data or to a file that older versions of the software can load, recognize they don't know this chunk and they can skip past it and just try to interpret the rest of it the best they can. Maybe this is something the web browsers do pretty well which is, you load a newer page in an older web browser, if there's some not supported features it doesn't completely break, it just does its best to render it. […] It's not that it just gives up the moment it sees something it doesn't understand. [McGranaghan]: And this gets into the peculiar challenges of longevity with distributed system software. We've been talking about longevity in terms of supporting the past, but with distributed systems you gotta support the future because the future arrives unevenly. So that's the sort of challenge you deal with in your working on distributed systems like the data synchronization layer from Muse. [Wiggins]: As usual Ink & Switch has some good research on this subject with the data lenses in the Cambria project. One of the things there is about data migrations, but it's assuming you need to translate both ways. You need to translate back in time the older versions and forward in time to newer versions, but in fact, it's actually more complex than that, because there may be just a branching tree of systems that all understand that the data in slightly different ways. Wiggins & McGranaghan, Metamuse Episode 49, Software Longevity, 34:55, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=A2dedJMzR4KhKIgM8COa2g
[Wiggins]: […] For archival purposes I have put a lot of effort in the past to preserving things I've created, whether it's a essay, or back when I used to make music, […] usually I find the kind of flattened artifact, read-only, is really kind of what I want in the long term because I do go back to reference those things or look at them for inspiration or just for take a walk down memory lane, but I'm very rarely want to edit, even if I could, and so in that sense just it's actually superior to flatten out to an image or a video or something. [McGranaghan]: Yeah, it's a good point, in many ways, it's a benefit, not a downside, especially when you have this habit of building up an archive that's all on these handful of formats. Everything is basically text, PNG, PDF, maybe video, MP3, you can browse it all together and you can do it very quickly, basically, in Finder or the equivalent, whereas you can imagine, if for every picture you want to check out you had to open up Adobe Photoshop or whatever, you’d wait 2 minutes, so it's nice to have that very lightweight, archival copy. Wiggins & McGranaghan, Metamuse Episode 49, Software Longevity, 38:21, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=ttBllaQQTES8dUyfX6kGMA
[Wiggins]: And the other small thing, it's sort of obvious in some ways, it's kind of the Lindy Effect of file formats. So, Lindy effect here being sort of the idea that something that’s been around a long time probably will be around a long time. So, I'm always a big fan of those flat file formats, PNG, JPEG, PDF, plain text, because they've proven themselves to be durable over the long term and they don't always do everything you need but where you can, it's really nice to just go down to that simple common format that is understood by many applications both now and in the past and potentially will be in the future. Wiggins & McGranaghan, Metamuse Episode 49, Software Longevity, 36:41, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=ozNX-91xRhKag6u_febOKw
In the fullness of time, the files you create are more important than the tools you use to create them. Apps are ephemeral, but your files have a chance to last. The ancient temples of Egypt contain hieroglyphs that were chiseled in stone thousands of years ago. The ideas hieroglyphs convey are more important than the type of chisel that was used to carve them. The world is filled with ideas from generations past, transmitted through many mediums, from clay tablets to manuscripts, paintings, sculptures, and tapestries. These artifacts are objects that you can touch, hold, own, store, preserve, and look at. To read something written on paper all you need is eyeballs. Today, we are creating innumerable digital artifacts, but most of these artifacts are out of our control. They are stored on servers, in databases, gated behind an internet connection, and login to a cloud service. Even the files on your hard drive use proprietary formats that make them incompatible with older systems and other tools. […] If you want your writing to still be readable on a computer from the 2060s or 2160s, it’s important that your notes can be read on a computer from the 1960s. You should want the files you create to be durable, not only for posterity, but also for your future self. You never know when you might want to go back to something you created years or decades ago. Don’t lock your data into a format you can’t retrieve. These days I write using an app I help make called Obsidian, but it’s a delusion to think it will last forever. The app will eventually become obsolete. It’s the plain text files I create that are designed to last. Who knows if anyone will want to read them besides me, but future me is enough of an audience to make it worthwhile. Steph Ango, File over App, https://stephango.com/file-over-app
[Wiggins] […] one of the things that points to that to me feels like maybe the most fundamental is to just think about longevity in the first place and just care about it in the first place. Part of the reason I thought this topic would be an interesting one.I think, again, tech industry skews (?) young, I think also had the same gap in my thinking as a younger person, which was I just didn't have the longer experience to be able to see what it is like for software that I've created that's in production for three years, five years, ten years, and longer and what happens over time. It's a part of what I like with the SQLite long-term support page, they basically lead with a statement that our goal is to support this until the year 2050 and so, will they achieve that? hard to know… But just by making that statement, calling it out as a value for themselves and thinking about it actively, maybe this comes back to the long-now clock idea, which is, they may or may not succeed but certainly one good way to not succeed is to not think about it in the first place. And so this makes it a first-class concern. Wiggins & McGranaghan, Metamuse Episode 49, Software Longevity, 30:14, https://open.spotify.com/episode/1DCNy94TBt3aoGbM8PNlRv?si=UT2qfBJAQWWjr5QsUGkunw
Local-first and software longevity
An important aspect of data ownership is that you can continue accessing the data for a long time in the future. When you do some work with local-first software, your work should continue to be accessible indefinitely, even after the company that produced the software is gone. “Old-fashioned” apps continue to work forever, as long as you have a copy of the data and some way of running the software. Even if the software author goes bust, you can continue running the last released version of the software. Even if the operating system and the computer it runs on become obsolete, you can still run the software in a virtual machine or emulator. As storage media evolve over the decades, you can copy your files to new storage media and continue to access them. On the other hand, cloud apps depend on the service continuing to be available: if the service is unavailable, you cannot use the software, and you can no longer access your data created with that software. This means you are betting that the creators of the software will continue supporting it for a long time — at least as long as you care about the data. Although there does not seem to be a great danger of Google shutting down Google Docs anytime soon, popular products do sometimes get shut down or lose data, so we know to be careful. And even with long-lived software there is the risk that the pricing or features change in a way you don’t like, and with a cloud app, continuing to use the old version is not an option — you will be upgraded whether you like it or not. Local-first software enables greater longevity because your data, and the software that is needed to read and modify your data, are all stored locally on your computer. We believe this is important not just for your own sake, but also for future historians who will want to read the documents we create today. Without longevity of our data, we risk creating what Vint Cerf calls a "digital Dark Age." Some file formats (such as plain text, JPEG, and PDF) are so ubiquitous that they will probably be readable for centuries to come. The US Library of Congress also recommends XML, JSON, or SQLite as archival formats for datasets. However, in order to read less common file formats and to preserve interactivity, you need to be able to run the original software (if necessary, in a virtual machine or emulator). Local-first software enables this. Martin Kleppmann, Adam Wiggins, Peter van Hardenberg, and Mark McGranaghan. Local-first software: you own your data, in spite of the cloud. 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!), October 2019, pages 154–178. doi:10.1145/3359591.3359737
In the fullness of time, the files you create are more important than the tools you use to create them. Apps are ephemeral, but your files have a chance to last. The ancient temples of Egypt contain hieroglyphs that were chiseled in stone thousands of years ago. The ideas hieroglyphs convey are more important than the type of chisel that was used to carve them. The world is filled with ideas from generations past, transmitted through many mediums, from clay tablets to manuscripts, paintings, sculptures, and tapestries. These artifacts are objects that you can touch, hold, own, store, preserve, and look at. To read something written on paper all you need is eyeballs. Today, we are creating innumerable digital artifacts, but most of these artifacts are out of our control. They are stored on servers, in databases, gated behind an internet connection, and login to a cloud service. Even the files on your hard drive use proprietary formats that make them incompatible with older systems and other tools. […] If you want your writing to still be readable on a computer from the 2060s or 2160s, it’s important that your notes can be read on a computer from the 1960s. You should want the files you create to be durable, not only for posterity, but also for your future self. You never know when you might want to go back to something you created years or decades ago. Don’t lock your data into a format you can’t retrieve. These days I write using an app I help make called Obsidian, but it’s a delusion to think it will last forever. The app will eventually become obsolete. It’s the plain text files I create that are designed to last. Who knows if anyone will want to read them besides me, but future me is enough of an audience to make it worthwhile. Steph Ango, File over App, https://stephango.com/file-over-app
In the fullness of time, the files you create are more important than the tools you use to create them. Apps are ephemeral, but your files have a chance to last. The ancient temples of Egypt contain hieroglyphs that were chiseled in stone thousands of years ago. The ideas hieroglyphs convey are more important than the type of chisel that was used to carve them. The world is filled with ideas from generations past, transmitted through many mediums, from clay tablets to manuscripts, paintings, sculptures, and tapestries. These artifacts are objects that you can touch, hold, own, store, preserve, and look at. To read something written on paper all you need is eyeballs. Today, we are creating innumerable digital artifacts, but most of these artifacts are out of our control. They are stored on servers, in databases, gated behind an internet connection, and login to a cloud service. Even the files on your hard drive use proprietary formats that make them incompatible with older systems and other tools. […] If you want your writing to still be readable on a computer from the 2060s or 2160s, it’s important that your notes can be read on a computer from the 1960s. You should want the files you create to be durable, not only for posterity, but also for your future self. You never know when you might want to go back to something you created years or decades ago. Don’t lock your data into a format you can’t retrieve. These days I write using an app I help make called Obsidian, but it’s a delusion to think it will last forever. The app will eventually become obsolete. It’s the plain text files I create that are designed to last. Who knows if anyone will want to read them besides me, but future me is enough of an audience to make it worthwhile. Steph Ango, File over App, https://stephango.com/file-over-app
Information Storage
But why should things be saved? Everything is deeply intertwingled. We save for knowledge and nostalgia, but what we thought was knowledge often turns to nostalgia, and nostalgia often brings us deeper insights that cut across our lives and very selves. Ted Nelson, Computer Lib/Dream Machines, The Heritage
Computers offer an interesting daydream: that we may be able to store things digitally instead of physically. In other words, turn the libraries to digital storage (see Hypertexts); digitize paintings and photographs (see "Picture Processing);even digitize the genetic codes of animals, so that species can be restored at future dates (see "The Hitiest Computer,"). Digital storage possesses several special advantages. Digitally stored materials may be copied by automatic meanB; corrective measures are possible, to prevent errors from creeping In— i.e., "no deterioration" In principle; and they could be kept in various places, lessening man- kind's dependence on its eggs being all in one basket (like the Library at Alexandria, whose burning during the occupation of Julius Caesar was one of the greatest losses in human history). But this would of course require far more compact and reliable forms of digital storage than exist right now. Nevertheless, we better start thinking about it. Ted Nelson, Computer Lib/Dream Machines, The Heritage
Paper moulders. Microfilm is inconvenient. In the best libraries it takes at least minutes to get a particular thing. But as to linking them together - footnoting Aeschylus with Marcus Aurelius, linking genetic data to 15th-century accounts of Indian tribes - well, you can only do it on paper by writing something new that ties them together. Isn't that ridiculous? When you could do it all electronically in seconds? Ted Nelson, Computer Lib/Dream Machines, Hypertext