{"id":2510,"date":"2026-10-30T12:41:51","date_gmt":"2026-10-30T12:41:51","guid":{"rendered":"https:\/\/www.oubliette.org\/blog\/?p=2510"},"modified":"2026-10-30T23:03:26","modified_gmt":"2026-10-30T23:03:26","slug":"the-last-place-you-look","status":"publish","type":"post","link":"https:\/\/www.oubliette.org\/blog\/index.php\/2026\/10\/30\/the-last-place-you-look\/","title":{"rendered":"The Last Place You Look"},"content":{"rendered":"\n

This<\/a> post from Raymond Chen – about helping a customer figure out why running a program through a script didn’t work but launching it by double-clicking it in Explorer did – triggered a flashback, and a bit of PTSD.<\/p>\n\n\n\n

Prepare for a trip in the waaaaay back machine.<\/p>\n\n\n\n

In 1996 I joined the newly-formed \u201cDisney Online<\/a>\u201d as a systems guy. I built and maintained internal networks and network services – file services, email, that sort of thing.<\/p>\n\n\n\n

The Disney web properties, such as they were at the time, were hosted on a few “big” (for 1996) Sun SPARC machines. A lot of the site was dynamic, written in Perl.<\/p>\n\n\n\n

[Before you snicker, it was 1996<\/strong>. And there was one notable exception to the \u201cperl cgi\u201d architecture – the Disney Store Online, where you could buy things like customized Winnie the Pooh balloon bouquets. Thanks to some ex-NeXT folks, like Kevin Wells, the store was written in WebObjects – developed on intel boxes running NeXTStep<\/a> and deployed on OpenStep<\/a> on Solaris. I\u2019d used NeXT hardware in labs at uni, but Disney was where I first wrote “real code” in Objective C<\/a>, a language I’d still choose over C++ every day and twice on Sunday.]<\/p>\n\n\n\n

One day, Disney Online decided to hook up with Microsoft and signed a deal<\/a> to use their tech. It was mostly, IIRC, about the forward-looking “Disney’s Daily Blast,\u201d but it also meant migrating “the main site” (disney.com) from Solaris and Netscape Enterprise Server (I think, it was a long time ago) to Windows NT (3.1) and Internet Information Services (IIS).<\/p>\n\n\n\n

A plan was made to replace the SPARCs with (I’m not kidding) a few racks<\/em> of Compaq-made Windows boxes hosted in the UUNET<\/a> data center in Reston<\/a> Virginia.<\/p>\n\n\n\n

\u201cPorting\u201d the site (i.e. adapting the code for ActivePerl<\/a>) had gone ok, and in the days long before good off-the-shelf tools for graceful transitions, the webmasters fiddled DNS records and TTLs to start pointing live traffic at the new infrastructure.<\/p>\n\n\n\n

Things would seem fine for a while, but boxes would start mysteriously becoming unresponsive. The web developers were trying to figure out what was going wrong, and not making much progress. My bosses boss<\/a>, knowing I was “reasonably good at figuring shit like this out,\u201d asked me to help get the site stable by going to the data center in Virginia and working with the devs in California. We agreed that once I’d fixed this we\u2019d have a discussion about why back office folks weren\u2019t included in the bonus program, and I got a one way ticket to Reston, VA.<\/p>\n\n\n\n

(This was all happening in late November\/early December. Dawnise and I were getting married just before Christmas. She was, let’s say, “not thrilled” about the one way, no definite return date aspect of all this.)<\/p>\n\n\n\n

I got to Virginia and started poking around. Machines wouldn\u2019t die on a predictable schedule, and once they were unresponsive to web requests there was a really<\/em> short window before they became unresponsive to their console. I managed to get logged into a couple before they went totally tango-uniform and all I could initially tell was that there were a ton of processes running – and they looked like perl scripts that were part of the site. If it was a *nix machine I\u2019d have said it had been \u201cfork bombed<\/a>.\u201d<\/p>\n\n\n\n

To add icing to the cake, I don’t think we had remote power (PDU<\/a>) control “back then,” so if the machine couldn\u2019t be rebooted before it went totally catatonic, someone had to go physically push a reset switch.<\/p>\n\n\n\n

I spent a couple long days (and nights) in that data center, talking to the engineering teams in California, testing different theories. Nothing seemed to be helping.<\/p>\n\n\n\n

And then, magic happened.<\/p>\n\n\n\n

I left myself logged in on one of the machines, and our UUNET (or Compaq, I forget) account rep wandered over to it and started clicking around.<\/p>\n\n\n\n

And that<\/em> machine stopped crashing.<\/p>\n\n\n\n

What the actual fuck<\/em>?<\/p>\n\n\n\n

I asked her (she wasn\u2019t super technical, but she definitely wasn\u2019t dumb) to show me what she\u2019d done.<\/p>\n\n\n\n

She hadn’t knowingly “done” anything, but we took another machine, and I watched over her shoulder as she \u201cdid it again.\u201d<\/p>\n\n\n\n

And as soon as she \u201cdid it\u201d I was confident I understood the problem.<\/p>\n\n\n\n

She was clicking around through the site\u2019s files on disk and double-clicked a perl script (the same one, to her credit, she thought she\u2019d double clicked on the other machine).<\/p>\n\n\n\n

Explorer popped up the \u201cthere\u2019s no association for this file type, what do you want to use to open it\u201d dialog.<\/p>\n\n\n\n

And she picked something.<\/p>\n\n\n\n

That was it.<\/p>\n\n\n\n

You\u2019d be totally forgiven if your response is \u201cwhat? huh?\u201d<\/p>\n\n\n\n

What I realized in that moment was that the site had been authored on *nix. So, of course, when a perl script needed to run something else it just\u2026 ran it, using popen(), or system() or whatever. It called some perl function that ultimately fork\u2019d and exec\u2019d the target, waited for it to exit, and returned its exit status. <\/p>\n\n\n\n

The code gave that function the command line it wanted run, and assumed the thing was runnable.<\/p>\n\n\n\n

That was true even if the thing it wanted to run was another perl script, which of course was runnable, because it started with shebang<\/a> (aka. “bang magic\u201d).<\/p>\n\n\n\n

But bang magic relies on a bunch of stuff that\u2019s pretty *nix specific – all the way down. And was not a thing on Windows. Still isn\u2019t.<\/p>\n\n\n\n

So when a perl script \u201csystem\u2019d\u201d an actual executable (like ImageMagick<\/a>, which has been a swiss army knife jabbed in the side of the internet for basically forever) it was fine. Windows would launch the exe, the exe would do its job and exit (or crash and exit) and life would continue.<\/p>\n\n\n\n

The hypothesis I formed in that moment was that in the absence of any other mechanism, IIS was using the file extension type mappings in the Registry (exposed and managed by Explorer) to figure out how to run things that weren’t natively executable. But in this case there was no type mapping<\/em>, and there was no GUI to ask for one.<\/p>\n\n\n\n

I wasn\u2019t sure what exactly would happen, but I thought there was a pretty good chance I was seeing it, and it wasn\u2019t good.<\/p>\n\n\n\n

So I picked another machine, and associated Perl scripts with ActivePerl, and waited. <\/p>\n\n\n\n

And it didn\u2019t crash.<\/p>\n\n\n\n

And in the days before fleet management tools for Windows (cue jokes about how these days are still those days), my next step was to log in to each machine and lather, rinse, repeat.<\/p>\n\n\n\n

And just like that, the fleet was stable.<\/p>\n\n\n\n

I stayed another day to make sure it wasn\u2019t a fluke, and survived reboots and redeploys before flying home.<\/p>\n\n\n\n

While I’d been away, the guy who’d asked me to help had sent some flowers and a nice potted ficus to my fiancee.<\/p>\n\n\n\n

We had that ficus for years. <\/p>\n","protected":false},"excerpt":{"rendered":"

This post from Raymond Chen – about helping a customer figure out why running a program through a script didn’t work but launching it by double-clicking it in Explorer did – triggered a flashback, and a bit of PTSD. Prepare for a trip in the waaaaay back machine. In 1996 I joined the newly-formed \u201cDisney … Continue reading “The Last Place You Look”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[7,5],"tags":[],"class_list":["post-2510","post","type-post","status-publish","format-standard","hentry","category-i-hate-computers","category-life-the-universe-and-everything"],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/www.oubliette.org\/blog\/index.php\/wp-json\/wp\/v2\/posts\/2510","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.oubliette.org\/blog\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.oubliette.org\/blog\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.oubliette.org\/blog\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.oubliette.org\/blog\/index.php\/wp-json\/wp\/v2\/comments?post=2510"}],"version-history":[{"count":7,"href":"https:\/\/www.oubliette.org\/blog\/index.php\/wp-json\/wp\/v2\/posts\/2510\/revisions"}],"predecessor-version":[{"id":2517,"href":"https:\/\/www.oubliette.org\/blog\/index.php\/wp-json\/wp\/v2\/posts\/2510\/revisions\/2517"}],"wp:attachment":[{"href":"https:\/\/www.oubliette.org\/blog\/index.php\/wp-json\/wp\/v2\/media?parent=2510"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.oubliette.org\/blog\/index.php\/wp-json\/wp\/v2\/categories?post=2510"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.oubliette.org\/blog\/index.php\/wp-json\/wp\/v2\/tags?post=2510"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}