This post from Raymond Chen – about helping a customer figure out why running a program through a script didn’t work but launching it by double-clicking it in Explorer did – triggered a flashback, and a bit of PTSD.
Prepare for a trip in the waaaaay back machine.
In 1996 I joined the newly-formed “Disney Online” as a systems guy. I built and maintained internal networks and network services – file services, email, that sort of thing.
The Disney web properties, such as they were at the time, were hosted on a few “big” (for 1996) Sun SPARC machines. A lot of the site was dynamic, written in Perl.
[Before you snicker, it was 1996. And there was one notable exception to the “perl cgi” architecture – the Disney Store Online, where you could buy things like customized Winnie the Pooh balloon bouquets. Thanks to some ex-NeXT folks, like Kevin Wells, the store was written in WebObjects – developed on intel boxes running NeXTStep and deployed on OpenStep on Solaris. I’d used NeXT hardware in labs at uni, but Disney was where I first wrote “real code” in Objective C, a language I’d still choose over C++ every day and twice on Sunday.]
One day, Disney Online decided to hook up with Microsoft and signed a deal to use their tech. It was mostly, IIRC, about the forward-looking “Disney’s Daily Blast,” but it also meant migrating “the main site” (disney.com) from Solaris and Netscape Enterprise Server (I think, it was a long time ago) to Windows NT (3.1) and Internet Information Services (IIS).
A plan was made to replace the SPARCs with (I’m not kidding) a few racks of Compaq-made Windows boxes hosted in the UUNET data center in Reston Virginia.
“Porting” the site (i.e. adapting the code for ActivePerl) had gone ok, and in the days long before good off-the-shelf tools for graceful transitions, the webmasters fiddled DNS records and TTLs to start pointing live traffic at the new infrastructure.
Things would seem fine for a while, but boxes would start mysteriously becoming unresponsive. The web developers were trying to figure out what was going wrong, and not making much progress. My bosses boss, knowing I was “reasonably good at figuring shit like this out,” asked me to help get the site stable by going to the data center in Virginia and working with the devs in California. We agreed that once I’d fixed this we’d have a discussion about why back office folks weren’t included in the bonus program, and I got a one way ticket to Reston, VA.
(This was all happening in late November/early December. Dawnise and I were getting married just before Christmas. She was, let’s say, “not thrilled” about the one way, no definite return date aspect of all this.)
I got to Virginia and started poking around. Machines wouldn’t die on a predictable schedule, and once they were unresponsive to web requests there was a really short window before they became unresponsive to their console. I managed to get logged into a couple before they went totally tango-uniform and all I could initially tell was that there were a ton of processes running – and they looked like perl scripts that were part of the site. If it was a *nix machine I’d have said it had been “fork bombed.”
To add icing to the cake, I don’t think we had remote power (PDU) control “back then,” so if the machine couldn’t be rebooted before it went totally catatonic, someone had to go physically push a reset switch.
I spent a couple long days (and nights) in that data center, talking to the engineering teams in California, testing different theories. Nothing seemed to be helping.
And then, magic happened.
I left myself logged in on one of the machines, and our UUNET (or Compaq, I forget) account rep wandered over to it and started clicking around.
And that machine stopped crashing.
What the actual fuck?
I asked her (she wasn’t super technical, but she definitely wasn’t dumb) to show me what she’d done.
She hadn’t knowingly “done” anything, but we took another machine, and I watched over her shoulder as she “did it again.”
And as soon as she “did it” I was confident I understood the problem.
She was clicking around through the site’s files on disk and double-clicked a perl script (the same one, to her credit, she thought she’d double clicked on the other machine).
Explorer popped up the “there’s no association for this file type, what do you want to use to open it” dialog.
And she picked something.
That was it.
You’d be totally forgiven if your response is “what? huh?”
What I realized in that moment was that the site had been authored on *nix. So, of course, when a perl script needed to run something else it just… ran it, using popen(), or system() or whatever. It called some perl function that ultimately fork’d and exec’d the target, waited for it to exit, and returned its exit status.
The code gave that function the command line it wanted run, and assumed the thing was runnable.
That was true even if the thing it wanted to run was another perl script, which of course was runnable, because it started with shebang (aka. “bang magic”).
But bang magic relies on a bunch of stuff that’s pretty *nix specific – all the way down. And was not a thing on Windows. Still isn’t.
So when a perl script “system’d” an actual executable (like ImageMagick, which has been a swiss army knife jabbed in the side of the internet for basically forever) it was fine. Windows would launch the exe, the exe would do its job and exit (or crash and exit) and life would continue.
The hypothesis I formed in that moment was that in the absence of any other mechanism, IIS was using the file extension type mappings in the Registry (exposed and managed by Explorer) to figure out how to run things that weren’t natively executable. But in this case there was no type mapping, and there was no GUI to ask for one.
I wasn’t sure what exactly would happen, but I thought there was a pretty good chance I was seeing it, and it wasn’t good.
So I picked another machine, and associated Perl scripts with ActivePerl, and waited.
And it didn’t crash.
And in the days before fleet management tools for Windows (cue jokes about how these days are still those days), my next step was to log in to each machine and lather, rinse, repeat.
And just like that, the fleet was stable.
I stayed another day to make sure it wasn’t a fluke, and survived reboots and redeploys before flying home.
While I’d been away, the guy who’d asked me to help had sent some flowers and a nice potted ficus to my fiancee.
We had that ficus for years.