That Wasn’t Really Six Months Just Now

Saying it’s hard to believe that Tuesday will mark our six month anniversary in Luxembourg is so much an understatement that calling it an understatement is an understatement.

And I’ve barely written at all, each time thinking that there’s too much, and somehow hoping it’ll be easier later.

So that’s working out about as you’d expect.

So… Let me ‘splain. No, there’s definitely too much, lemme try ineffectively to sum up…

If you haven’t been following my status updates on Facebook since my last post here in January, here are the highlights:

I bitched a lot about Numericable – my absolutely horrible Internet service provider – and the lack of real competition in that market. They made Comcast^wXfinity seem like the pinacle of customer-centricity and service. It took five months to get to the point where I was positive they weren’t going to ever fix the problem, or even give a crap that there was a problem. At which point I sent them the mandatory certified letter to cancel our contract. Dawnise dropped the gear off at their office yesterday, and I paid the last bill I ever expect to get from them. So that’s done.

The new ISP is the luxembourg state-run telco – P&T – and so far the experience on the customer service side is much better. Unfortunately, they haven’t run fibre up my street yet, and I’m at the very edge of their serviceable DSL range, so the quality of the connectivity hasn’t really improved much.

My parents came over for two weeks – and everyone survived the experience. It was my mom’s first time in Europe, and probably the most walking she’s done in years, certainly since she had both her knees replaced. Their visit was a good excuse to get Nise to Paris – a city she was disinterested in visiting – and we agreed that we’ll definitely be back, but there are other places to see first. The four of us also went to London, which is a city I don’t imagine I’ll ever get bored of.

Hopefully the experience convinced my parents that travel is something they should do more of while they can.

Around the same time, there was an epic adventure to acquire all the bits needed to make fire in our propane barbecue. It seemed like such an easy thing – we bought the barbecue at Auchan and my dad and I assembled it. In fact, we assembled all the patio furniture the day they arrived – it ended up being a great way to fight jet lag.

As I suspected, it didn’t come with a regulator, so we bought one from Cactus Hobbi while grocery shopping.

Then we started looking for propane.

We were told to start at petrol stations. The shell station up the street sent us to the shell station a town over. There we ran into a frazzled and grumpy attendant who didn’t speak any English, and I couldn’t get the point across in french despite several attempts. So we tried Hornbach, and they sent us to the Total station a kilometer away, who didn’t actually have propane. We tried the Total station up the street from the apartment, and after managing to communicate that we needed a bottle of propane, and paying for said thing, we went to the cage only to discover they didn’t have any full bottles.

The next attempt took me to a Total station across town, where I did manage to get a bottle of propane, only to get it home and realize after breaking the seal that the bottle fitting didn’t match the regulator. I tried all the places I could think of that were open – Hornbach again, Batiself, no dice.

The final trip took Nise and I to the Energus store that I had found after it had closed on the prior attempt. Thanks to Nise’s french and my picture of the cylinder we left with what we hoped was the correct regulator and hose clamps. Shortly after getting it all home we had a fire in the barbecue.

We grilled steaks – I’ve never been quite so happy to be able to make fire.

And of course it’s been wet and raining pretty much ever since.

In other news, we’ve managed to expand our social circle a bit. When we were heading back to Seattle earlier in the year, we needed someone to check on the cats for a few days when literally everyone we knew in the country would be gone. Dawnise availed herself of the American Women’s Club mailing list and found someone. We then proceeded to hand this someone the keys to our apartment and leave the country, hoping the cats, the apartment, and all the stuff in it would be in tact when we returned in a couple weeks.

Turns out everything was just fine, and it further turned out that Dawnise managed to choose someone who plays Cards Against Humanity, which is either the funniest or most offensive game you’ve ever played (those aren’t necessarily mutually exclusive).

More recently, our friends Adam and Helen managed to swing through Luxembourg at the tail end of coincident business trips and introduced us to another expat couple who fortuitously like the same game.

So that seems positive.

Not speaking french in a predominantly french speaking country still sucks – a lot. About a month ago I dropped out of the french lessons we were taking as a group. The immersion-once-a-week-for-a-couple-hours format wasn’t working for me, and my inability to practice from the written materials on my own turned the whole thing into one big ball of frustration. We thought about Rosetta Stone, but were turned off by the immersion approach (and the price tag).

Before moving here I would have said (in fact sorta did say, when it comes right down to it) that I’d try living almost anywhere in the first world. Since then I’d say my filter has gotten more discriminatory. Spending a week in Japan made it clear that not sharing an alphabet is a force multiplier on the challenge of not speaking the language. It was fun for a week, but I’m not sure it would stay fun for long. Similarly, as much as I want to be the guy who’s comfortable not speaking the language, and can pickup new languages easily and quickly, it’s turning out I’m not that guy.

Not speaking the language is incredibly isolating – and being functionally illiterate is both frustrating and humbling.

Aside from that, things are going pretty well – we’ve basically figured out a workable routine, the cats are doing well, the guest room has been a hit with those who’ve stayed in it, and we’re starting to do more than just focus on “figuring shit out.”

Oh, and I had horse for lunch the other day. It was good – I’d eat it again. I don’t see what all the fuss is about. If you’re deeply offended that I ate horse, pretend I meant to type goat and slipped.

Our goal for the next couple months is to do some more travel – see some of the places that are easier to get to from here than they were from the states. I’m not sure where the first stop will be, but writing this makes me think I should stop writing this and go plan a trip.

Are We Having Fun Yet?

Every once in a while – every few days, really – Dawnise or I look at each other and say something to the effect of “We live in Europe. Can you believe that shit?”

This afternoon, after spending the morning on a shopping expedition (more on that later) Dawnise followed it on with a question she’d never asked before.

“Are we having fun yet?”

It took us both a moments pause before we decided, not exactly…

It’s not that we hate it here, or even dislike it, it’s just – we ultimately decided – it still feels like we’re trying to get our lives together. We’re not to the “having fun” stage yet.

Which seems like a damn shame – and something we should work on.

Our sofas – which we ordered on our first big furniture trip to Ikea when we arrived – were finally delivered the other day. Apparently they left Dortmund on a truck in time to be delivered on the 5th, then fell into the Bermuda Triangle, and failed to arrive in Belgium. And this didn’t seem to bother Ikea in the least. I contacted them every week or so asking if the driver had been heard from, if they’d sent out a search party, if they’d tried turning on the gps unit in the truck. They didn’t seem to get the irony. Finally, they called and used the age old “we must have had the wrong phone number” excuse – apparently oblivious to the fact that I’d spoken to their customer service about the order – by phone – every week.

But hey, now we have sofas and – thanks to having to buy something to sit on when the sofa’s failed to appear – a couple comfortable chairs. I’m sitting on one of those chairs, computer in lap, tea on the table next to me, looking out the window at the light snow fall as I type. We’ve got plans with the other couples here for dinner tonight, and I’m hoping we don’t end up stranded in a snow drift somewhere before the night’s over.

Speaking of snow – it’s been doing that a fair bit here. Not heavily, but enough to accumulate several inches. I bus and walk to work, and we don’t have to shovel our walk, or driveway, so it’s kinda nice – makes things pretty and doesn’t really affect us much at all.

I mentioned our shopping expedition – Dawnise got word through the American Women’s Club (she’s a card carrying member) about a place called Provencale – which was billed as “like a Costco” and a mere 15 minute drive from the apartment. Their hours are tailored to the trade, so they open at 5am and close – on Saturdays – at 11:30(a). We got up this morning and went over, managed to get a membership application done and a day pass, and ended up not buying anything.

I rekon that won’t last another trip.

The selection was great, and the prices about on par with the hypermarkets. Produce, fish, meat, and all the non-consumables you might need to run a cafe or brasserie. I’d say it’s more like a Cash & Carry (Smart & Final, for you Southern California folks) than a Costco, but it’s still a welcome find, no doubt.

Then we came back to Cactus, our local hypermarket, and did the weeks shopping, which is getting less and less time consuming and frustrating every time we do it.

So things are falling into something of a routine, broken up by a weekend trip to London for Dawnise’s birthday (which was a lot of fun, and a welcome return to being surrounded by English).

This coming Friday I head back to the UK on business: Manchester for a day, then London – where Dawnise will fly over and join me for a couple days – through the week, home the following Friday.

This past Thursday we completed the last bit of immigration process and picked up our residence cards – I now have a “blue card” – meaning I’m authorized for a couple years to work in any country in the EU, and that I no longer need to carry my US passport and residency letter while traveling in the EU. So that’s sorta nifty.

In other news, a bunch of stuff has happened in the US – and I’m surprised at how totally disconnected from itI feel. We (they?) inaugurated a president who’s likely to be completely stymied in trying to solve the major issues facing the country, assuming he tries. The government played political brinksmanship (again) with tax policy that affects the majority of the population far more than it affects the aristocracy that debated it. They’ll play the same brinksmanship game again about paying the countries bills.

I wish I could say the damage didn’t look as bad from out here, but in fact it looks worse.

I was a disenfranchised and disaffected voter before, I shudder to think how I’ll feel by the time I return.

And on that cheery note, I’m off to make another cup of tea.

Further bulletins as events warrant.

Time Flies When You’re in a Foreign Country

It’s hard to believe we’ll have been here three weeks on Tuesday, and while we’re talking about things that are hard to believe – Tuesday really can’t be Christmas, can it?

I’ve started writing this at least once every couple days, and each time I’ve abandoned what meager progress I’ve made. Maybe this time I’ll figure out how to summarize the couple weeks that have passed since my last update….

When last we left our heroes they were generally awash in a sea of French, and had managed to negotiate the purchase and delivery of a clothes washer and dryer.

Washing clothes, it turns out, makes a huge difference in our ability to feel settled in a place. Never mind that loads take two hours to wash. And another ninety minutes to dry. Or that the dryer is a condenser, so you have to empty two lint screens and a water collection vessel after every load. We have clean clothes, damnit.

Since buying the washer/dryer – we’ve purchased pretty much an entire apartments worth of furniture from Ikea, learned that for 40€ they’ll deliver up to 500kg to the curb outside our (4th floor) apartment, and learned that moving 500kg of flat pack furniture up to the fourth floor is serious freaking work.

Especially hoisting the 160cm mattress up five flights of stairs when we discoverd it wouldn’t fit into the lift.

Never mind getting a bunch of that 500kg up the spiral stairs in the apartment. Or putting it all together.

As of now, the bedroom (bed, including mattress and linens, our nightstands and reading lamps, our dressers, an Nise’s armoir), the dining room (table and six chairs), the lounge room (a couple sideboards, coffee table, and sofas that’ll be delivered on the 5th), our dishes, glassware, silverware, dish towels, and nearly everything else non-consumable is Ikea sourced.

And somewhat to my surprise, the place doesn’t feel particularly generic. Ikea has figured out how to provide enough variation in each category that you can mix and match, and feel like you’re putting “your” collection together.

And I have to say Ikea’s designers are nothing short of amazing. Not only coming up with finished pieces that can be assembled with a hex key, a phillips screwdriver, and a mallet, but optimally packaging, and producing nearly idiot-proof assembly instructions without a single word of any language. Seriously impressive.

We’ve seen a movie (Wreck it Ralph), bought a TV, ordered cheese in french from the cheese monger and got what we wanted and managed to sign up for the various hypermarche loyalty cards (ok, those are really Dawnise’s victories, but I’m sharing it vicariously).

We’re getting down to the last few major items we know we need – which is mostly rugs, ’cause every surface in the apartment is hard, and it’s a giant echo chamber.

And that’s good, ’cause were both totally sick of shopping.

Perhaps most impressively, we’ve managed to feed ourselves, which – if you know us – is no mean feat. We realized that we hadn’t seen prepared chicken stock in any of the shops we’d been at – and chicken stock is something of a staple of our diets. Dawnise makes great risotto, starting with chicken stock. And fantastic split-pea and lentil soups – starting with – you guessed it – chicken stock. Turns out the closest we could get was either bullion or fond – so we decided to do what any self-respecting cook would do.

Roast a chicken and make stock.

At which point we learned that the chickens we could find are quite small compared to what we’re used to, but were actually quite tasty as well. (And the roast chicken and veggie stock was a great base for the lardon and mushroom risotto, which we’re totally making again.)

We’ve enjoyed the baked goods, the cured meats, the cheeses, and I’ve quite enjoyed the wine.

And we’ve both managed to lose weight since arriving.

So that’s pretty nice.

We’re keeping our eyes out for a suitable quiche pan. I fear the weight loss won’t last.

Oh, and we’ve gone to Germany. Specifically, we spent our anniversary wandering around Tier and seeing their Christmas market.

Including Belgium (Ikea is just across the border) that’s three countries in as many weeks.

Four for me if you count the day business trip to Paris.

Now that’s what I’m taking about.

Day Three: Wherein our heroes get a shot, buy appliances, and generally fail to comprehend things

Thanks to a sales guy who was willing to use his rough English (still worlds better than my French) we’ve bought a washer & dryer and they’ll be delivered and installed Wednesday morning.

After looking a bit online and comparing prices, we went to Conforama just up rte. d’Arlon. The sales guy recommended Zanussi (a division of Electrolux) saying they offer good value for money, they’re reliable, and if you do need service it’s quick to schedule. He steered us away from some of more expensive *cough*German*cough* brands, mostly because getting service if needed can be a real scheduling hassle.

Unfortunately, the model he pointed us at was only 6kg capacity, and we were pretty sure we wanted at least 7kg. (Actually, we wanted an American I-only-do-laundry-every-couple-weeks-and-cram-it-all-in capacity, but they don’t seem to make those here.) Turns out we wanted *exactly* 7, ’cause going to 8 or more added over 100€. He didn’t have a 7kg Zanussi, so between us we found a suitable model. But he didn’t have any in stock. So he pointed us at a normally more expensive model and said it was on promotion for the same price as the one we picked.

So he entered the order and gave us the charge slip. We did a bit more looking around before heading to the counter to pay. The cashier proceeded rattle of a bunch of french that we didn’t understand, and then we signed something.

I told Dawnise that the form was permission for them to steal our soles.

Subsequently we were in a housewares store in the mall and Dawnise was looking at a hanging spice rack. The sales lady wanted to show her that it rotated the spice at the press of a button and said something in French. Dawnise looked (even to me) that she understood.

Moments after the exchange, I told her “that’s it, she can steal your soul.”

Dawnise lost it and then admonished me for making her snort while laughing in a foreign store.

So that was today. Mostly.

We also saw two doctors, got the first part of the government mandated TB test (the injection) and have to return on Tuesday for chest X-rays.

Oh, and we bought a toaster, a coffee grinder (huzzah!), and a french press.

Banking in Luxembourg – First Impressions

Had the appointment to get a bank account setup this morning. It seemed to go ok, until I realized that the last bank account I opened state-side I did completely online, had immediate access to the funds electronically (once they hit the account) and had my debit card within a couple days.

The process today took the better part of an hour-and-a-half in an office with a person (two, actually), and involved me handing over a small stack of cash which won’t be accessible for two days, and we won’t get our debit cards for two weeks.

I tried their online banking this evening. Every time I login I need to provide two mostly un-memorizable bits of information (a 10 digit client id and a 6 digit secret code) along with three randomly chosen digits from a string of 16 random characters they gave me on a credit card with a silver scratch off strip.

I learned I could go through a lengthy and buggy activation process to replace the last of those three with a one-time password from the (physical) token they gave me, but I gave up when the registration process switched from english to french in the middle.

It gets better – regardless of how much I have in said account, my weekly ATM cash withdrawal limit is 1500€, and I can transact another 1500€ at point of sale per week. If I want to make a large purchase, I can call them and “within a couple hours” they can raise the limit for a week or two.

I’m sure the cashier at Ikea will be happy to hang out for a couple hours while that gets worked out. Note to self: bring American credit cards on our shopping trips.

So I went to the ATM in that branch, after opening an account with that bank, and used one of my US debit cards to withdraw a couple hundred Euro in walking around cash.

I’m beginning to undersand why Europeans save so much more than Americans – they can’t get easy access to their money.

In other news, we (with a bit of help from Irene, our helpful relocation contact) started the registration process with the local government this morning. That was a reasonably short meeting – 45 minutes or so, most of it taken by the state agent making multiple certified copies of (all pages of) our passports.

Tomorrow morning we go for the mandated medical exam, tomorrow afternoon we have someone coming to fix a leaking faucet in the apartment, and hopefully once they leave we have time to go shop for a washer dryer, which I imagine I’ll rant about later.

Finding an Apartment in Two Days

Now that we have a signed lease it seems safe to tell the apartment hunting story…

We had a bit of a rocky start – the company had partnered with a local relocation firm in Luxembourg, and I had started trying to communicate with them a month in advance of our planned trip. Literally the weekend before we were to leave, I finally got contacted by one of their representatives to inform me that the person we thought we were dealing with (but who hadn’t ever been communicative or responsive) wouldn’t be working with us.

Which was fine by me.

We spent a bit of time chatting about our requirements for an apartment – which were accepting of cats, 100 square meters or so, at least two bedrooms, at least 1.5 bath, and preferably walking distance to the centre-ville.

We arrived in Luxembourg Monday evening, and were greeted by an email with a list of a dozen apartments we’d be seeing the following day.

Tuesday morning we breakfasted and met “Irene” – our local contact – outside our hotel around 9am to start the day.

We quickly realized that the only way to get out of this process with useful data was to be ruthless – each property we saw got compared to the other properties we had seen – always keeping a stack-ranked list of our “top three.” The day was long – longer for me than Dawnise, who’s both good at and enjoys house hunting.

We finished the day around 5pm with a confident list of our top three – which we were referring to as “the duplex,” “lord of the flies,” (so-called because there were a bunch of dead house flies in the property when we viewed it) and “the modern.” All were a bit further outside the city than we had hoped – about a 20-25 minute walk – but they met all the other criteria – and the only place we had seen *in* the city was aging and in real need of a refresh.

The following morning we got up to do it all again – only with fewer properties this time. We also asked Irene to call “the duplex” and “lord of the flies” back to arrange second viewings for the end of the day, figuring we’d want to see them again to break any ties. We learned that “lord of the flies” had been rented since our viewing the previous day.

This was our first concrete indication of how quickly the rental market moved.

We kept on the same tact – comparing each new property to the top three – and despite seeing one or two we really wanted to like – owing to their location, mostly (or the one that was super swank, but crazy expensive with the private elevator, hammam in the master bedroom, and trompe-l’œil pantry hidden behind a kitchen cabinet) – none of what we saw Wednesday broke into the top three.

‘Till we saw “the penthouse”.

Literally the last property we were to look at before reviewing our top choice from the previous day, it seemed like a great choice – closer in to the city by a bit, walking distance to a large super market, open plan living space, panoramic views and a terrace that I’m already looking forward to spending spring and summer evenings on. It was bigger than we wanted, and the laundry facilities weren’t in the unit, but Dawnise and I agreed after discussing that it had vaulted into the number one spot.

Given that we’d seen one of our choices get rented from under us, we told the agent that we’d have an answer for her that afternoon.

The only rub was the lease.

Leases in Luxembourg are typically, we were told, three years long – but as a concession to the number of folks in our situation (moving into the country on business, and possibly moving out just as abruptly) they typically include a “diplomatic clause,” which says you can escape the lease without penalty if (and only if) you’re leaving the country. The rub is that recently landlords have been trying to write into leases that the diplomatic escape clause doesn’t apply for the first year or two of the term. This landlord wanted a two year minimum, and that wasn’t acceptable.

Their agent said she’d talk to the landlord and have an answer for us the following morning. Irene chatted with the agent for “the duplex” and arranged for them to hold it for us until the following morning.

So we figured we were in reasonably good shape.

The next morning Dawnise and I breakfasted and set out to wander around the city a bit. By mid morning I was a little nervous that I hadn’t heard anything back from Irene, but I knew she was viewing housing with other folks, so we waited.

And waited.

Until around 1pm we got a call saying that despite repeated efforts to contact the agent for “the penthouse” she’d not heard anything back, and we’d delayed “the duplex” as long as we could. We needed to make a choice. Keep waiting, or take what was behind door number two.

After a quick consult we agreed that we’d be pretty disappointed if we lost both of these possibilities – and that seemed like a likely outcome if we waited. We knew “the duplex” had other interested parties – and we’d seen “lord of the flies” get rented for what we thought, and Irene agreed, was probably a higher than reasonable price. So we called Irene back and told her to take the duplex.

And she called back a few minutes later to tell us it was in process – we’d have the lease draft the next day. So, we thought, there we go.

We chatted about the possibility that “the penthouse” responded late saying they’d accept our terms – would we back out of “the duplex” if we could? We decided we wouldn’t – and we spent the rest of the afternoon (subconsciously) convincing ourselves and each-other that “the duplex” – our second choice – was a better choice than the one that got away.

The following morning we breakfasted in the hotel, gathered our things, and caught a taxi to the airport for our LUX to LHR flight, a couple hour layover, and then the flight from LHR to SEA.

Walking between gates in Terminal 5 in Heathrow I saw an email from Irene asking me to call her, which I did – and she said she’d gotten the information she needed, no worries.

We hung out in the lounge during our layover. At one point we got a call from Irene saying that “the penthouse” got back to her (finally) and would accept our change to the minimum lease duration, did we want to change? No, I said, we’d already discussed it. Stay the course.

As flight time approached we got to our gate, were sitting in our seats waiting for door closure, when the email came in from Irene saying that the landlord of our apartment (“the duplex”) had decided to rent it to someone else, despite our in-process agreement. And we had just told “the penthouse” we weren’t interested.

Oh, shit.

I didn’t have time to call her and try sorting out what had happened, so I forwarded the message to my colleague, Mike, in Luxembourg and asked him to see what he could do.

And then the doors closed.

And we went dark for 9 hours.

And Dawnise managed not go completely insane.

When we got back to Seattle I had email from Mike saying that there was nothing to be done – we had lost “the duplex” – but that “the penthouse” was still available and Mike had started the process of getting lease paperwork from them. We had until Sunday evening our time to make a go/no-go decision.

Rationally, we knew that “the penthouse” had been our first choice going into Thursday – so clearly this was good, right? Only we’d spent most of a day talking ourselves out of it and into “the duplex” and undoing that much self deception isn’t a trivial thing.

It took a little time for the cognitive dissonance to fade, and even longer for the lease to be agreed upon and paperwork actually be signed, but it looks like it’s all worked out.

And yet, I hesitate to post this, wondering what email I’ll get with more unexpected news.

Let me explain. No, there is too much. Let me sum up.

It’s been quite a while (two years) since I’ve posted anything here. The last two years have flown past – work has been busy, we’ve done a bit of traveling (in hind sight, I sorta wish I posted about our trips here), and generally had more good days than bad days.

So why, you might ask, am I resurrecting this blog?

Well, ’cause Dawnise and I are moving in December.

To Luxembourg.

As in that (tiny) little duchy sandwiched between Germany, France, and Belgium.

Valve is starting the process of opening a small office there, and I’m going to help do that.

Dawnise and I have been talking about this for a couple months – since the Valve office plan started really gaining momentum. We spent this past week in Luxembourg looking for an apartment. We found one about 2km outside thecentre ville that will accommodate us, the cats, and some as-yet-to-be-determined subset of our stuff.

When I said small office, I do mean small – there will be three of us initially; one who’s already there, one who’s heading over at the start of November, and me – others may join, and we’ll start looking to hire.

This could very well be one of the craziest things we’ve ever done – neither of us speak french (yet), and my memory of High School German isn’t even enough to get me in trouble anymore. We’ll know a total of five other people in the country, and unlike when we moved to Washington, I won’t be surrounded by an office full of potential friends.

We’ve thought a bunch about this, and there are lots of questions – it’s unclear what my role needs to become in this new setting, and Valves unique structure brings a unique set of challenges to working with folks who are several timezones and thousands of miles away.

There are about two million things that can still go wrong with this stunt.

Dawnise and I have said, several times while traveling in other countries, that we’d both like to experience of living abroad – so when this opportunity came up, it was basically too good to pass by.

We’ve been comparing this to being at the top of a roller coaster – looking over the edge into the drop. Very soon, things are going to start moving very fast.

So if you’re in Seattle – or are going to be in Seattle before the beginning of December – and want to grab dinner or something before we vanish for a year or two, drop us a line.

Wheeeeeee!!!!!

Rest In Peace, Junior

It’s been a while since I’ve made the time to post anything here – not that there hasn’t been stuff going on, just that life’s been, well, life.

This afternoon, Junior left us after a rocky couple of months. His diabetes, which had been under control without insulin for a couple years, suddenly returned, along with a smorgasbord of other symptoms. This afternoon, I walked into the bathroom to find him in the midst of a violent seizure – and he never came back.

He was with us for 15 years, and ingratiated himself with even the most stalwart anti-cat people. We have lots of record of his time here – as Dawnise has pretty much always thought of cameras as existing solely to capture images of the cats.

He will be missed, and the world has within it a bit less cute. Looking out the window, I’m sure the sky looks just a bit more silvery-grey than normal…

Postmortem of a catastrophic RAID failure

Wednesday of last week, I came home to find my three new 1TB hard disks waiting for me, destined to upgrade our ReadyNAS NV+.

Being a hot-plug-online-upgradable-all-singing-all-dancing sort of widget, I followed the recommended upgrade procedure and popped out one of the current 500GB drives, waited a few seconds, slotted one of the new 1TB replacements, waited ’till it started resynchronizing the volume, and went down to make dinner.

And spent the next several days picking up the pieces…

One critical bit of background – the NAS had three disks in a single RAID-5 volume. RAID-5 can tolerate one disk failure without data loss, but if two disks fail (regardless of the number of disks in the volume), kiss your data good bye.

When I went back upstairs after dinner to check on progress I discovered that the NAS had locked up, and completely dropped off the network. Wouldn’t answer it’s web management UI, and wasn’t responding to pings.

Hesitantly, I power-cycled it. It started booting, and hung about a quarter of the way through checking the volume.

After several reboot attempts all locking up at the same place, I applied a bit of coercion and convinced the box to boot. I checked the system logs and found nothing telling, removed and re-seated the new 1TB drive, and watched it start the resync again.

A couple hours later, sync still proceeding, I went to bed.

And woke the next morning to find the unit again fallen off the network.

Buried in the log messages – which I’d left scrolling past over night – was a warning that disk 2 was reporting SMART warnings about having to relocate failing sectors.

In other words, one disk of the three was being rebuilt while another one was busy dying.

At this point it became a race – would the rebuild complete (leaving me with two good disks, and intact data) before the failing one died completely.

In order to try to buy some insurance, I shut down the NAS, transplanted the failing drive into a spare PC, and started a disk-to-disk copy of it’s data onto the working 500GB disk I had removed at the start of this mounting disaster.

Despite valiant attempts by both dd_rescue and myrescue, the disk was dying faster than data could be retrieved, and after a day and a half of effort, I had to face the fact that I wasn’t going to be able to save it.

Fortunately, I had setup off-site backups using CrashPlan, so I had Vince bring my backup drive to work, and retrieved it from him on Friday.

Saturday was spent restoring our photos, music, and email (more later) from the backup.

Unfortunately, despite claiming to have been backing up Dawnises inbox, it was nowhere to be found in the CrashPlan backup set, and the most recent “hand-made” backup I found was almost exactly a year old (from her PC to Mac conversion). Losing a year of email is better than losing everything, but that seems like meager consolation under the circumstances.

By Saturday night I had things mostly back to rights, and had a chance to reflect on what had gone wrong.

The highlights:

1. SMART, as google discovered (and published) is a terrible predictor of failure. The drive that failed (and is being RMAd under warranty, for all the good it’ll do me) had never issued a SMART error before catastrophically failing.

2. In retrospect, I should have rebooted the NAS and done a full volume scan before starting the upgrade. That might have put enough load on the failing drive to make it show itself before I had made the critical and irreversible decision to remove a drive from the array.

3. By failing to provide disk scrubbing (a process whereby the system periodically touches every bit of every hard disk) the ReadyNAS fails to detect failing drives early.

4. While I had done test restores during my evaluation of CrashPlan, I had never actually done a test restore to Dawnise’s Mac. Had I done so, I might have discovered the missing files and been able to avoid losing data.

I have a support ticket opened with the CrashPlan folks, as it seems there’s a bug of some kind here. At the very least, I would have expected a warning from CrashPlan that it was unable to backup all the files in it’s backup set.

5. In my effort to be frugal, I bought a 500GB external drive to use as my remote backup destination – the sweet spot in the capacity/cost curve at the time.

Since I had more than 500GB of data, that meant I had to pick and choose what data I did and didn’t backup. My choices were ok, but not perfect. There’s some data lost which should have been in the backup set, but wasn’t due to space limitations.

6. CrashPlan worked well – but not flawlessly – and without it, I’d have been in a world of hurt. Having an off-site backup means that I didn’t lose my 20GB worth of digital photos, or several hundred GB of ripped music.

Aside from digital purchases, the bulk of the music would have been recoverable from the source CDs, but at great time expense. The photos would have just been lost.

7. In this case, the off-site aspect of CrashPlan wasn’t critical, but it’s easy to imagine a scenario where it would have been.

8. The belief that RAID improves your chances of retaining data is built largely on what I’m going to refer henceforth to as “The RAID fallacy” – that failure modes of the drives in the array are completely independent events. The reality is that many (most?) RAID arrays are populated with near-identical drives. Same manufacturer, same capacity (and model) , and often the same or very similar vintage. So the drives age together under similar work loads, and any inherent defect (like, say, a firmware bug that causes the drives not to POST reliably) is likely to affect multiple drives, which spells disaster for the volume.

Loss

My uncle Alan died last week, on April 1st, after being removed from life support. I’ve sat down to write about it several times since we got home, maybe this time I’ll actually finish…

We got a call last Saturday that he was going to be unplugged on Sunday, with as many family members who could be, and wanted to be, present. I wanted to be there to say good bye, all accounts were that the man was already gone – only the shell remained.

After discovering that flights would have cost us the better part of $1100, we hit the road Sunday morning for the drive down. We got a call from my dad just as we were crossing the OR/CA border that Alan had passed peacefully and quickly after being removed from the respirator.

Alan was one of eight children – he was my mom’s baby brother – and the only one of my mom’s siblings I really know (knew?). Alan and Amy lived near us for most of my life, and I spent a lot of time with them – holidays, parties, and for no particular reason at all. I guess that’s what people mean by “family.”

When Dawnise and I were looking for our first place together, Chino Hills had the double advantage of being vaguely centrally located to our relative commutes, but more importantly – it was close to Amy and Alan. So we rented a condo, and became neighbors. When we bought our first house, it was literally around the corner from them – and we’d invite them to dinner at their place regularly : Amy’s cat allergies prevented us from actually hosting. Dawnise came to think of Amy and Alan as the aunt and uncle she never had.

I only wish that I had been able to tell him how much he meant to me while he was still around to hear it.

The service was well attended – with much laughter during the shared remembrance. As an ER nurse, Alan touched countless lives, and many of his co-workers – and at least one former patient – came to pay their respects. Many folks took the time to share stories and memories of Alan – starting with my mom, and including myself, my aunt Dawn, Dawnise and many others.

My brother couldn’t be at the service – he’s on a 40-odd day wilderness trek through Utah – and he (and my sister) are at least as close to Amy and Alan as I am. I said a few words on his behalf – ’cause he couldn’t be there to say them – and I suspect even had he been there, he’d have had a hard time doing so.

I was told afterwards that I spoke well – but to be honest, I didn’t know what I was going to say when I walked up there, and I had little recollection of what I said when I was finished.

The centerpiece of the service was a photo montage, assembled by Shawna, Laurie, Dawnise and Christopher. I’m working on making it available online for those who couldn’t be there to see it.

My aunt Amy, in the midst of all this, managed to throw a party Alan would have attended – which is exactly what he would have wanted.

One of the songs used in the montage was Tim McGraw’s Live Like You Were Dying. When it came on the radio late Thursday night on our drive home I finally lost it, and sat sobbing in the car while Dawnise drove.

Gone, but never forgotten.