Jeremy Dennis is Jeremy Day (cleanskies) wrote,
Jeremy Dennis is Jeremy Day
cleanskies

.... and the beat goes on ....

People do this, don't they?

After it's happened. It's supposed to make you feel better.

10.00am I get in, discover to my relief that the site's up, and hopefully I can put the problems of the last few days behind me. I start on a more thorough check to make sure the site's working properly. First hint that something is wrong when the webmaster email doesn't forward to me immediately.

10.30am Sophos pings against a page on my site. Says there's a Nimda-A virus on it. I call up XXXXDELETEDXXXX and suggest, sweetly, that this is an urgent problem. Actually get to speak to one of their engineers! who feels that its fine, and it shouldn't be a problem, as they're currently clearing up and it shouldn't be a problem for more than about an hour. Put the phone down, much relieved. Log onto the home page, get the same message again, read it properly, and think some.

10.35am Considering my options. Phone IT helpdesk to see if I can get some advice. Get through to the Administrator, who promises to tell some quite irrelevant people about my problem and see if they can give me a call maybe. Attempts to get her to let me speak to someone else, anyone else, in the office fail.

10.40am Talk to housemate from heaven, D, who works in a similar field. He agrees with my gut feeling that this needs sorting now, before it does any damage.

10.45am Phone my boss. She agrees that people need warning. But what would the warning say? "Urgent problem -- virus risk on Spired.com, don't log on until further notice?" I Make a few abortive moves towards this then realise that if I'm advising my users not to use my site because it's unsafe then it's unsafe enough I need to take it down.

11.00am Phone XXXXDELETEDXXXX. No-one answers. Screaming fit.

11.10am Phone XXXXDELETEDXXXX.Get through. Talk to the service person. She won't put me through to an engineer. She tells me there are two engineers already working on the problem. She wants to take my number and have someone call me back. I don't leave my number. They already have my number from the previous calls. This is just a delaying tactic.

11.22am Email arrives from the IT Helpdesk that wouldn't talk to me earlier, asking what's being done about the problem, as people are at risk of infection. Fire the following email right back:

"i'm taking the site down. Until it's clear. BRB."

and grab the other phone in the office so I can wait on one line while I hang on the other to XXXXDELETEDXXXX until I get through. Get through. Shout at service person. Get put through to engineer. Thankfully they can follow direct orders. Site goes down.

11.25am Talk to service engineer at XXXXDELETEDXXXX. Answers unsatisfactory. Tell him not to put my site up again without calling me first, so I can test it with him on the line to pull the plug again if necessary. Realise grimly that this company may not survive much longer. Thoughts about getting new host ensue. As I talk, type second email to IT helpdesk:

"Sorry for the abruptness of the previous message.

Because XXXXDELETEDXXXX is hosted externally, I'm not in control when something like this happens. It took me a while to get to them, and when I got to them, it took me a few phonecalls to convince them to just pull the plug on the site.

They seemed sure that the situation was non-urgent, and as I am not a programmer, I went against my better judgement and trusted them. I phoned the [internal] helpdesk but was unfortunately not able to talk to any support staff to get advice.

I'm going to try again to talk to the [internal] helpdesk, so that we can get an informed warning message out to all XXXDELETEDXXXX
users.

All the best"

unaware as yet that I'm talking to the people who wouldn't give any help earlier.


11.30am Call IT Helpdesk, get straight through to senior support staff. Bingo. Explain situation. Realise that I'm talking to the same person as has been emailing me. Explain how I was unable to get any advice earlier. He mumbles something which is not an apology. But, on the bright side, there have been no service calls about infection.

11.40am Cry with relief.

12.00pm Damian calls back to ask how it's going. Am moderately coherent in return. Agree to go to lunch. Need coffee.

12.20pm Start telling people what's happened, starting with my boss's boss's boss (she just happens to be handy). Decide to leave telling anyone else until after lunch, damnit.

12.30pm Lunch.

1.30pm Get back to the office. Finally get that service call from the IT Helpdesk to give me some advice about viruses and the seriousness of the problem. A bit late, I'd say, but at least I can confirm that there have been no reports of infection.

1.50pm Phone up my boss. She asks how I'm doing. "Well," I say, "For starters, the site's down .... "

2.20pm Go looking for the boss I've not told yet. Fail to find her. Uncomfortable about telling her over the phone. Decide to email a few key users instead.

3.00pm Call XXXXDELETEDXXXX ....

(to be continued)

3.00pm Call XXXXDELETEDXXXX. Listen to their tinkly music for a short spell. They're doing something to it right now, apparently. They'll call me back. This time, I leave my number.

3.25pm First report from someone who thinks they've been infected. Still using Outlook, so it can't be too bad. Advise them to start shouting at IT support anyway (she's called them once and they've not got back to her). Better safe than sorry.

3.31pm Decide to see if doing something else for a bit will get them calling me up.

4.00pm Still waiting. My head aches.

4.15pm Decide it is time to rake my zen garden. Am interrupted by a call. It's someone rather more senior-sounding from XXXXDELETEDXXXX. He explains what happened, as briefly as humanly possible, while I click through some pages, checking the site's working OK. I let him go as soon as I'm sure the site is working. I imagine he has many more similar calls to make.

So what had happened?

As Damian had suggested, they had been fitting up a a server, and it had become infected. They had then been trying to clean up all their servers while maintaining service in the face of repeated re-infection. My insistence that my website was taken down was maybe the best thing: it meant they could isolate and clean the server my website is on.

And in the end? I'm relieved, but also exhausted, and angry. I've learnt that even for a terminal problem you have to yell to get it recognised as such, and that the fact that knowledge and advice is there doesn't mean you'll be able to access it when you need it most. I've also discovered the hard way that you'd better be very aware of the risks to your users because even when they're screamingly obvious they are not your host's problem and they won't spot them.

I'm also late with my weekly strip, shaking, I have a half-raked zen garden, and my neck is killing me.

Purely administrative role. That's what they told me, when I took on this job.

Purely administrative role.
Subscribe
  • Post a new comment

    Error

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.
  • 3 comments