- Plan every detail of the move. When we plan we have it down to the minute of who is going to work on which cabinet, when each server will be shutdown, when it will be unracked, moved, racked, cabled, powered up, etc. Be prepared, though, to throw the plan out. Things never go according to plan, so don’t get hung up when things slip. By having gone through the planning process you will be so familiar with what needs to happen you will be able to make good “heat of the moment” decisions.
- Make sure you order food. Not everyone likes the same thing, so make sure you get your order in, otherwise you could get stuck with nothing but cheese or Hawaiian pizza. No offense to the people who actually prefer these foods, but seriously pineapple on pizza?
- Only have one copy of the documentation. If everyone has a different copy of the documentation, throw all of it out and go without it. You won’t be any worse off and at least then you can blame it on lack of documentation.
- Make sure the documentation isn’t on a server that is being moved. Trust me, if it’s on the server you are moving, it won’t do any good.
- Communicate outages clearly and often. One of the worst things that can happen is for one of the VP’s to call 10 minutes after the truck left claiming they didn’t know about it and need to ship a $10 million order.
- Plan for people downtime. I’ve heard the whole “Sleep is for the weak” argument, but at some point you cause more damage than good. Taking a break and getting some sleep will make you more effective.
- Double check the tools and other supplies (moving carts, paper, pencil, tape, etc.) are available. Typically when data centers are moving, no one else is around and all the stores are closed. Getting a roll of tape Tuesday afternoon is easy. At 3:00AM on Saturday in the middle of nowhere is not.
- Go over the process ahead of time. A walkthrough the day before and again an hour before will make sure people don’t get confused when they start putting servers back where they came from.
- Servers take 5 minutes to rack, and 5 minutes per cable with two people. I questioned the time once and the team had me do a cabinet (pre-move, of course). It really does take that long. Also that is 5 minutes per cable. If you have a server with 8 Ethernet ports, 2 power supplies and an out of band management port it will really take an hour. Anything less and it will look awful.
- When planning the schedule ensure people aren’t all working on the same cabinet at the same time. It won’t work.
- Make sure everyone understands the port numbering. If the switches go 1-24 on the top row and some people think it is odd on the top and even on the bottom, you will have problems.
- Verify the documentation before you move. It’s much easier to write down which cable goes to which port when it is still plugged in, than it is to remember where it came from. Have it done once, and then have someone else double check it. It really is that important.
- People-friendly environment. Servers don’t mind the noise and like 65 degree air, people not so much. If you can, turn the temp up and the noise down.
- Separate application test teams. By the end of the weekend you will be tired and can get sloppy. If possible, have a separate team to test the environment.
- Startup order is important. If you try to bring up servers before the domain, or some applications before the database servers, you can cause issues. If there is an order, make sure the people racking and cabling the servers leave them off until they are ready to come up.
- Record issues and lessons learned. Keep track of every problem you run into and what you did to resolve it. This does two things, it reminds everyone how many hurdles the team overcame, and it helps down the road when you see the same problem.
- Setup a conference call and make sure everyone can dial in with cell phones, and mutes them (unless they are talking, of course) to help troubleshoot issues.
- Always have a plan B. If the elevator breaks down, can you really carry the servers up the stairs?
- If you have redundancy or a DR site, use it, but only if it makes sense to. In our case we didn’t have a hot site, but if you do and it is truly redundant, use it and move during the week. It will make the move less stressful and test your DR.
- Sometimes things just break. If you have something that is broken, don’t automatically assume it is move related. It probably is, but don’t assume it.
- If you can, reboot all the servers before the move. This helps find startup issues like patches that were downloaded but not installed.
- Have a priority list. You may run out of time, so make sure you know what applications and servers need to be up.
- All the teams are team members. Many times other departments, contractors or subcontractors will help. Treat them as if they were your own employees. The success of your project depends on them too.
- Celebrate after. Moving a data center is a huge undertaking. Take the time to recognize the team for a great job. You never know when you will have to do it again.
Enterprise Data Center Transformation
In my first post, I talked about the trials and tribulations of building out a new data center. Well, that’s only half the story. In this article, I’ll dive into some practical tips you can use when actually moving a data center from one location to another.
One last tip I’ve learned. If you are the manager for the project or team, be on site and involved in the move, not in the way, and don’t try to help, but get coffee, food, snacks, coil up the old cables, sweep the floor, etc.
Let’s be honest, you probably aren’t that much help, but having you there emphasizes how important the project is. Besides, if it goes really badly at least you know you can sleep in late on Monday. You’re probably going to get fired anyway, might as well be rested.