We are getting ready to move our data enter yet again. This will be the fourth time we move this particular one so we are getting pretty good at it. It's funny how much goes into one of these that people just don't realize.
A few examples...
1. Every connection needs to be documented. If not you will not know where to plug it in once you arrive at the new data center.
2. Every server location must be documented both where it is currently and where it is going. By location I mean the cabinet and "U location". A typical cabinet has 42 U locations, some servers take 1U some can take 20U. There is a big difference...
3. Then you need to have a server move document that shows where the server came from and goes to, so that it goes on the right pallet. The right pallet is the one that matches the "go to" cabinet. It makes it quicker putting them back in.
4. All the cables at the new location need to be labelled and ideally color coded. It's much easier to find the one cable out of 300 that you need, if only ten of them are red...
5. Print out the labels ahead of time but don't put them on the cables. Otherwise you will spend too much time trying to find the next cable...
6. Unbag and untie all of the cables ahead of time too. It may not sound like much but when you have 1000 cables, even that 30 seconds per cable adds up. In this case to around 6 hours of time...
7. You can't have more than three people working in a cabinet, one in the front and one in the back, maybe one handing the cables. You also need to leave a space in the cabinets you are working on or people will just be in the way....
8. Have a separate test team and troubleshoot team, if possible. After working until Midnight on Friday taking stuff out, and then working 16 hours on Saturday putting them back in, you will be too tired to actually be that effective...
9. Have a priority list, just in cast you don't get to finish everything on time you can at least make sure the most important systems are available as quickly as you can.
10. After you are done, celebrate then figure out what you could have done better. Number 5 is a great example of learning from something that didn't quite go as good as we expected. Everyone likes to think they will only do this once, but never say never.
I'm sure once we are done I'll have some post move tips too, so stay tuned.
Some timing numbers we use...
Setting up a new cabinet (moving it in place, securing it etc) takes 30 minutes.
Racking (or unracking) a server takes 10 minutes
Plugging in one network cable is 5 minutes
Power cables are 2 minutes.
Thursday, January 3, 2013
Wednesday, January 2, 2013
The right tool for the job - netflow
It's been a few weeks since I last posted. With the holidays, moving our corporate headquarters (and 500 people) it's been busy. Plus I didn't have anything insightful to add and I don't want to be one of those folks that feels the need to blog just to blog.
Then as Ray Barone, on "Everybody Loves Raymond" said, "sometimes material presents itself" This past week material presented itself...
In Maine having a "plow truck" is pretty much a requirement. For those not from the area a plow truck is usually an old truck, mostly rusted out that isn't safe to drive on the road with a plow on the front, brakes are optional My plow truck's power steering stopped working along with the heater so I spent a few days working on it.
I hate working on vehicles because, well, I have no idea what I'm doing.... But with the help of the internet I searched around and found out my power steering was leaking at a fitting going to the power steering gear box. It looked like an easy fix to change it so I disconnected the hose and broke the fitting in the process.
Unfortunately I couldn;t get the now broken fitting out. It was too rusted and the wrenches and sockets I had just kept trying to strip the head off and ruin the fitting. I spent all of day 1 trying to heat the fitting, and spraying it with "Liquid Wrench" to try and get it out. Finally when it got too dark and cold to work I went back online and searched for "rusted fitting removal"
Sure enough there were a lot of people with the same problem and it turns out something called an impact wrench seems to be the tool of choice. It turns out I have an impact wrench and after 30 minutes of dragging out the hoses, compressors and fittings I used that to remove the fitting in about 45 seconds. It took about 2 more minutes to put the new fitting in, connect the hose and that problem was fixed.
Having the right tool, and knowing it is the right tool is key. Clearly I had the right tool but didn't know it because I had never used it. I could tell because it was still in the plastic.
Many IT administrators have plenty of tools that they have forgotten about. One of the most powerful ones is netflow. Netflow lets you see every conversation on the network, but is very rarely used. It takes a high end switch to be able to do "full netflow" without impacting performance. I think Enterasys switches are one of the few, if not the only ones, that can do this.If you have it though it becomes one of the tools you use very frequently.
If you aren't familiar with it, check out some of the posts and uses for it here.
Then as Ray Barone, on "Everybody Loves Raymond" said, "sometimes material presents itself" This past week material presented itself...
In Maine having a "plow truck" is pretty much a requirement. For those not from the area a plow truck is usually an old truck, mostly rusted out that isn't safe to drive on the road with a plow on the front, brakes are optional My plow truck's power steering stopped working along with the heater so I spent a few days working on it.
I hate working on vehicles because, well, I have no idea what I'm doing.... But with the help of the internet I searched around and found out my power steering was leaking at a fitting going to the power steering gear box. It looked like an easy fix to change it so I disconnected the hose and broke the fitting in the process.
Unfortunately I couldn;t get the now broken fitting out. It was too rusted and the wrenches and sockets I had just kept trying to strip the head off and ruin the fitting. I spent all of day 1 trying to heat the fitting, and spraying it with "Liquid Wrench" to try and get it out. Finally when it got too dark and cold to work I went back online and searched for "rusted fitting removal"
Sure enough there were a lot of people with the same problem and it turns out something called an impact wrench seems to be the tool of choice. It turns out I have an impact wrench and after 30 minutes of dragging out the hoses, compressors and fittings I used that to remove the fitting in about 45 seconds. It took about 2 more minutes to put the new fitting in, connect the hose and that problem was fixed.
Having the right tool, and knowing it is the right tool is key. Clearly I had the right tool but didn't know it because I had never used it. I could tell because it was still in the plastic.
Many IT administrators have plenty of tools that they have forgotten about. One of the most powerful ones is netflow. Netflow lets you see every conversation on the network, but is very rarely used. It takes a high end switch to be able to do "full netflow" without impacting performance. I think Enterasys switches are one of the few, if not the only ones, that can do this.If you have it though it becomes one of the tools you use very frequently.
If you aren't familiar with it, check out some of the posts and uses for it here.
Subscribe to:
Posts (Atom)