Data Consolidation Phase 3: Implement the New Architecture

With the increased use of high-bandwidth, fiber-optic networks and cloud computing, more and more companies are looking to consolidate their data center operations. In his book Administering Data Centers Servers, Storage, and Voice Over IP, Kailash Jayaswal provides insight regarding the best practices to consolidate data center operations. Phase 3 of data center consolidation consists of implementing a consolidated environment, which is discussed in this section.

After evaluating the existing environment, creating a design proposal, and testing a prototype, and with the revised documentation in hand, it is time to start building the consolidated design. Following are the main steps in the implementation phase:

  1. Draft all low-level specifications.
  2. Construct the new environment.
  3. Create a data migration process.
  4. Back up data in the old environment.
  5. Migrate data and application services to the new environment.
  6. Train the operations staff.

Although all this work seem like a tall order, it is easier than Phases 1 and 2.

Implementation Step 1: Draft All Low-Level Specifications

Write a detailed layout of the environment that will be configured. If equipment will be installed in a to-be-built data center, you must write out the physical environment specifications. For applications that will be migrated to a new server, you must decide and write out the server model and resources, storage configuration, and software to be installed. Nothing can be left out. Think as if someone else will take the documents, build it as outlined, and deliver it to you after a month. The design specification must contain some or all of these:

  • Physical environment such as power circuits, floor space in the data center, racks, and environmental requirements (such as cooling and humidity).
  • A description of the servers, including manufacturer, model, amount of memory, CPUs, adapters, and attached storage.
  • Network specifications for administration, backups, NAS, production, and so forth.
  • Operating system, including patches, network management, and backup software.
  • User applications such as Web server software and databases. Specify the vendor and software release numbers.

The specification document will depend on the type of consolidation, but the more detail you provide, the better, because most of the work may be performed by others.

Implementation Step 2: Construct the New Environment

This is the first act of constructing the target environment and preparing it for data to be migrated over. The building blocks are as follows:

  1. Build the physical environment — Ensure that the data center area allocated for the new environment has adequate floor space, electricity outlets, cooling, and patch panel ports, and enough space to bring in the equipment from the docks. After the equipment is relocated here, there must be enough space around it for staff to work while installing OS and connecting cables.
  2. Build the servers and install the operating environment — Once you have the servers in their respective tile locations, it is time to install the OS. Once, at a customer site, I was involved in installing about 20 servers. Five of those did not boot past their power-on self test (POST). I had to call the vendor, and they had to replace several faulty components over the next three days.
    Use automated OS install programs. They save time and effort. Popular technologies are Linux Kickstart, Microsoft Unattended Installer, Solaris Jumpstart, HP-UX Ignite, and IBM Automated Install Manager (AIM).
    Ensure all the network connections are working well. Set up all routing tables. You must be able to access the server from the backup, administrative end-user networks.
    Mirror the operating system to another local disk. It is a common best practice and saves you from an OS installation and setup after an OS disk failure. Also, install and configure administrative applications such as network management client software and backup client software. It is best to take a full backup. During the ensuing installation and reconfigurations, it may be necessary to revert a few files or the entire OS.
  3. Attach storage subsystem — NAS- or SAN-based storage should be used for storage consolidation. Check storage vendor documentation for the order of installing device drivers (if applicable) and connecting the storage subsystem.
    Like the servers, the storage subsystem must be thoroughly tested and burned-in before configuring logical volumes on the devices and migrating data.
  4. Install application software — Now that the OS and devices are set up, it is time to install and configure the applications. It is best to install them from vendor distribution CDs. Although the same version may be installed on the old server, avoid the temptation to copy entire application directories or configuration files from the old server. But, you must use the old server configuration as a guide. When you must configure several instances of the same application (for example, four Apache instances or two database instances), use non-default names, directory locations, and filenames for each configured instance.

Implementation Step 3: Create a Data Migration Process

Now that the servers, storage, and applications are ready and tested in the new environment, it is time to move all data from the old to the new servers. Data integrity, ease of migration, and minimum time taken and service outage are key objectives.

Data migration is simple if you have a large downtime window or a small data set for a non-database application. NFS mounts, common backup utilities (such as cpio, dump, and tar on UNIX) and transfer programs (such as scp, FTP, and tape backups) work well in such scenarios.

Complex migrations are those where the available downtime to transfer data is very small and for databases that reside on raw logical volumes. Application managers and database administrators must outline the migration process because it usually requires application-specific utilities provided by the vendor.

Database export/import programs are often used to back up an entire instance of a database and later import it on another instance. This is a well-tested, versatile procedure when moving data to a different server, OS, or database version, or another platform.

The migration process must be tested at least three times on a part of the new environment, part of the old servers and storage, or on the prototype. Each test must be monitored and timed, and results must be noted for communicating to staff involved in data migration and administration of the environment.

It is also imperative to create a backout process, should something go wrong. The backout process must include two items:

  • The conditions under which you should stop progress and initiate the backout action plan
  • The steps required to put services back to the old environment

Implementation Step 4: Back Up Data in the Old Environment

After building the new environment and deciding on the migration process, you must make a full backup before going ahead with the service migration. The backup must be made just before you migrate the data and services. It is important that you also test the backup media to verify that you can retrieve data from it. The backup tapes will be required if the same storage media is being attached to the new environment and it gets damaged during migration.

Implementation Step 5: Migrate Data and Services to the New Environment

There are certain prerequisites before data migration. The data and service migration time must be during off-hours and all teams (network, application, database and system administrators, and facilities) must be involved and available within short notice. The time period should be one when no other large-scale work or network activity is scheduled. Network and systems monitoring must all be in place. Migration of data entails three steps:

  1. Migrate data.
  2. Cut-over services to the consolidated environment.
  3. Test services and tune services.

The data migration steps outlined earlier should be followed. The team that tested the steps must be involved in actual migration. OS utilities can be used to transfer files and small directories. It is faster to transfer the database export files over the network, unless the data must come from a remote location. In that case, backup tapes must be shipped in advance and data can be copied to the target environment before the scheduled downtime. Once the data is copied over, test the environment. Tune it as best as possible before directing services over to the new, target environment. Tuning parameters (such as network and I/O settings, kernel parameters, and shared memory settings) must be configured now.

Migrating services involves changing the IP addresses of consolidated servers to those that are used by users. This can be host-based services. Alternatively, you can change service IP addresses in DNS to be the IP addresses of the consolidated servers. This is preferred for services that go to a virtual IP address on a load balancer or a cluster of servers.

Testing and tuning the consolidated services are critical and should be done within the scheduled migration downtime. Monitor server and network load, application behavior, and result integrity. Test the new applications for functional accuracy. If any of these are not up to the desired level and cannot be remedied within reasonable time, it is imperative to start the backout process.

Implementation Step 6: Train the Operations Staff

Training is critical to ensuring high uptime of services from the consolidated environment. However, good documentation is a prerequisite for proper training of personnel that must support the services and hardware. There are two important questions: What level of training should be provided, and when should it be provided?

The level of training should match the environment's setup. Modern servers, storage arrays, and network devices are feature-filled and have layers of complexity, various applications, and their exclusive management interface. Basic training of the hardware configuration, OS, and applications must be supplemented by group meetings and formal knowledge-exchange sessions. Low-level how-to's are more important than architectural concepts.

When should the training be held? Not too soon before the consolidated environment is deployed for live production use. It is ideal to schedule the training sessions about one month before the migration. People will forget the material if taught too much in advance before hands-on usage, and if training is held too close to migration, say a week before, they will not have time to review the training material before delving into the trenches.

Continue to Phase 4: Control and Administer the new architecture.

From Administering Data Centers Servers, Storage, and Voice Over IP by Kailash Jayaswal. Copyright 2006 John Wiley & Sons, Inc. All Rights Reserved. Used by arrangement with John Wiley & Sons, Inc.