Data Center Automation

An Introduction to Data Center Automation

Over time, organizations have placed increasingly heavy demands on their IT departments. Although budgets are limited, end users and other areas of the business rely increasingly on computing resources and services to get their jobs done. This situation raises the important issue of how IT staff can meet these demands in the best possible way. Despite the importance of IT in strategic and tactical operations, many technical departments are run in an ad-hoc and reactive way. Often, issues are only addressed after they have ballooned into major problems and support-related costs can be tremendous. From the end-user standpoint, IT departments can never react quickly enough to the need for new applications or changing requirements. Clearly, there is room for improvement.

This guide explores data center automation—methods through which hardware, software, and processes can work together to streamline IT operations. Modern data center challenges include increasing demands from business units with only limited resources to address those demands. This guide focuses on topics in the following major areas:

• Business processes and frameworks—The fundamental purpose of IT is to support business operations and to enable end users and other departments to perform their functions as efficiently as possible. IT departments face many common challenges, and various best practices have been developed to provide real-world recommendations for ways to manage IT infrastructures. From a business standpoint, the specifics include establishing policies and processes and implementing the tools and technology required to support them.
• IT as a service provider—The perceived role of IT can vary dramatically among organizations. One approach that helps IT managers better meet the needs of users is to view IT as a service provider. In this approach, the "customers" are end users that rely upon IT infrastructure to accomplish their tasks. This method can help in the development of Service Level Agreements (SLAs) and IT processes and better communicate the business value that IT organizations provide.
• Agile management—Modern IT environments are forced to constantly change in reaction to new business requirements. In the early days of IT, it was quite common for network administrators, server administrators, and application administrators to work in isolated groups that had little interaction. These boundaries have largely blurred due to increasing interdependencies of modern applications. With this convergence of servers and networks comes new management challenges that require all areas of a technical environment to work in concert.
• Network and server automation—The building blocks of IT infrastructure are servers and network devices. In an ideal world, all these complex resources would manage themselves. In the real world, significant time and effort is spent in provisioning and deploying resources, managing configurations, monitoring performance, and reacting to changes. All these operations are excellent opportunities for labor-reducing automation.

Through each of the topics in this guide, we'll cover important terms and concepts that will enable IT departments to perform more tasks with fewer resources. The importance and value of automating standard IT operations can be significant in data centers of any size. The goal is to significantly lower IT operational expenses while at the same time improving the end-user experience. Whether you're a CIO or IT manager looking for ways to improve efficiency or a member of the down-in-the-trenches IT staff, you'll find valuables concepts, methods, and techniques for better managing your computing infrastructure.

Information Technology Infrastructure Library

The Information Technology Infrastructure Library (ITIL—http://www.itil.co.uk/) is a collection of IT-related best practices. It is developed and maintained by the United Kingdom Office of Government and Commerce (OGC). ITIL was created to address the lack of standard recommendations for managing IT resources. The goal of ITIL is to provide a framework and guidelines that allow IT organizations to deliver high-quality services in a manageable way. The original content was developed in the late 1980s and continues to be updated with improved recommendations to support modern IT environments. The material is copyrighted by the UK OGC and the information is available in a variety of different formats. ITIL has become one of the most popular standards for IT-related best practices worldwide and is currently being used by thousands of IT organizations.

Benefits of ITIL

Many IT organizations tend to operate in an ad-hoc and reactive fashion. They often respond to issues after they occur, leading to problems such as downtime and lower quality of service (QoS). In many cases, this scenario is understandable as IT organizations are often faced with providing increased levels of service with insufficient staff and minimal budgets. Many organizations either cannot afford to provide additional resources to IT departments or cannot justify the reasons to increase investments.

On the surface, this problem might seem very difficult to solve. However, one approach— increasing overall efficiency—can improve IT service delivery without requiring significant expenditures. It is in this arena where the implementation of IT management best practices comes in.

The recommendations included in ITIL were developed based on studies of methods used by successful IT organizations worldwide. These approaches to solving common IT problems have been compiled and organized into a set of recommendations. Although implementing ITIL practices can take time and effort, most organizations will find that the potential benefits clearly justify the cost. The following sections look at some of the potential ways in which implementing ITIL practices can benefit IT operations.

Improving Levels of Service

The quality of an IT organization is often measured by its ability to respond to business-related change requests and to provide reliability, availability, and performance. Many IT organizations do not have an organized process for responding to new issues and requests, and often several of the requests "fall through the cracks." ITIL prescribes ways in which organizations can improve the reporting and management of problems and incidents. It helps IT organizations define how particular problems should be addressed and how to communicate with end users. By better managing these aspects of service delivery, IT departments can often identify potential areas for improvement.

Reducing IT Costs

Many IT departments suffer from inefficiencies that lead to increased costs. Problems caused by lack of communication, poor issue tracking, and ad-hoc changes can add up quickly. Often, IT managers are unaware of the true costs of purchasing capital assets, configuring and deploying new equipment, and maintaining this equipment. ITIL best practices include methods for calculating true costs and for translating this information into business-related terms. This information can then be used to make a strong case for investments in automation and other labor-saving technologies.

Enforcing Well-Defined Processes

Policies and processes are crucial to a well-managed environment. When policies are implemented and enforced, IT management can ensure that issues are dealt with consistently and completely. ITIL recommendations provide suggestions for designing and implementing successful processes.

Often, it seems that no matter how quickly responses are handled, users' expectations are higher. Through the use of SLAs, IT departments can communicate to users the type of response they should expect for various problems. Developing an SLA is easier when service delivery is managed through clearly defined processes.

ITIL Framework Content Organization

The total amount of information that is conveyed by the ITIL encompasses hundreds of pages. To present the information in a more manageable way, this guide divides it into eight sets, each of which focuses on a specific portion of the total framework. Figure 1.1 provides an overview of the different sets and how they're related. The most important point is that the box on the left represents business requirements, and on the right is the actual technology. The ITIL framework focuses on the content in between—the ways in which technology can be used to meet business goals.

Figure 1.1: An overview of the ITIL framework.

Each set covers specific topics:

• Service Support (ISBN 0113300158)—An important aspect of IT operations is determining how services are provided and how changes are managed. The beginning of service operations is usually a request for a change from an end user, and the process involves communicating with a service desk. It is the service desk's responsibility to ensure that the issue is documented and eventually resolved. Specifically, this area includes problem management, incident management, configuration management, and Help desk operations.
• Service Delivery (ISBN 0113300174)—Service Delivery focuses on defining and establishing the types of services and infrastructure an IT department must provide to its "customers." Topics include creating SLAs, managing overall capacity, developing availability goals, and determining financial management methods. These topics are particularly useful for identifying the purpose and business value of IT. Both Service Support and Service Delivery are parts of the overall Service Management topic.
• Planning to Implement Service Management (ISBN 0113308779)—Many organizations quickly realize the value of using the ITIL approach and want to know how best to move to this model. As few IT organizations have the luxury of starting completely from scratch, it's important to understand how to migrate to ITIL recommendations. This set provides details about how an organization can develop a plan for implementing the best practices suggested within the ITIL framework. It includes information about justifying the use of ITIL (potential benefits). This area is an excellent starting point for IT managers that are prepared to "sell" their organizations on the value of the ITIL approach.
• Security Management (ISBN 011330014X)—In recent years, computer security has moved to the forefront of issues for technical staff. As businesses store and provide larger amounts of information, protecting that data has become a critical part of operations. This set focuses on best practices for managing security throughout an IT organization.
• ICT Infrastructure Management (ISBN 0113308655)—The term Information Communications Technology (ICT) refers to traditional computer-based resources such as workstations and servers as well as the applications that they run (for example, office productivity suites, accounting packages, and so on). The acronym ICT (which is not widely used in the United States) generally refers to the end purpose of IT infrastructure. This volume focuses on managing computing resources, including network service management, operations management, and the installation and management of computing resources.
• The Business Perspective (ISBN 0113308949)—It is important for both business leaders and technologists to understand the overall benefits that can be provided by IT. This set focuses on ways in which IT can meet requirements through managing changes, establishing business continuity practices, and working with outside help through outsourcing. These topics are all critical to the business value of an IT environment.
• Application Management (ISBN 0113308663)—The primary purpose of IT infrastructure is to support the software that is required by users to perform their job functions. This set covers best practices related to managing the entire application life cycle, beginning with gathering and documenting the business requirements for the software. Although this topic is particularly helpful for organizations that develop custom applications, the practices are also useful for evaluating and implementing third-party products.
• Software Asset Management (ISBN 0113309430)—Managing applications throughout an entire IT environment can be a daunting and time-consuming task. Furthermore, the process must be ongoing as new programs are frequently added or updated. This set describes best practices for creating an inventory of software applications and managing the installed base. The topic enables IT to accurately track licensing compliance and to ensure that purchasing does not exceed requirements.

The content is applicable to many different levels within an IT organization, ranging from CIOs to systems administrators; it can also be helpful for business management professionals. From an implementation standpoint, the ITIL framework is intended to provide a set of flexible guidelines. There is definitely room for interpretation of the specific best practices, and it's up to IT management to determine the best way to implement the recommendations. It is important to note that many of these areas are interrelated and the ideal infrastructure will take advantage of all the best practices presented in the framework.

ITIL Compliance

In some cases, organizations might find that they're already following at least some of the ITIL practices (regardless of whether they have consciously done so). Using ITIL's methodology and recommendations can give structure to these efforts. In other cases, IT departments may be able to benefit greatly from implementing the features of the framework.

Unlike some other business-related standards, there is no official certification or testing process that can "approve" an organization's use of ITIL. It is up to organizations to determine the best methods and approaches for implementing these practices in their environments. There are, however, voluntary compliance certificates. These are known as the Foundation Certificate, the Practitioner's Certificate, and the Manager's Certificate (see Table 1-1 for more information about the certifications).

ITIL Content and Resources

The ITIL content is copyrighted, and can be obtained through books, CD-ROMs, or licensed intranet content. Many online book resellers offer the books and related media (they're easiest to find using the ISBNs listed earlier in this topic).

Table 1-1 provides a list of good online starting points for additional information. Additionally, numerous independent books and papers have been written. Each of these focuses on one or more of the topics presented by the ITIL framework. A web search for "ITIL" or any of the specific content topics will also uncover numerous vendors and publishers that offer related content. In addition, professionals that are looking for more information can join one of many different online forums and professional organizations dedicated to the ITIL methodology.

Table 1.1: ITIL-Related Web Sites.

The Business Value of Data Center Automation

Over time, modern businesses have grown increasingly reliant on their IT departments. Networked machines, multi-tier applications, and Internet access are all absolute requirements in order to complete mission-critical work. However, in many organizations, the clear business value of IT is difficult to estimate. Unlike departments such as sales and marketing, there are often few metrics available for quantifying how IT benefits the bottom line. Part of the reason for this disparity is that IT departments have evolved based out of necessity and have a history of filling a utilitarian role. Instead of presenting clear business value propositions, they tend to grow as needed and react to changing business requirements as quickly as possible. In many cases, this situation has caused IT budgets to shrink even while organizations are placing a greater burden on IT staff. Furthermore, business units often see IT as out of touch with the rest of the business.

To ensure success for modern companies, it's critical that all areas of the business recognize common goals and that all contribute toward achieving them. It's difficult to deny the basic business value of IT departments, but the quandary that emerges revolves around how to measure, quantify, and communicate those benefits to business decision makers. This guide looks at the specific business benefits of IT, including details related to measuring benefits and costs. It then explores how data center automation can help increase the overall value that IT departments provide to their organizations.

Basic Benefits of IT

Practically everything that a business does relies upon the business' underlying computing infrastructure. Accordingly, IT departments' internal "customers" expect a certain level of service. They depend upon IT to perform various functions, including:

• Maintaining the infrastructure—If asked what their IT departments do, many end users would point to the computing infrastructure: setting up workstations and servers, keeping systems up-to-date, and installing and managing software. Reliable and high-performance Internet connectivity has become almost as vital as electricity; without the Internet, many business functions would cease. IT is responsible for implementing and maintaining an efficient infrastructure that supports these requirements.
• Reacting to business changes—New business initiatives often place new (or at least different) requirements on the computing infrastructure. For example, a new marketing campaign might require new applications to be deployed and additional capacity to be added. Alternatively, an engineering group might require a new test environment in order to support the development of a new product. Usually, there is an organized process to be followed whenever an employee starts or leaves the company. These changes often need to be made as quickly as possible and in a cost-efficient manner.
• Troubleshooting—From the Help desk to critical network and server support, the service desk is often the first point of contact with IT for users that are not able to do their jobs. Users rely on these resources to quickly and efficiently resolve any issues that arise.

These benefits of IT generally point to tactical operations—performing maintenance-related operations. When enumerating the benefits of IT, often the first metrics that come to mind are those involving reliability, availability, and performance. Although these are certainly important considerations, they do not necessarily demonstrate the strategic advantage of how IT initiatives and projects can contribute to the bottom line. Consequently, it's easy to simply look at IT as just a cost center. Regardless of whether end users realize it, IT departments do much to help their organizations.

Calculating the Value of IT

As with any business department, it's important for management at all levels to see the benefits that are provided to the business as a whole. In some cases, this can be quite simple. For example, there are many metrics that can be used to measure sales and marketing performance. Most organizations realize that in addition to these other business areas, IT is a vital portion of operations.

To calculate the business value of IT, organizations should establish well-defined metrics that reflect the overall business benefit of the computing infrastructure. The information required to do so extends far beyond the boundaries of the IT organization. Instead, it must involve all areas of the business, ranging from end users to executive management. The goal is to demonstrate how IT affects the business.

Identifying Costs

An important consideration for IT management is to be able to calculate and clearly communicate the real costs and benefits of the services that they provide. This identification usually starts with determining the Total Cost of Ownership (TCO) of a specific portion of the infrastructure. Often, when business leaders think of the costs related to increasing capacity, they think only of capital expenditures (such as the purchase price of a workstation or a server). In most environments, however, this cost represents only a very small portion of the total cost. IT departments must add in network-related costs, labor costs (for installation, configuration, and management), software licensing costs, and depreciation.

Often, just the act of collecting this information can provide visibility into an IT department's purpose and structure. It can also be very useful for identifying areas of improvement. Most importantly, however, when true costs are communicated, other areas of the business can begin to understand how their operations affect the overall finances of the company.

Discovering Business Benefits

Members of IT organizations have a tendency to think of the value of their services from a technical standpoint. It's easy to look at server racks or performance statistics as evidence of a job well done. However, the best measurements of the value of IT involve the real impacts these measures have had on the business. For example, suppose that a new test lab environment has helped the Quality Assurance department reduce testing time by 25 percent; this metric is an important one for business leaders to recognize. Similarly, if the implementation of new antispam measures have increased productivity (if through nothing else than decreasing the negative productivity impact of spam), it's important to capture this information.

Enumerating business benefits requires strong communications with other areas of the organization. A good first step is to identify which areas of the business are directly benefiting from technology. IT leaders must understand how new infrastructure components such as servers and workstations are being used. They must be sure that the implemented solutions closely fit the problem. Based on this data, establishing metrics related to employee productivity and business results (such as sales improvements) can be directly tied back to IT initiatives and projects.

Communicating Strategic Business Value

Once the costs and benefits have been identified, business and technical leaders can realize how IT performs a strategic function—not just an operational one. In order to communicate strategic business value, IT departments should focus on overall business goals. For example, a key goal for a software company might be to reduce development time and shorten release cycles. The implementation of a new server or technology (such as server virtualization) can often provide dramatic benefits. If mobile sales personnel are having problems entering orders while on the road, improvements to the network infrastructure and better training might help alleviate the pain.

Improving the Business Value of IT

Once IT is identified as a critical part of business operations, the entire organization can work together to improve overall value. This step often starts with the planning phases for new projects and new initiatives. When business leaders can clearly see the benefits of investing in their IT departments, overall business performance improves.

Decisions should be based on cost-benefit analysis calculations. Although this task might seem simple on the surface, it can actually require a significant amount of information. The costs related to processes should be as accurate as possible and should be based on capital asset costs (including servers, workstations, network devices, and software) as well as personnel costs. Additionally, there can be many "hidden fees," including opportunity costs. Because IT resources are often stretched to the limits, a new project or initiative might result in labor and resource reductions for normal operations. All of these costs and potential tradeoffs should be clearly communicated to business decision makers so that they can make informed decisions about projects. When taking on new projects and initiatives, departments can work together with IT to determine the best approach.

The Value of Data Center Automation

So far, we've seen how a major component of overall IT costs and overall service levels relate to labor. It takes time and effort to maintain even small IT environments, and these factors can clearly affect the bottom line. One initiative that can provide clear benefits and a quick return on investment is data center automation. Data center automation solutions can dramatically reduce charges for one of the most expensive resources—labor. Tools and features that allow for automated deployment, provisioning, change management, and configuration tracking provide an excellent payoff.

For example, a common challenge for most IT environments is that of keeping systems up to date. Managing security patches and other software changes can easily use up large amounts of time. Furthermore, the process tends to be error-prone: It's easy for systems administrators to accidentally overlook one or a few systems. Through the use of data center automation, the same tasks can be performed in much less time with far less involvement from IT staff. This provides numerous benefits, including freeing systems administrators to work on other tasks. Often, automation increases the server-to-administrator ratio and reduces the amount of time required to perform operations. Other benefits include improved consistency, the enforcement of policies and processes, and improved security. Additionally, by implementing best practices (such as those provided with the ITIL), efficiency and operational reliability can improve.

The bottom line is that data center automation can significantly improve the business value of IT. By reducing costs and removing data center-related bottlenecks, data center automation enables IT and business leaders to focus on more important tasks. The entire organization will be able to react more quickly and surely to changes, providing both strategic and tactical advantages to the entire enterprise.

Implementing Charge-Backs

A major problem for some IT organizations is that various departments often compete for infrastructure resources such as new servers or workstations. IT managers are often in the difficult position of deciding which projects are approved based on their limited resources and budgets. This can lead to an adversarial relationship and to some less-than-ideal decisions.

One potential solution is to implement a system of charge-backs. In this system, the IT department would pass costs for various projects back to the departments that request them. The charges would affect these departments' bottom lines. The idea is that business leaders will be much more judicious in their decisions when they directly experience the costs to the business. Although implementing and managing charge-backs can increase administration overhead, the overall cost savings can justify it. Of course, in order for this system to be successful, cooperation from the entire organization must be obtained.

Enabling Better Decisions

IT can leverage business value data to help the entire organizations make better decisions. For example, when considering ways in which to improve organizational efficiency, IT initiatives can play a pivotal role in controlling costs and adding capabilities. A well-managed IT department will have standards and processes in place to ensure that all aspects of the environment are properly managed. This can help answer important questions, such as "Are resources being allocated optimally?" and "Are the right projects being worked on?" With this new view, businesses can clearly see the IT department as a strategic partner instead of just a cost center.

Service Provider

Modern organizations often rely upon many vendors and outside resources to meet business objectives. For example, a marketing group might recruit outside talent to develop a Web site or to work on creative aspects of a new campaign. Alternatively, engineering groups might rely on outsourcing to contractors or consultants to build a portion of a product. IT departments, however, are often seen as cost centers that provide only basic infrastructure services. By treating IT departments as service providers, however, a strategic relationship can be established, and IT can be seen as a business partner.

Benefits of Operating IT as a Service Provider

The value of a service provider is often measured by its abilities to help its customers reach their goals. In this arena, customer service is most important. By having IT departments serve its customers in this arrangement, both can work together to ensure that the best projects and solutions—those that provide the most value to the individual business units—are delivered.

When IT works as a service provider, it should act like an independent business. Its "customers" are the end users and departments that it serves, and its "products" are the services and technology solutions that are provided for use by the customers. Although this concept might at first seem like a strange approach for an internal department, there are many potential benefits. First, IT services are better communicated so that end users know what to expect (and what to do if expected service levels are not met). Second, all areas of the business can see how IT operations are helping them achieve their objectives.

Implement the Service Provider Model

There are several aspects that must be taken into consideration before an internal IT department can be seen as a business partner. This section will look at parts of the overall approach of becoming a service provider to internal customers.

Identifying Customers' Needs

A good salesperson will always work hard to determine a customer's needs. If he or she truly believes in their products, they can quickly identify which are relevant and which will provide the best benefit. For IT as a service provider, this process can start with meetings with individual department leaders as well as end users that might have specific requirements. The overall goal for the service provider is to focus on the business goals of the customer, and not on technology itself.

The first step is to identify the primary purpose of the department. This includes details related to how success is measured and approaches to achieving the success. The details will likely differ dramatically between, for example, sales and engineering organizations. Next, it is important to identify current "pain points"—problems or limitations that are reducing the level of success. Based on this input, IT service providers can develop proposed solutions that address those issues.

As with a pre-sales effort, it's important for IT to gather as much information as possible early in the game—well before any implementation has been discussed. During this phase, it's important to identify functionality that is absolutely required as well as items that are not required but would be nice to have. If there is any ambiguity at this point, details and risks should be identified. Important high-level questions to ask include whether the benefits justify the costs and whether business demands truly present a need for the solution.

Determining "Product Pricing"

IT organizations should come up with complete pricing for their products and solutions. This pricing scheme should include details related to capital asset charges (including hardware, software, and network costs) as well as labor-related costs. Higher costs might be incurred by using non-standard hardware or software. Presenting such costs will help the customer determine whether a particular solution is cost-effective for their department and whether it benefits the organization as a whole. Additionally, other factors (such as a lack of personnel or when other high-priority projects are underway) should also be communicated to the customer.

Identifying Service Delivery Details

Once a customer has agreed to purchase a specific product or service from the IT department, it's time to look into the implementation details. It's important to identify the key stakeholders and to establish points of contact on the IT side and on the customer side. The goal should be to identify who is responsible for which actions.

Milestones should be designed and mutually agreed upon before moving forward. Also, processes for managing changing requirements will help eliminate any surprises during the implementation of the solution. For larger projects, a change management process should be created, complete with approval authority from the customer and the service provider.

Measuring Service Levels

An IT service provider can create products of various types. Some might be closed-ended initiatives, such as the installation of a Customer Relationship Management (CRM) solution, or the expansion of a development test lab. In those cases, service levels can be measured based on milestones and the quality of the implementation. Stakeholders can sign off on project completion just as they would with external vendors.

Other products might involve expected levels of service. For example, when new servers and workstations are added, customers should know what type of response to expect when problems occur. Service Level Agreements (SLAs) can be instrumental in developing mutually agreedupon expectations. For less-critical systems, longer turnaround times might be acceptable. For mission-critical components, greater uptime and quicker response might be justified. Of course, those services will likely come at a higher cost because they will involve additional staff allocation, the purchase of high-availability solutions, and other features.

Prioritizing Projects

All businesses are constrained with limits on their amount of production, and IT departments are no exception. Based on labor capacity and technical constraints, only some of the proposed projects might prove to be feasible. In the traditional IT arrangement, departments often have to compete for infrastructure resources. Often IT departments are faced with the difficult situation of deciding which projects should continue and which simply cannot be taken on.

However, when IT works as a service provider, the vendor and customer can work together to determine what is best for the business overall. If a particular implementation is extremely costly, both can decide to hold off until more resources become available. However, if multiple projects are similar and efficiency can be gained by combining them, the business will see an overall benefit.

Network Configuration Management

When things are working properly, most users barely realize that the network is there. But when network problems cause downtime, the costs to business operations can be tremendous. Still, IT organizations are faced with the difficult task of managing increasingly complex and distributed networks with limited staff and resources.

Although configuring and managing network devices is a task of critical importance, it can be very difficult to perform accurately and consistently. Network Configuration Management (NCM) refers to the use of an automated method to configure and manage network devices throughout an IT environment.

NCM Tasks

The act of managing the components of a network can place a significant burden on IT staff. The process starts with the deployment of new routers, switches, firewalls, and other devices. New hardware has to be purchased and configured before it's brought online. The deployment must be tested, and network administrators must verify that it is working according to the network guidelines. And that's just the beginning.

Maintenance operations include regularly updating to the latest available security patches. Other routine maintenance functions involve changing passwords and updating configurations. Business-related changes can often require significant upgrades or modifications to the network infrastructure and adding capacity is a regular task in growing organizations. The goal of configuration is to rapidly respond to change requests that range from opening a single firewall port to redesigning entire subnets—without introducing new problems to the environment.

Configuration Management Challenges

There are many challenges that are related to managing network configurations. Some of these challenges include:

• Making configuration changes—In all but the smallest of network environments, the time and effort required to manually modify configuration settings on dozens or hundreds of devices can be a tedious, time-consuming, and error-prone task.
• Enforcing processes—In many IT environments, it's very easy to perform technical operations in an ad-hoc manner. Due to the stress and pressure of reacting to business demands, network administrators often take shortcuts and directly make modifications. Although this can lead to what seems like a quick response, it can lead to serious problems in network configurations later. Clearly, processes must be enforced.
• Adhering to best practices—Network security best practices include frequently changing passwords, ensuring that patches are applied quickly, and monitoring devices for suspicious activity. Often, due to time and resource limitations, these tasks are lowered in priority. However, ensuring consistent configurations and adhering to change control processes are critical for reliability of the network.
• Communication and coordination—Network administrators might understandably make a change to resolve an urgent situation. Once the situation is resolved, however, they might fail to communicate this to their peers. Should a problem occur in the future, this can complicate tracking down the root cause of the issue. Distributed administration can also cause problems. Although it's often necessary for multiple network administrators to have access to the same devices, when two or more administrators modify a device, they may inadvertently overwrite the other's changes. Such "collisions" can lead to complex problems that are difficult to troubleshoot.

Regardless of the amount of work involved, IT departments are often limited in labor resources to perform these tasks. That is where automation steps in.

NCM Solutions

Automated NCM solutions can help address many of the challenges related to maintaining a network infrastructure. The key feature of an automated NCM solution is that all modifications are made through the system. Ideally, network administrators do not have direct access to the actual device configurations themselves. All modifications must occur based on a specific workflow and changes are tracked for later review (see Figure 1.2).

Figure 1.2: Configuration management using an NCM solution.

Benefits of Automating NCM

The list of benefits related to using an automated NCM solution is a long one. Specifically, NCM solutions provide:

• Improved efficiency—Manually configuring routers, switches, and other devices can take a significant amount of effort. Some important changes might simply require too much time and effort and may never be performed at all. Automated solutions can handle changes for hundreds of devices without requiring manual intervention. Network administrators can use the time that they save to focus on other tasks that better use their skills. The end result is that the network infrastructure can be more reactive to business changes, and costs can be lowered.
• Policy enforcement—In a manually managed environment, it's up to each individual to be responsible for adhering to processes. It's often difficult to remember all the processes, and in some cases, network administrators might take shortcuts. Related problems can be difficult to isolate and resolve. Through the use of automated configuration management, IT managers can be assured that all changes are coordinated, tracked, and done in accordance with the defined policies.
• Automated network discovery—Modern networks tend to be very complex and have hundreds or even thousands of devices that must be accounted for and managed. It's understandably easy to overlook important pieces of the infrastructure. Automated solutions aid in the process of collecting information and can store and display data about the environment without requiring a manual scavenger hunt. This setup helps prevent surprises when managing the entire environment.
• Improved security—Neglecting to keep network infrastructure devices up to date can lead to security violations or reliability issues. Automated NCM solutions can quickly identify and resolve any maintenance- or configuration-related problems according to company policies.
• Configuration consistency—When dealing with complex environments, consistency is an important factor. Without automation, it's very easy for human error to creep into configuration files. Ad-hoc changes are difficult to detect, and a less-than-ideal configuration may persist for months or years. In the worst case, the configuration problem will be detected only after a security violation or downtime is experienced. Making the change process easier can also avoid putting off important modifications simply because of the amount of effort required. The improved responsiveness means that significant changes can be performed with minimal disruption to the business.
• Backup and recovery—Network device configurations can be complex and vital to the proper operations of a business. An automated configuration management tool can regularly collect configuration information for an entire network environment and store it securely. In the event of a device failure, the configuration can be quickly restored, reducing downtime and the loss of setup details.
• Monitoring—Network performance is a critical aspect of business operations in many environments. NCM tools can regularly measure performance statistics throughout an environment and can report on any potential problems—often before users even notice delays.
• Auditing and reporting—Various business processes can benefit from visibility into the entire infrastructure that is supported by an organization. Auditing allows network administrators to compare the intended configuration of a device with its actual configuration. For organizations that must adhere to regulations such as the SarbanesOxley Act or the Health Insurance Portability and Accountability Act (HIPAA), auditing can significantly help in proving compliance. Any inconsistencies can be quickly identified and resolved. Additionally, reporting provides IT managers with the ability to gain insight into what has been deployed along with how it's being used.

Choosing an NCM Solution

The many benefits of using an automated NCM solution are difficult to overlook, but this leads to the question of how to choose the best product. First and foremost, an NCM solution should allow IT managers to define and enforce change control policies and processes. The solution should ensure that changes can be made only by authorized individuals and only after the appropriate steps (such as peer review) have been taken. All changes should be tracked, and the solution should provide ways for auditing settings to ensure that everything is working as desired. The solution should also provide for automatic backup and restore of configuration data—automatic backups so that the latest authorized configuration is always in safe storage, and automated restore to, for example, roll back after a failed change deployment or after an unauthorized change deployment.

Organizations should look for solutions that use a centralized configuration management database and that allows for tracking details of other computing resources such as workstations and servers. With all these features, the otherwise difficult task of maintaining a network infrastructure can become simpler and much more efficient.

Selecting an NCM solution can be complicated. Your business' own requirements—both technical and procedural—will define the exact feature set you need. However, there are some requirements for which most businesses and organizations can find common ground:

• Security—An NCM solution should provide granular, role-based security so that each individual using the system can have exactly the permissions they need and no more. This includes the ability for auditors (for example) to review configuration information but not to make changes. Authentication should be centralized and, when appropriate, integrated with any existing directory or authentication (such as TACACS+) service you have in place.
• Configuration repository—The solution should provide a version-controlled repository, enabling the retrieval of past versions of a device's configuration. Capture of configuration data into the repository should be made as part of change deployments, on a regular basis, and on-demand.
• Logging—All activity should be logged. This should include pass-through Telnet or SSH activity, where such logging usually takes the form of keystroke logging so that all administrative activity can be properly audited.
• Workflow enforcement—If your company has a process for managing change, the solution should help enforce that process. For example, solutions should be able to enforce a peer review or managerial approval requirement before allowing changes to be deployed.
• Notification—Full notification—of unauthorized changes, successful deployments, and other events—capabilities should be built-in to the solution. Such capabilities help alert your IT staff to problems that need their attention or to recent events that might require manual follow-up or verification.
• Configuration policies and remediation—When possible and desirable, you might want a solution that is capable of analyzing device configurations and comparing them with a standard configuration template or policy. By alerting you to nonstandard configurations, the solution can help you identify devices that do not meet, for example, security or compliance requirements. Automated remediation goes a step further by automatically reconfiguring non-compliant devices to meet your configuration standards.
• Configuration comparison—The solution should provide the ability to compare different versions of a device's configuration, visually highlighting differences for quick identification and review.
• Automation—When possible, the solution should respond automatically to configuration events such as reconfigurations that occur outside the solution. This support might derive from syslog or Simple Network Management Protocol (SNMP) monitoring or through other means.
• Multiple vendor support—A solution should support, of course, every brand and model of device you have in operation. Further, the solution should be architected in a way that facilitates easy addition of additional device support, helping make the solution "future proof."

By using these broad requirements as a starting point, you can begin to identify key features and capabilities that are important to your organization and conduct pilot programs and product evaluations to locate products and solutions that meet your specific needs.