Moving data in a coordinated and controlled fashion can be an elusive goal for many organizations. Digitalization of business is increasing the importance of moving data effectively. Legacy attempts to improve file transfers have created a patchwork of batch files, scripts, and use of file transfer tools that offer some improvement, but create silos of automation that are difficult and inefficient to maintain. Companies need a consolidated and coordinated approach to automation. Workload Automation (WLA) tools bring many advantages to centralizing the management of file transfers. Enterprise WLA tools provide cross-platform visibility, permissions control, file security, SLA awareness, compliance reporting, and other best practices that benefit the management of file transfers. EMA believes that modern, enterprise-class workload automation tools are the most logical place to manage and control repetitive workflows, including file transfers. A WLA tool helps an organization learn to speak a common automation language and builds skills that reduce errors, speeds development, and removes repetitive, low-skilled work, which increases productivity and reliability in defining automated tasks.
Moving data efficiently and effectively has always been a problem for IT. As soon as there were two computers there was a need to move data between them. Once data was in motion, there was an opportunity for things to go wrong. Over time networks became faster, protocols improved, and tools improved, yet many of the same challenges remained. Those challenges are still prevalent today. Transfers fail to start, they fail mid-way, restarts are not always clean, files can be corrupted, and files may arrive with no one noticing. Data security is an issue whenever data is in motion. Even for files that arrive intact in the right place at the right time, there are challenges with write permissions, naming issues, and reporting and compliance concerns.
The challenges are increasing as the size and number of files are continuously increasing, and so is the speed with which files must be moved. File transfers are becoming more important as businesses focus on digitalization. There is no time for failed transfers or interruptions because processing times are very tight. Reliability is a key concern, and the interconnected nature of business means interruptions can be far-reaching, impacting customers and trading partners. To ensure consistent and secure data processing, organizations need to confirm governance and compliance with controls, security checks, and comprehensive reporting.
A patchwork of legacy tools such as batch files, scripts, time-based schedulers, and file transfer management tools have been employed to tackle the problems with moving data. These independent layers of automation can create complexity while only marginally solving the problems. While some improvements result from using these methods, they create new problems as silos of automation build up over time. Seemingly low-cost, custom-made solutions can be quick to deploy, but ongoing maintenance and integrations with other systems add cost over time. Cost is always a concern, but should be viewed through a strategic lens. Such solutions do not provide the flexibility needed to adapt to an ever-changing digital business landscape. A morass of poorly documented unique processes do not create a common language or common processes across an organization. Companies need a consolidated and coordinated approach to automation.
Collecting and transferring data from production systems is a repetitive function, whether it occurs daily, overnight, hourly, or even sweeping data every few minutes from online systems. IT has long had answers to handling repetitive functions – from batch processing, to job scheduling and now to Workload Automation (WLA) tools. WLA tools are built to manage repetitive functions with consistency and control. They consolidate the automated controls within a common language in a self-documenting environment, and provide visibility with support for audit logging and reporting. Global variables and job templates allow for reusability and make updates to hundreds or thousands of file transfer workflows a quick task. These tools matured from basic job scheduling to supporting complex triggers to launch automated functions on demand. Triggers can be time-based events driven by custom calendars, or event triggers such as messages written to log files, notifications of a process starting, stopping, or becoming unresponsive, events sent by third-party applications, or a file being created, updated, or deleted.
WLA tools provide an easy interface to define automated actions as many have hundreds of predefined actions and structured screens and wizards to walk users through needed information. They support global definitions that make future changes more efficient, because these definitions can be changed in one place and apply to all actions that make use of them. Retries and other auto-remediation features can be defined to help attain predictable and repeatable results. Integrations with multiple systems are predefined, tested, and often certified. Where such integrations do not exist, WLA tools have APIs allowing easy integration wherever web services are supported. EMA research shows that 75% of those using APIs to integrate WLA tools are extremely or very satisfied. The status of automated actions can be monitored through central dashboards, and proactive alerts and notifications through email or SMS messaging keep all stakeholders apprised of status. EMA finds that 83% of organizations take advantage of dashboards to allow business units to monitor their own workloads. This results in coordinated visibility and control as well as improved time to resolution when an issue occurs.
File transfers are often supplying data to mission critical processes. WLA tools were built to handle mission-critical business processes. They provide reliability and scalability while meeting security, governance, and auditing requirements. Staff members responsible for defining automated actions learn a common language and gain proficiencies over time that are transferrable across the organization. Executing file transfers through workload automation software provides a single point of control and visibility to reduce risk and quicken issue resolution. This results in supporting uniform policies across file transfers, downstream jobs, and other automated actions. With the inclusion of predictive analytics, WLA tools can predict job completion times and warn of downstream issues when files have not arrived or file sizes are larger than normal. EMA finds that 88% of organizations find value in using predictive analytics, with 85% stating that it helps to better align IT with the business. WLA tools can be aware of service-level agreements (SLAs), monitoring compliance and warning of problems that endanger meeting such agreements. Enterprise-class WLA tools are the logical place to centrally manage file transfers.
One of the leading WLA tools in the EMA WLA Radar Report is ActiveBatch Enterprise IT Automation by Advanced Systems Concepts, Inc. (ASCI) (advsyscon.com). ActiveBatch puts users in control of file transfers with powerful and intuitive cross-platform and cross-application scheduling, automation, alerting, and auditing capabilities. With ActiveBatch, users can move scheduled transfers of any size based on workflow, operations rules, and corporate policy. It supports Managed File Transfer (MFT) through both job steps as well as events. Workflows can be triggered based on incoming files, including via an integrated FTP Event Trigger. Files can be encrypted and transferred securely using a variety of protocols, including FTP, SFTP, FTPS, MFT, SSL, SSH, Web Tunneling, and more, as well as support for OpenPGP.
ActiveBatch includes hundreds of production-ready actions for common file system operations like MoveFile, CopyFile, For-Each-File, IfFile, adding directions, renaming files with date and time stamps, and more. Built-in checks can determine whether or not files are successfully transferred (where supported by the FTP server). Content is included for connecting and disconnecting, creating or changing remote directories, downloading or uploading files with filters based on file size and/or relative or absolute time, listing files, moving files, and deleting files. The MFT User Account Object securely stores keys, host IP, protocol and encryption types, restart options, and other reusable information, and can be associated with one or multiple file transfer workflows. This allows managing and updating file transfer workflow information in one place for hundreds or thousands of workflows. Automatic transfer restart options enable the start of transfers upon error detection and prevents workflows from beginning with incomplete or empty files by ensuring a file is fully populated and closed before a workflow begins.
ActiveBatch empowers operations teams with faster development, more reliable maintainability, and extended control of automated MFT and related workflows. Advanced filtering allows users to restrict file transfers based on factors like relative or absolute age of a file, or file size. Advanced parallel transfer capabilities increase throughput. ActiveBatch can handle enormous workloads consistently across a distributed, extended enterprise, including its partners and customers. With strong authentication and granular, role-based access control capabilities, there is clear separation of duties as operations personnel manage transfers without access to sensitive information. SLA tracking and monitoring is aware of the success of a transfer, performance, effect on the business, and adherence to policy, compliance, and security practices. ActiveBatch is the right choice to support complex business-to-business file transfer and workflow integration, with native support for major enterprise-level applications, such as databases, ERP, CRM, and ordering and transaction systems.
To keep pace with evolving digital business needs, companies depend heavily on the successful flow of critical information in files. For business processes to run smoothly, they must rely on secure, compliant, and consistent file transfers. Many organizations look to homegrown FTP and scripted-transfer processes to better manage file transfers. Some deployed specific MFT tools. While these solutions solve some problems, they create independent pockets of automation with challenges to maintain them over time and offer little gain in proficiency. Staff must learn multiple, unrelated tools that lack centralized visibility, control, or governance and audit capabilities.
EMA believes that modern, enterprise-class workload automation tools are the most logical place to manage and control repetitive workflows, including file transfers. WLA tools are built to support the consistent operation of mission-critical business processes. In the age of digitalization of business, file transfers have become key to many mission-critical business processes. Enterprise WLA tools provide cross-platform visibility, permissions control, file security, SLA awareness, compliance reporting, and other best practices that benefit the management of file transfers. A WLA tool helps an organization learn to speak a common automation language and builds skills that reduce errors, speeds development, and removes repetitive, low-skilled work, which increases productivity and reliability in defining automated tasks. Managed File Transfer, coordinated through an enterprise-class WLA tool, enables secure automated file transfers from a central interface, integrating them with other applications and activities, and increasing convenience, visibility, and control.
Founded in 1996, Enterprise Management Associates (EMA) is a leading industry analyst firm that provides deep insight across the full spectrum of IT and data management technologies. EMA analysts leverage a unique combination of practical experience, insight into industry best practices, and in-depth knowledge of current and planned vendor solutions to help EMA's clients achieve their goals. Learn more about EMA research, analysis, and consulting services for enterprise line of business users, IT professionals, and IT vendors at www.enterprisemanagement.com or blogs.enterprisemanagement.com.