In every industry, digital transformation is creating large amounts of data. Much of this data has to be processed and analyzed to make it useful and then the results stored for longer term use. Organizations are increasingly using public cloud processing and storage to supplement or even replace some of their on premises operations. However, public cloud pricing for various services including processing and egress of data can be complex and confusing and as a result, many customers are shocked by the cost of their off premises operations. Being able to make intelligent trade-offs between performance and cost when using cloud-based resources versus on-premises is an urgent need, but few tools have been available to help IT managers and cloud customers find the right cloud services for their needs, until now.
Uses for Data Movers
Data is the life-blood of modern organizations and moving that data around is critical to the organization’s operations. For that reason, data mover software that controls the migration of data as well as its movement between clouds and on-premises operations is a vital operational tool. This is especially important during the Covid-19 pandemic when most work is being done remotely. These tools must be able to help organizations balance the capital costs for on-premises storage and processing against the operating expenses (OpEx) of storage and processing in the public cloud.
When using remote cloud-based operations, movement of data is required. Data, much of it unstructured data, may be generated locally and needs to be moved to a private data center (private cloud) or to a public data center for analysis and perhaps for longer term storage (including disaster recovery, data protection and archiving). Data movers may also be used to move data out of the cloud or between various remote data centers (multi-cloud operations). The costs for moving data often depend upon the required performance levels, but are ultimately gated by the organization’s network performance.
Data movers are important for moving content between clouds and between clouds and on premises storage for many applications. The media and entertainment industry is an interesting example. Most post production work has only been possible during the Covid-19 pandemic because industry professionals have been able to work remotely, often using cloud-based tools. In this case, using video proxies, collaborative work has continued, often times including international collaboration.
In addition, some popular public cloud providers charge for the movement of data out of their data centers (egress fees). Because of the various tiers of storage performance and data transfer performance as well as egress fees and the increasing use of multiple clouds for various operations, it can be difficult to compare costs for different options for actual data workflow requirements. At the same time, most organizations are more sensitive than ever about the costs of operations and, for a large organization with multiple cost centers, being able to allocate costs to the appropriate cost centers is critical for cost effective operations.
Cloud data services charge differently for various types of storage, depending upon the speed to access the data and for large movements of data they also charge different fees for bulk data movement, standard data movement and expedited data movement. Understanding the tradeoffs between the costs of these services and organizations business requirements is important. In addition, data movers must integrate with cloud native APIs to be optimally effective.
SoDA and IMT
Integrated Media Technologies (IMT), headquartered in Los Angeles, CA and with offices in Boston, NYC and Dallas was founded in 2007. The company provides consulting and systems integration, video collaboration tools and software for performance optimization and security. The company’s customers span media and entertainment companies and data centers, including cloud implementations.
IMT has created its SoDA enterprise software application to streamline the process of intelligent data transfers to and from the public cloud. The company says that SoDA is ideal for working with unstructured data environments, such as those found in media and entertainment and many other industries. SoDA provides predictive, actionable cost and data transfer metrics for optimizing on-premises and cloud storage and is available on a subscription basis.
The image below shows the various capabilities of SoDA to migrate data, report on job operations and manage costs, and allow active cloud data management, including providing insights on performance versus cost. In addition, SoDA has full featured API integration with a RESTful interface and easy and fast integration into existing workflows.
SoDA supports major file-based storage (SMB and NFS) and private and public clouds. It has a policy-based engine to move and analyze data. It’s dry run feature can be used to compare options for data movement in order to predict costs and performance and its real-time and historical reporting of cloud spending can help in cost control. SoDA offers multi-tenancy cost control and billback features.
SoDA is integrated with Amazon Web Services (AWS) as well as with all S3 client object stores. SoDA can be deployed on a virtual machine or on server nodes. The company offers one node S100 servers as well as three node S300 configured servers and offers automated software updates. These services are offered on a monthly subscription service with charges tied to the number of nodes or virtual machines. The cost is independent of the actual data moved. This subscription model, independent of use, provides a predictive cost for using SoDA.
One NFL football organization has relied heavily on video content captured from games, practices and interviews from its 60-year historical archive. In addition, with the move to 4K and 8K video their legacy storage infrastructure could no longer support the performance and storage requirements needed. The team needed a simple and cost-effective way to migrate
off their legacy storage system.
Using SoDA, the team was able to tackle their Tier 1 and cloud migration, leveraging the elasticity of the cloud for working with 4K and 8K content and brought their archive online. Their data migration software was installed and configured on their VM environment. It enabled content migration from their legacy storage to a new high-performance SAN.
SoDA was integrated with Levels Beyond reachENGINE media asset manager (MAM) and provided with automated policy-based data movement from on premises to the AWS cloud. Furthermore, a dry run feature provided insights into cloud storage and retrieval costs. As a consequence, the organization’s active content is now searchable and available to users and the automated migration reduced required IT resources, saving time and money and minimized on premises storage costs by using elastic cloud storage coupled with real-time analytics to control the cloud budget and spending.
A representative from the team’s IT staff said, “We were concerned about data migration from multiple legacy NAS storage systems into a hybrid cloud model. With the data movement software our migration was refreshingly easy. The software connected into our Media Asset Management environment to enable our long-term preservation and archive workflows.”
Electronic sports, or esports using video games, is a rapidly expanding entertainment industry. A company active in Esports company has multiple publicly viewed games and leagues that are inspired by card games, board games and role-playing fantasy (RPF) games. Their media workflows include video of competitions with players, in-game footage and Esports commentary. Media is also needed to create in-game video content.
This company had disparate islands of data with cloud data stored in proprietary formats and workflows were folder and file name-based with data flow to the cloud decoupled from the creative processes. The company could not search for content quickly and easily and the company needed more insight into their cloud costs. The organization also had a large amount of legacy content and artwork they wanted to bring to the Esports gaming community.
IMT helped this Esports company implement IPVCurator with SoDA in a hybrid cloud deployment with archive on ingest into AWS Glacier. SoDA handles data movement to and from the cloud within IPVCurator. Video editing is done using lightweight video proxies which IPV streams into Adobe Premier. As a consequence, Esports gaming videos, legacy game material, artwork and promos are now searchable and available to multiple groups. Editors can now work from home (WFH) and edit via proxy and conform on-premises, increasing employee productivity. The real-time cost analytics help control the cloud budget and spending.
Today’s big data growth and the availability of fast networks is enabling organization to combine on-premises storage and processing with cloud and multi-cloud elastic storage and processing resources. IMT’s SoDA provides an easy to use data movement tool that allows clients to understand and control the costs of data movement to and from the cloud. SoDA has helped organizations free their archives for rapid search and use while optimizing on-premises versus cloud storage and data movement costs.
About the Author
Tom Coughlin, President, Coughlin Associates is a digital storage analyst and business and technology consultant. He has over 39 years in the data storage industry with engineering and management positions at several companies. Coughlin Associates consults, publishes books and market and technology reports and puts on digital storage-oriented events. He is a regular storage and memory contributor for forbes.com and M&E organization websites. He is an IEEE Fellow, Past-President of IEEE-USA and is active with SNIA and SMPTE. For more information on Tom Coughlin and his publications and activities go to www.tomcoughlin.com.