45
loading...
Load, Store, and Protect LinuxBased NFS Workloads in AWS
This article outlines the Linux-Based Network File Systems migration strategy from on-premise to AWS cloud.
Network File Systems (NFS) are widely used in on-premise applications in order to synchronously share the same content over multiple servers. This document describes the best practices for migrating Linux file-based applications to the cloud smoothly, efficiently and securely.
AWS is offering the most suitable way for its customers to see their data transferred to the cloud taking advantage of managed services that were designed to jointly provide a smooth migration process. This process would leverage the capabilities offered by AWS DataSync, Amazon EFS & AWS Backup.
AWS is offering the most suitable way for its customers to see their data transferred to the cloud taking advantage of managed services that were designed to jointly provide a smooth migration process. This process would leverage the capabilities offered by AWS DataSync, Amazon EFS & AWS Backup.
It is used mainly to move large amount of data from on-premise to AWS. It can synchronize to: Amazon S3,Amazon EFS, Amazon FSx for Windows and it moves data from your NAS or file system via NFS or SMB. The data transfer via AWS Datasync can be secure, fast and cost-effective comparing to similar open source tools.
In this document, we will be focusing on using Datasync in conjunction with on-premise NFS and Amazon EFS.
In this document, we will be focusing on using Datasync in conjunction with on-premise NFS and Amazon EFS.
EFS is Managed NFS that can be mounted on many EC2 in multi-AZ and provides highly available, scalable and and shareable storage. EFS is POSIX file system that scales automatically without capacity planning needed.
AWS Backup is fully AWS managed service that enables you to centralize and automate data protection. In this document, It will be used at to backup the EFS file system that we are moving data to.
Elastic File System:
EFS could be created either using AWS Console, CLI or SDK. As discussed earlier, No setup, provisioning or capacity management are needed. The high availability level that customer may require depends on different storage classes offered by AWS (Based on redundancy options and Access frequency). It is up to the customer to define the application needs and criticality. The clients (Usually EC2) are allowed to access the EFS via mount targets that should be separately created. Each set of EC2 instances tied to an AZ needs its new target mount to be created in that particular AZ. Like EC2 instances, the EFS security is managed by Security Groups. It could be accessed by its DNS name. The DNS resolution of cross AZ target mounts is automatically managed.
High availability ,Durability, File system Life cycle among many other options could be set while creating the EFS. Encryption at rest is a recommended option for data protection purposes but can only be set during the file system creation while encryption in transit can be set later.
EFS offers two performance related modes:
EFS could be created either using AWS Console, CLI or SDK. As discussed earlier, No setup, provisioning or capacity management are needed. The high availability level that customer may require depends on different storage classes offered by AWS (Based on redundancy options and Access frequency). It is up to the customer to define the application needs and criticality. The clients (Usually EC2) are allowed to access the EFS via mount targets that should be separately created. Each set of EC2 instances tied to an AZ needs its new target mount to be created in that particular AZ. Like EC2 instances, the EFS security is managed by Security Groups. It could be accessed by its DNS name. The DNS resolution of cross AZ target mounts is automatically managed.
High availability ,Durability, File system Life cycle among many other options could be set while creating the EFS. Encryption at rest is a recommended option for data protection purposes but can only be set during the file system creation while encryption in transit can be set later.
EFS offers two performance related modes:
For the purposes of this document, we recommend selecting the General Purpose performance mode and to use provisioned throughput mode if your transfer rate should exceed 100MB/s.
AWS provides four EFS storage classes:
AWS provides four EFS storage classes:
Choosing IA storage can reduce costs but increasing latency. The EFS lifecycle can automatically manage the file move based on already set custom policy. From operational point of view, EFS serves its clients transparently regardless the EFS storage class.
In this article, we recommend enabling lifecycle management for cost effectiveness.
In this article, we recommend enabling lifecycle management for cost effectiveness.
AWS DataSync:
AWS Datasync is a AWS fully managed service useful for data migration and data synchronization between the on-premise and cloud environments. This document assumes that a Direct Connect is set for a better transfer rate(up to 10Gbps per Datasync Agent). AWS Datasync offers encryption in transit and data integrity checks. It performs some data processing for a more efficient bandwidth utilization.
The needed steps to perform this operation are:
AWS Datasync is a AWS fully managed service useful for data migration and data synchronization between the on-premise and cloud environments. This document assumes that a Direct Connect is set for a better transfer rate(up to 10Gbps per Datasync Agent). AWS Datasync offers encryption in transit and data integrity checks. It performs some data processing for a more efficient bandwidth utilization.
The needed steps to perform this operation are:
Finally,you need to run the DataSync task either on-demand or periodically via the DataSync API.
AWS Backup:
AWS backup offers an extra data protection layer. Automatic AWS backup feature could be turned on/off during the creation of the EFS or later depending on the EFS storage class.
AWS backup offers an extra data protection layer. Automatic AWS backup feature could be turned on/off during the creation of the EFS or later depending on the EFS storage class.
AWS Backup provides automated backup schedules, retention management, and lifecycle management,. It protects the data from any unintended events. AWS Backup is incremental which means that only the changes applied on the file system are recorded.
The backup plan is the policy expression that defines when and how you want to back up your AWS resources. Predefined backup plan are already provided by AWS but customer can add it own plans.
You can optionally add rules handling the data migration conditions and the purge timeline. Moving data to ‘cold’ storage may cost you few extra hours during the restore process. After you create your backup plan, you can choose which of your EFS file systems are governed by it. Two options are available to select the target file systems: The tag-based mechanism which allow the AWS backup process to consider each file system with a particular tag or by simply selecting the files systems IDs.
Backups are encrypted with a KMS key for security purposes. This process is managed by backup vaults created with AWS backup service.
The restore process can be triggered with the AWS console and the backup can be restored to an existing file system or to a newly created one.
The backup plan is the policy expression that defines when and how you want to back up your AWS resources. Predefined backup plan are already provided by AWS but customer can add it own plans.
You can optionally add rules handling the data migration conditions and the purge timeline. Moving data to ‘cold’ storage may cost you few extra hours during the restore process. After you create your backup plan, you can choose which of your EFS file systems are governed by it. Two options are available to select the target file systems: The tag-based mechanism which allow the AWS backup process to consider each file system with a particular tag or by simply selecting the files systems IDs.
Backups are encrypted with a KMS key for security purposes. This process is managed by backup vaults created with AWS backup service.
The restore process can be triggered with the AWS console and the backup can be restored to an existing file system or to a newly created one.
After setting all needed services for data migration to the cloud, we need to monitor the performance of each one of the services based on metrics provided by the service itself or the centralized monitoring service AWS CloudWatch.
AWS DataSync:
Basically, what we need is to make sure that the AWS DataSync is able to reach the data source and the data target all the time and to get status information about each task which helps to troubleshoot any issue that may occur. The information about every task phase is available as well to provide progress status.
The DataSync API can provide operational health information to external monitoring systems . You can also leverage AWS CloudWatch log group to centralize logs in AWS.
DataSync is well integrated also with Amazon CloudWatch to send events and metrics useful to set alarms for state change in an agent,location,task,or task execution. You can also get the real-time statistics about the transferred volume of data as a function of time.
Basically, what we need is to make sure that the AWS DataSync is able to reach the data source and the data target all the time and to get status information about each task which helps to troubleshoot any issue that may occur. The information about every task phase is available as well to provide progress status.
The DataSync API can provide operational health information to external monitoring systems . You can also leverage AWS CloudWatch log group to centralize logs in AWS.
DataSync is well integrated also with Amazon CloudWatch to send events and metrics useful to set alarms for state change in an agent,location,task,or task execution. You can also get the real-time statistics about the transferred volume of data as a function of time.
Amazon EFS:
The metrics provided by Amazon CloudWatch could be useful to decide which EFS performance mode to be used. You can set initially the EFS performance to be at General purpose mode and permanently monitor the PercentIOLimit metric. If the metrics consistently stands around 100%, you may want to set your EFS file system at the Max I/O mode. Alternatively, you can split the workload across multiple General Purpose mode EFS.
AWS proposes two techniques to drive additional throughput while not reaching the I/O limit:
1. EFS recommends leveraging parallel I/O as much as possible taking advantage of the distributed design of EFS. A GitHub tutorial is available for more details regarding this technique.
2. You can also determine if your throughput is being rate-limited based on your storage. The CloudWatch Throughput utilization metric can be useful for this purpose. The MeteredIOBytes metric (ratio of metered throughput you driving on your EFS to PermittedThroughput) could be useful as well. If the ratio is equal to one, the file system is consuming all available throughput. You can see the same behavior if your EFS is on Bursting Throughput mode with a small file system with no burst credit. If this is the case and you have deterministic application throughput requirements, you can consider the Provisioned Throughput mode.
All discussed metrics are available on Amazon CloudWatch dashboards.
The metrics provided by Amazon CloudWatch could be useful to decide which EFS performance mode to be used. You can set initially the EFS performance to be at General purpose mode and permanently monitor the PercentIOLimit metric. If the metrics consistently stands around 100%, you may want to set your EFS file system at the Max I/O mode. Alternatively, you can split the workload across multiple General Purpose mode EFS.
AWS proposes two techniques to drive additional throughput while not reaching the I/O limit:
1. EFS recommends leveraging parallel I/O as much as possible taking advantage of the distributed design of EFS. A GitHub tutorial is available for more details regarding this technique.
2. You can also determine if your throughput is being rate-limited based on your storage. The CloudWatch Throughput utilization metric can be useful for this purpose. The MeteredIOBytes metric (ratio of metered throughput you driving on your EFS to PermittedThroughput) could be useful as well. If the ratio is equal to one, the file system is consuming all available throughput. You can see the same behavior if your EFS is on Bursting Throughput mode with a small file system with no burst credit. If this is the case and you have deterministic application throughput requirements, you can consider the Provisioned Throughput mode.
All discussed metrics are available on Amazon CloudWatch dashboards.
The service pricing is $0.0125/GB or $12.80/TB transferred. This price is globally applicable.
EFS pricing depends on the file system storage class, infrequent access IO request and the provisioned throughput. This document focuses on US-EAST-1 prices:
| Storage Class | Price (per GB-month) |
|---|---|
| EFS Standard storage | $0.30 |
| Standard IA storage | $0.025 |
| EFS One Zone Storage | $0.16 |
| EFS One Zone IA | $0.0133 |
If you choose the Infrequent Access storage classes, an extra fee is applied for retrieving data or moving data to the IA storage class. The fee is $0.01/GB transferred.
You pay also for the provisioned throughput at the rate of $6/MB/s. You only pay if you provision more throughput than the what your EFS class storage/Size provides initially.
You pay also for the provisioned throughput at the rate of $6/MB/s. You only pay if you provision more throughput than the what your EFS class storage/Size provides initially.
The backup service costs you $0.05/GB-month for warm storage and $0.01/GBmonth for cold storage.
Restores cost $0.02/GB from warm storage and $0.03/GB from cold storage.
Restores cost $0.02/GB from warm storage and $0.03/GB from cold storage.
The following example is made based on simplifying assumptions satisfying the industry estimates:
Assumptions:
Assumptions:
| Service | Cost |
|---|---|
| AWS DataSync | $0.0125/GB * 1000 GB/TB * 100 TB = $1,250 |
| Amazon EFS Standard Storage | $0.30/GB-month * 1000 GB/TB * 100 TB * 20% frequently accessed * 12 months/year = $72,000 |
| Amazon EFS Standard IA I/O | $0.01/GB * 1000 GB / TB * 100 TB * 80% infrequently accessed = $800 |
| Amazon EFS Standard IA Storage | $0.025/GB-month * 1000 GB/TB * 100 TB * 80% infrequently accessed * 12 months/year = $24,000 |
| AWS Backup | $0.05/GB-month * 1000 GB/TB * 100 TB * 12 months/year = $60,000 |
| Total Cost | $157,250 |
Assumptions:
| Service | Cost |
|---|---|
| AWS DataSync | $0.0125/GB * 1000 GB/TB * 100 TB = $1,250 |
| Amazon EFS Standard Storage | $0.16/GB-month * 1000 GB/TB * 100 TB * 20% frequently accessed * 12 months/year = $38,400 |
| Amazon EFS Standard IA I/O | $0.01/GB * 1000 GB / TB * 100 TB * 80% infrequently accessed = $800 |
| Amazon EFS Standard IA Storage | $0.0133/GB-month * 1000 GB/TB * 100 TB * 80% infrequently accessed * 12 months/year = $12,768 |
| AWS Backup | $0.05/GB-month * 1000 GB/TB * 100 TB * 12 months/year = $60,000 |
| Total Cost | $112,418 |
This document explains how to set up,tune and monitor your migrating applications to the AWS cloud. The provided solution is based on AWS DataSync for migrating data, Elastic File System for storing the data and AWS Backup for data protection. The above detailed examples show how much it will cost you approximately to migrate 100TB of data from your on-premise data center to AWS Cloud.
45