Get started with Amazon Elastic File System

Photo by Trent Erwin on Unsplash

Get started with Amazon Elastic File System

AWS Elastic File System (EFS) is a cloud-based file storage service offered by Amazon Web Services (AWS) designed to provide scalable, elastic, concurrent with shared access to file data for Linux-based workloads. It's built to be highly durable and available, and it's a fully managed service that can be used with AWS cloud services and on-premises resources.

How it works

Amazon Elastic File System (EFS) automatically grows and shrinks as you add and remove files with no need for management or provisioning.

Key features of AWS EFS

  1. Elasticity: As the name suggests, EFS is elastic, meaning it automatically grows and shrinks as you add and remove files. You only pay for the storage you use, and there's no need to provision storage in advance.

  2. Scalability: EFS can scale on demand to petabytes of data without disrupting applications. This makes it suitable for workloads and applications that require high levels of throughput and IOPS.

  3. Shared Access: Multiple Amazon EC2 instances can access an EFS file system at the same time, providing a common data source for workloads and applications running on more than one instance or server.

  4. Durability and Availability: Data is stored across multiple Availability Zones (AZs) within an AWS Region, offering better durability and resilience to AZ failures.

  5. Performance Modes: EFS offers two performance modes: General Purpose, which is suitable for most file system workloads, and Max I/O, which is optimized for large-scale, parallelized workloads that require high levels of throughput and IOPS.

  6. Throughput Modes: There are two throughput modes: Bursting Throughput and Provisioned Throughput. Bursting mode allows throughput to scale with the size of the file system, while Provisioned Throughput is for applications that require a specific amount of throughput.

  7. Security: EFS integrates with AWS Identity and Access Management (IAM) and Virtual Private Cloud (VPC), and supports POSIX permissions. It also offers encryption at rest and in transit.

  8. Compliance: EFS is designed to meet various compliance regulations, and AWS frequently achieves third-party validations for a variety of standards and regulations, which can simplify your compliance efforts.

  9. Access from On-Premises: You can mount EFS file systems on your on-premises data center servers using AWS Direct Connect or AWS VPN to easily migrate to AWS or burst to the cloud.

  10. Integration with AWS Services: EFS works with AWS services like AWS Lambda and Amazon ECS for serverless, containerized, and microservice-based applications, providing a persistent file storage layer.

  11. Lifecycle Management: EFS can automatically move files that are not accessed according to the lifecycle policy to EFS Infrequent Access (IA) with lower storage costs.

EFS is particularly suited for use cases such as content management, web serving, data sharing, and as a data layer for applications that need a shared file system. It's a go-to solution for applications that need traditional file system interfaces and file system semantics.

Use cases

AWS Elastic File System (EFS) serves a wide variety of use cases, especially where scalable, shared file storage is needed. Here are some of the common scenarios where EFS is typically used:

  1. Content Management and Web Serving: EFS can store and serve web content and media for content management systems (CMS) like WordPress or Drupal. It allows multiple instances to read and write data, ensuring content is up-to-date across all servers.

  2. Data Sharing: EFS can be used to share files across different compute instances, such as for applications that require access to the same set of files for processing or data interchange.

  3. Home Directories: EFS can act as a central location for user home directories, allowing users to access their files from any server or compute instance in the AWS environment.

  4. Software Development and Testing: Development teams can use EFS for storing and sharing source code repositories, build directories, and artifacts across development, testing, and production environments.

  5. Big Data Analytics: For applications that perform big data analytics, EFS can be used to store and share large datasets that need to be accessible by multiple compute nodes, possibly running Hadoop or other data processing frameworks.

  6. Container Storage: When using containerized applications with orchestration services like Amazon ECS or Kubernetes, EFS can provide persistent storage for containers, ensuring data persists beyond the life of a single container.

  7. Serverless Applications: EFS can be used with AWS Lambda to store and access files in a serverless architecture, where compute resources are managed by AWS and can scale automatically.

  8. Database Backups: It can be utilized to store database backups centrally, which can then be accessed by various instances or services for restoration or analysis.

  9. Media & Entertainment Workflows: In media production, EFS is used for video editing, sound design, and rendering workflows where multiple users and applications need access to the same set of large files.

  10. HPC (High Performance Computing): EFS is suitable for HPC workloads that need to store and retrieve large datasets quickly across multiple compute nodes.

  11. Disaster Recovery: EFS’s multi-AZ design can serve as a resilient and durable storage option that supports an organization's disaster recovery strategy.

  12. Machine Learning: Storing training data and machine learning models where multiple instances may need to access this data concurrently during the training or inferencing process.

  13. Lift and Shift Migrations: When moving legacy applications to the cloud, EFS helps maintain the existing file storage structure without significant refactoring, allowing for easier migrations.

By offering a fully managed file system that can be shared across different AWS services and instances, EFS is a versatile tool that addresses the needs of many cloud storage use cases.

Tutorial: Create and connect to EFS via AWS CLI

To create an Amazon Elastic File System (EFS) using the AWS Command Line Interface (CLI), you need to have the AWS CLI installed and configured with the necessary permissions. Below is a step-by-step guide on how to create an EFS file system:

💡
To create EC2 instances use the following guide: How to launch a single EC2 instance via AWS CLI

Overall scheme:

Step 1: Configure the AWS CLI

Before creating an EFS file system, make sure you have the AWS CLI installed on your local machine. If you haven't done so, download and install it from the AWS website. Then configure it with the following command:

aws configure

You'll be prompted to enter your AWS Access Key ID, Secret Access Key, region, and output format.

Step 2: Create EFS File System

Use the create-file-system command to create a new file system. You can specify tags or use the default configuration.

aws efs create-file-system --creation-token MyEfsFileSystem --tags Key=Name,Value=MyEfsFileSystem --region us-east-1 --output table

The --creation-token is a string that ensures idempotent creation (prevents creating duplicate file systems if the request is repeated). Replace us-east-1 with the AWS region where you want to create your file system.

Step 3: Retrieve File System ID

After you create the file system, the command will output details about the EFS. You'll need the FileSystemId for the next steps.

In the given example the FileSystemId is fs-0d378c1b3842657c7

Step 4: Create a Security Group

To create a Security Group via the AWS CLI that allows an EC2 instance to access a created EFS, follow these steps:

Step 4.1: Create a Security Group

First, you'll want to create a security group within your VPC. The following command creates a security group named EfsAccessSecurityGroup in your default VPC:

aws ec2 create-security-group --group-name EfsAccessSecurityGroup --description "Security group for EFS access" --vpc-id vpc-xxxxxx

Make sure to replace vpc-xxxxxx with your actual VPC ID.

Step 4.2: Capture the Security Group ID

The output of the previous command will provide you with a security group ID. It will look something like this:

{
    "GroupId": "sg-00f52bde422c86bc2"
}

Capture the GroupId value, as you will need it for the next steps.

Step 4.3: Add Inbound Rule for NFS

EFS uses the NFS protocol, which by default operates on port 2049. You need to add an inbound rule to your new security group to allow traffic on this port from your EC2 instances. If your EC2 instances are in a different security group, you can reference that security group instead of an IP range.

aws ec2 authorize-security-group-ingress --group-id sg-0123456789abcdef0 --protocol tcp --port 2049 --source-group sg-abcdefgh

In this command, replace sg-0123456789abcdef0 with the security group ID you got from the previous step, and sg-abcdefgh with the security group ID of your EC2 instances.

If you want to allow access from any EC2 instance in the VPC, you might specify the VPC CIDR block instead:

aws ec2 authorize-security-group-ingress --group-id sg-0123456789abcdef0 --protocol tcp --port 2049 --cidr 10.0.0.0/16

Replace 10.0.0.0/16 with the CIDR block of your VPC.

Step 4.4: Add Outbound Rule (Optional)

By default, new security groups allow all outbound traffic. If you have modified the default outbound rules, you will also need to ensure that outbound traffic to the NFS port is allowed:

aws ec2 authorize-security-group-egress --group-id sg-0123456789abcdef0 --protocol tcp --port 2049 --cidr 0.0.0.0/0

This command allows outbound traffic to any destination on port 2049. Adjust the CIDR block as necessary for your use case.

Step 4.5: Attach the Security Group to EC2 Instances (Optional)

If you have existing EC2 instances to which you want to attach the newly created security group, use the following command:

aws ec2 modify-instance-attribute --instance-id i-0123456789abcdef0 --groups sg-0123456789abcdef0 sg-abcdefgh

Replace i-0123456789abcdef0 with your instance ID, sg-0123456789abcdef0 with the security group ID for EFS access, and sg-abcdefgh with any other security group IDs that the instance should be a part of.

Note: This command will overwrite all existing security groups attached to the EC2 instance, so ensure to list all necessary security groups.

Step 5: Create Mount Targets

For your EC2 instances to access the EFS, you need to create mount targets in the VPC subnets. You'll need the subnet IDs and the security group IDs that allow NFS traffic.

aws efs create-mount-target --file-system-id fs-12345678 --subnet-id subnet-87654321 --security-groups sg-12345678 --region us-east-1

Replace fs-12345678 with your file system ID, subnet-87654321 with your subnet ID, and sg-12345678 with your security group ID.

Repeat this step for each Availability Zone in which you want to create a mount target. It's recommended to have a mount target in each Availability Zone for high availability.

Step 6: Check Mount Target Creation

You can describe your mount targets to check on their creation status:

aws efs describe-mount-targets --file-system-id fs-12345678 --region us-east-1

Step 7: Attach the Security Group to EFS Mount Targets

You need to attach the security group to your EFS mount targets. You can do this through the AWS Management Console or by using the modify-mount-target-security-groups action via CLI:

aws efs modify-mount-target-security-groups --mount-target-id mount-target-id --security-groups sg-0123456789abcdef0

Replace mount-target-id with the ID of your mount target, and sg-0123456789abcdef0 with the ID of your security group.

After following these steps, your EC2 instances should be able to access the EFS file system over the NFS protocol, as long as they're also within the allowed CIDR range or part of the specified security groups.

Step 8: Connect to the File System

Once the mount targets are available, you can mount the file system on your EC2 instances:

sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 fs-12345678.efs.us-west-2.amazonaws.com:/ /mnt/efs

Replace fs-12345678.efs.us-west-2.amazonaws.com with the DNS name of your EFS, and /mnt/efs with the local path where you want to mount the file system.

Don't forget to create a target mount directory /mnt/efs on your machine

You've created an EFS file system and mounted it on an EC2 instance using the AWS CLI. Remember to ensure that your EC2 instance's security group allows outbound connections on TCP port 2049 for NFS, and the EFS mount target's security group allows inbound connections from your EC2 instance's security group on the same port.

References:

  1. What is Amazon Elastic File System?

  2. Creating and managing mount targets and security groups

  3. Create a Network File System