Introduction to Nest.js File Uploads and S3
In this video, I'm going to show you how we can use Nest.js to upload files and upload those files to AWS S3. So we'll be able to send requests to a server, upload any kind of file, and make sure we see that file in S3 for storage and later retrieval.
I'll show you how we can provide validation to the file we're uploading to our server. And additionally, we're also going to apply rate limiting to our API to make sure we don't allow too many requests within a certain period. Let's jump right in and show you how we can accomplish all this, all easily with Nest.js. I'll see you there.
Before we jump in, I just want to let you know that my Nest.js microservices course is now out. Feel free to check out the description for a discounted link to this Nest.js course, which has about 300 students and growing. So thank you so much for everyone that has bought the course and is really seeming to enjoy it so far.
In this course, we dive deep into implementing complex Nest.js microservices, and if this is something you'd like, feel free to check this out. All right, let's jump right into the video.
Project Setup and Scaffolding
So we're going to use the Nest CLI to generate our project by running Nest new. I'll call my project uploader, and I'm going to use pnpm as my package manager. Then I'll cd into the uploader directory and run pnpm run start:dev to start up our server in development mode.
So I've opened up the project in VS Code, and you can see we have our default boilerplate Nest project out of the box with our HTTP server exposed on port 3000 right now. And we have our app.controller with a single GET route that will return a simple text string of 'hello world'. And we have our server started, so we can launch a test request at our server. So in Postman, I've created a new GET request at localhost:3000, and we can send this off and get that 'hello world' response back.
So let's use the Nest CLI again to generate a new module called upload. Then I'll use it again to generate a controller called upload. And lastly, we'll generate an upload service. This is all going to be so that we can accept requests to upload a new file, and we want a dedicated module to handle all of this.
And now we can see the new upload folder created here with our upload module, upload controller, and upload service. So I'll go ahead and actually get rid of the app.service and the app.controller files as we're not going to use them. And in the app.module, we'll also remove them from here so that we only have the upload module. And then here we have our controller and service, which is where we're going to start from.
Handling File Uploads with Multer
So before we start implementing file uploads, we need to install the types for the file upload package that Nest.js uses called Multer. So run pnpm install -D @types/multer. And this is the actual module that will handle the form data that is being sent to the server when we send a POST request over HTTP and send that multipart/form-data format. Multer is going to be what's used to actually parse that incoming request and give us a buffer that we can actually use to do things like upload to different services or another backend like AWS S3, which is what we're going to do.
And now in our upload.controller, let's go ahead and implement a new POST route to actually accept the upload. So I will leave this unlabeled because we're already at /upload. And then to actually parse the incoming request's file, we need to use an interceptor here. So we'll call UseInterceptors from @nestjs/common, and then we're going to accept the FileInterceptor. The FileInterceptor's first argument is the name of the field where the file actually resides in the HTML form that's being sent to this route. So we'll specify the name here and just call it file.
And now we'll have the actual function name that we'll call uploadFile, and we can get access to it by using the UploadedFile decorator. And now we have access to the file variable, which would be of type Express.Multer.file. So now we have access to the raw file this easily, and we have access to the underlying buffer here, which is going to be the actual data stored in memory of this file. And we get additional metadata about the file like the destination, MIME type here, the path, size, and so on. So we can use this information to further upload our file, including passing the buffer to our service layer, which is what we're going to do. Firstly, let's just go ahead and log out the file to make sure that we're receiving it correctly from the client. And let's go ahead and implement that next.
Testing the File Upload Endpoint
So with our server running, I'm going to go ahead and open up Postman and launch a POST request at our /upload route. So send a request to localhost:3000/upload, and importantly, we're going to click on 'Body' here and make sure we select 'form-data' now. So in this form data, this is where we're going to specify the actual key, and of course, we want to select a file because we're going to upload a file. And so we need to make sure this is the HTML key in our form that matches what we selected on our route in the upload.controller.
Remember, we specified the FileInterceptor and we specified the name of the form field where the file lives. Well, this is the same name here: file and file. So now we need to actually upload a file. So you can send whatever kind of file you want to the system right now and send off this request. So you can see we have a 201 Created sent back, and if we look at the logs, we can now see our sample data being logged out, including the file name, the type, and the actual buffer here, as well as the size.
Implementing File Validation
So additionally, I want to show you how we can apply simple validation to our file upload. So perhaps we want to restrict the file type or the size of the file. We can do this very easily by supplying a new ParseFilePipe from @nestjs/common. And inside of here, we're going to supply an object, an options object, where we specify an array of validators. And we can provide a new MaxFileSizeValidator where we're going to specify the max file size is going to be 1000 bytes. Additionally, we can add another validator, a new FileTypeValidator, and this will take an options object where we specify the file type that we want to allow. Let's say we only want to allow JPEG images in our system.
So now that we have this ParseFilePipe in place, if we try to send off a new POST request, you can see we have failed validation because the expected size is less than our limit of 1000. So we have a 400 Bad Request sent. So for now, I'm going to go ahead and comment out these validators so we can continue uploading our file to Amazon S3, which is going to be our next step.
Setting Up AWS SDK and Configuration
So we're ready to set up the AWS SDK in our application so that we can actually configure it and upload our new uploads to S3, which is where we're going to store our uploads. So in our terminal, let's stop our server and let's pnpm install a new package called @aws-sdk/client-s3. And we're also going to install the @nestjs/config package, which is going to allow us to read in environment variables into our application so we can securely store our AWS credentials and provide them to our AWS client.
So first things first, let's go into our app.module and make sure we set up the ConfigModule by calling ConfigModule.forRoot() and we'll initialize it with an isGlobal set to true so that we have this config module globally available and we don't have to keep re-importing it. Next, we're going to go into our upload.service and set up our S3 client.
So in our upload.service, let's set up a constructor where we inject the ConfigService of type ConfigService from @nestjs/config. And now that we have the ConfigService, we can set our S3 client up. So let's create a new read-only variable called s3Client and set it equal to new S3Client. And we import the S3Client from @aws-sdk/client-s3. And you can see we need to provide an options object where we specify the region where we actually are connecting to S3. So we can take this directly from the ConfigService and get the AWS_S3_REGION environment variable. And I'm actually going to change this to getOrThrow to make sure that we throw an error if this environment variable isn't available.
And by default now, the AWS S3 client is going to look for environment variables in our system, including the AWS access key ID and the secret access key, and it's going to use these credentials to authenticate this client and allow us to connect to AWS. So let's go ahead and set that up now.
Managing AWS Credentials and S3 Bucket
So in the AWS console, make sure you create an account or log in, and then we're going to go ahead and click on the account in the top right and then go to 'Security credentials'. Then we're going to scroll down to the 'Access keys' section and click on 'Create access key'. And feel free to create a separate IAM account with specific permissions to only upload to S3 if you want to be a bit more secure. For this example, I'm just going to create an access key for my root user. I'm going to take note of the access key here and the secret access key. So let's copy this access key, and in our project, we're going to create a .env file. Then I'm going to set the AWS_ACCESS_KEY equal to the access key, and I'm going to copy the secret access key, and I'll paste in the AWS_SECRET_ACCESS_KEY and set it equal to the secret access key. Of course, we never want to commit the .env file to Git because it can contain sensitive information. So we can easily add the .env to our .gitignore as well and this will not be committed to our repository.
Now, don't forget we also need to provide the AWS_S3_REGION, and I'm going to use us-east-1. Feel free to use whichever region you would like. Lastly, make sure this AWS_ACCESS_KEY is actually AWS_ACCESS_KEY_ID in order for the AWS client to pick this environment variable up correctly. It needs to be exactly as shown here. So now that we've defined these credentials, the S3 client will be able to authenticate us and we can now upload to S3. Before we do this, I'm going to go back into the AWS console and go to the S3 page where we're going to create a new bucket to upload our uploads to. So I'm going to call this bucket name nestjs-uploader, and I will be using us-east-1, and I'll use all of the default settings here to block all public access by default and create this bucket.
Implementing the S3 Upload Logic
So now that we have the name of the bucket, we can finally define a new method in our upload.service called upload. And in this function now, all we have to do is call await this.s3Client.send(), and we're going to send it a new PutObjectCommand which will take a PutObjectCommandInput parameter to specify the parameters of this object. So let's specify this object where we need to firstly define the bucket we're uploading to, so this will be the bucket we just created, whichever name your bucket is. In my case, it's going to be nestjs-uploader.
And then we need to define the Key, which is going to be the name of this file that we're uploading, as well as the actual Body, which is going to be the buffer content. So to define these, I want to actually go ahead and pass these in as parameters to this method. So let's accept a fileName of type string and then we'll take the actual file here, which we will define as a Buffer. So then we can set the Key equal to fileName and the Body will be equal to the file.
So now that we've defined our upload method, let's go back to our upload controller and make sure we firstly inject the upload service in here. So we'll define a new private readonly uploadService of type UploadService. And then we're just going to call await this.uploadService.upload(). So now I need to supply the name of the file and the file itself to our upload method. We're going to call file.originalName and then file.buffer to our upload method.
Now back in Postman, we can send off a request to upload our sample file. We can see we have a 201 Created and, importantly, if we go back to S3 now, we can open up our nestjs-uploader and see that we have our file that has been uploaded to Amazon S3. So we can see how we can quickly and easily implement file upload with Nest.js. Let's go ahead and see how we can easily implement some rate limiting to our file upload API to prevent brute force attacks.
Implementing API Rate Limiting
To implement rate limiting to our API, we're going to go ahead and install a new dependency called @nestjs/throttler. Now we can go ahead and restart our development server and go back in our application. In our upload.module, let's go ahead and set up the rate limiter module.
So we'll add a new imports array, and we're going to import the ThrottlerModule.forRoot(). So the throttler module is going to accept an options object where we can specify the ttl, which is going to be the maximum number of requests within this time period we want to allow. So let's say within 60 seconds we want to allow only three requests to come into our system. That's going to be these options. And then we need to specify a new guard to guard our routes. So open up an options object in the providers array. We're able to provide an APP_GUARD, and then we're going to call useClass and set it to ThrottlerGuard.
Now if we go back to Postman and send off one, two, three, four requests, we can see on the fourth request we're getting a 429 Too Many Requests and getting this exception from the throttler saying there's too many requests. So this is the exact behavior we want.
Dynamic Rate Limiting and Conclusion
Additionally, we can make this configuration a bit more dynamic by instead calling forRootAsync. And now we can provide a useFactory where we can inject the ConfigService from Nest.js config. And in this ConfigService, we can then pull out the limit and ttl from a config defined in our environment variable. So we'll call getOrThrow and call this UPLOAD_RATE_TTL. And then we'll specify the limit being set to configService.getOrThrow. We'll call this UPLOAD_RATE_LIMIT. Finally, don't forget to provide an inject property here where we specify the ConfigService.
Now in our .env, let's go ahead and specify these two environment variables. So the UPLOAD_RATE_TTL will keep at 60 seconds, and we'll change the UPLOAD_RATE_LIMIT to three requests. So now we're reading this in from our .env. Make sure we restart our server so these variables are read in and we should see the same behavior. If we send off three requests, the fourth one gets that 429. Hopefully, this video has been helpful for you. Thanks so much for watching, and I'll see you in the next one.
