Django, CSVs, FileField and S3 – How to stream a CSV file into an S3 Bucket and also save it into a Django Model’s FileField?

Solution for Django, CSVs, FileField and S3 – How to stream a CSV file into an S3 Bucket and also save it into a Django Model’s FileField?
is Given Below:

everyone. Hope you’re doing well. I would like to start this question by prefacing that I’m a real newbie when it comes to AWS. Right now I’m working on a Django app with a friend that already has some more experience with web development and he had already set up everything in AWS that is, we have most of our files uploaded there, a staging app running and some other configs.

I’m working on a backend implementation of transforming some database data from our web app into CSV files and uploading it into the cloud.

Basically, we have an endpoint where a User can ask for data from another URL of the site, and our app will transform the data into a .CSV file, export it to S3 and AT THE SAME TIME saving it on a FileField of the model. Then, once the File is uploaded, the user will receive it in its e-mail. Since the file could be really big, the BIG question is: How do you stream .CSV file into a S3 Bucket and have Django save it into the FileField attribute of a model? Here’s my code for reference (This is the only way it created a .csv locally so far):

--------------------------/// MODEL // ---------------------------------------------
class Export(models.ModelModel):
    user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE,)
    file = models.FileField(upload_to='s3://buckect/uploads/', blank=True, null=True)
# Some other fields that don't matter

-----------------------------///-EXPORT FUNCTION -----------------------------------
from io import StringIO
import csv
from smart_open import open

with open('s3://buckect/uploads/test.csv', 'a+') as fout:
        csv_buffer = StringIO()
        csv_writer = csv.writer(csv_buffer, escapechar=" ", quoting=csv.QUOTE_NONE)
        data = data_generator(
        for row in data:
        csv_file = csv_buffer.getvalue()
        fout.write(csv_file)'s3://buckect/test.csv', fout)

I haven’t tested it yet because I’m still without access to S3, but this seems to me as it would upload the file to S3 twice, wouldn’t it?

I did try it locally, and yes indeed the .CSV file is created at open() and just saved on the model. But based on the limited knowledge I have of S3, when you set it as your storage, I’m not really sure how Django’s FileField behaves.

Also, I was told to use Smart_open which supposedly streams my file into S3. But I only found one similar usecase that doesn’t really explain IF or HOW it’ll stream the file and it’s also a really old post.