This is a submission for the Netlify Dynamic Site Challenge: Visual Feast.
This is a submission for the Netlify Dynamic Site Challenge: Build with Blobs.
This is a submission for the Netlify Dynamic Site Challenge: Clever Caching.
What I Built
I developed a full-stack image sharing platform similar to Giphy or Imgur in order to showcase the power of building full-stack applications with only Netlify Primitives
So what are Netlify Primitives?
Netlify Platform Primitives are a set of features provided by Netlify that can be used to build and enhance web applications. They are framework-agnostic, meaning they can be used with any web development framework you choose.
Most full-stack applications basically require a frontend, backend, database, authentication mechanism, caching and hosting.
Netlify provides services for all of the requirements above.
Gallerify is hosted on Netlify and leverages the Netlify Platform Primitives such as Netlify Serverless Functions, Netlify Blobs and Netlify Image CDN
For authentication Netlify Identity was leveraged.
Gallerify Tech Stack
Frontend: Next.js with Bootstrap 5 customized with SASS
Backend: Netlify Functions
Image Storage: Blob Data
Database Storage For Image Metadata: Blob Metadata
Authentication: Custom Netlify Identity in Netlify Functions
Even though Gallerify was built during the Netlify Dynamic Site Challenge the goal was not to build the next Giphy or Imgur.
It was tempting to build most of the features of Giphy or Imgur in order to be considered for the first spot.
But later I realized that just focusing on the final product will not allow me to shine by detailing my creative journey.
Instead of building most of the features of Giphy or Imgur I would rather focus on the basics and then explain the choices I made, the challenges I faced, and how you overcame them.
Doing this will enable me to share an engaging and relatable content to the community.
Who knows? I might inspire someone to rather build the Next Giphy or Imgur on Netlify Primitives.
If you got to this point, I encourage you to continue reading to learn how I built Gallerify.
Demo
Before I get to the technical details, you can checkout Gallerify by clicking the link below. You can register for a new account and it’s free or you can use the demo credentials below as well.
Demo Credentials:
Email: demo@demo.com
Password: demo1234
Demo URL: https://gallerify-app.netlify.app
Screenshots:
Platform Primitives
Netlify Image CDN
For caching of images I leveraged Netlify Image CDN.
To transform an image, you make a request to /.netlify/images of your site with query parameters that specify the source asset and the desired transformations
For the Masonry layout I render the cached version of the image without any transformations.
Example URL:
Mansory Layout Images:
https://gallerify-api.netlify.app/.netlify/images?url=https://gallerify-api.netlify.app/api/gallery?uid=uploader_id_here&image_id=image_id_here&image_name=image_name_here
Thumbnail Layout Images
https://gallerify-api.netlify.app/.netlify/images?url=https://gallerify-api.netlify.app/api/gallery?uid=uploader_id_here&image_id=image_id_here&image_name=image_name_here&fit=cover&w=440&h=440
For the thumbnail layout an optimized 440px x 440px image served by Netlify Image CDN on the fly.
The url query parameter is a Netlify functions endpoint that serves the image with the appropriate tag so that the img HTML tag can render the image.
Once you choose a particular layout the cached images are served. Also note that since my url endpoint also has query parameter I had to decode it using JavaScript function encodeURIComponent in order to prevent issues with the Netlify Image CDN if I also pass multiple query parameters.
Challenges I Faced:
When an image is added from the dashboard, it did not immediately appear on the homepage and dashboard.
Decisions Made:
To solve the problem of images not appearing immediately on the homepage and dashboard, I leveraged Cache Control Settings in Netlify Serverless Functions response headers to revalidate and fetch a fresh copy of the images after 60 seconds of upload for the public homepage. For the dashboard, the control settings was set to 0. Images will appear immediately once they are uploaded
In a real world application, it’s good practice to prevent your server from constant public requests. Thats why I set the revalidate option of 60 seconds for the homepage. Since the dashboard will not have as many requests like the homepage it’s ideal to allow images to appear immediately so that users will not be confused about the images they uploaded.
return new Response(JSON.stringify(data_here), {
headers: {
“Content-Type“: “application/json“,
“Cache-Control“: “no-cache, no-store“,
“Netlify-CDN-Cache-Control“: “no-cache, no-store“,
“Access-Control-Allow-Origin“: “*“
},
status: statusCode
});
return new Response(JSON.stringify(data_here), {
headers: {
“Content-Type“: “application/json“,
“Cache-Control“: “public, max-age=60, must-revalidate“,
“Netlify-CDN-Cache-Control“: “public, max-age=60, must-revalidate“,
“Access-Control-Allow-Origin“: “*“
},
status: statusCode
});
Netlify Identity
In order for users to login or register to your application, you need an authentication mechanism.
I could have used Netlify Blobs to store user information and JWT tokes. However it’s not ideal to be storing user credentials in blobs. Also since Netlify already provides authentication using Netlify Identity, it was better to leverage Netlify Identity as I makes authentication a breeze and then handles all authentications mechanisms out of the box.
Netlify provides Netlify Identify in order to sign up, login, reset password, verify and confirm users.
In order to use Identity you can either use the Netlify Identity Widget or GoTrue(https://github.com/netlify/gotrue) JavaScript plugin for custom configurations.
Challenges I Faced:
I wanted a custom authentication for Gallerify and I did not want to use the Default Identify widget as it will not match with the styling of my website.
Also even though the GoTrue plugin can help with customization I did not want authentication request to come from the frontend as an attacker can go through the GoTrue source code and the send malicious requests.
I wanted to use Netlify Identify in my Netlify Functions. At the time I did not know GoTrue was only for client-side so I installed it for my Netlify Functions only to be thrown multiple errors. I realized the package only works for the client-side
Decisions Made:
So solve the problem of using Netlify Identity in Netlify Functions I decided to go through the source code of the GoTrue project on GitHub and then learn how the requests were made.
I figured out how requests were sent to Netlify Identify and then I wrote my custom authentication logic and then sent requests using fetch to login, sigup, and verify users JWT.
Once send a request to create an account a JWT is returned and then stored in cookies for making authenticated requests by users.
I was able to customize my website and then make authentication requests from the frontend form to the Gallerify Netlify Function Backend without using the Identity Widget or GoTrue plugin.
// Netlify Identity Signup
const registerResponse = await fetch(netlifyIdentitySignupUrl, {
method: “POST“,
body: JSON.stringify({ email, password, data: { full_name: username } }),
headers: { “Content-Type“: “application/json“},
});
const responsedata: any = await registerResponse.json();
// Netlify Identity Login
const loginResponse = await fetch(netlifyIdentityLoginUrl, {
method: ‘POST‘,
headers: { ‘Content-Type‘: ‘application/x-www-form-urlencoded‘ },
body: `grant_type=password&username=${encodeURIComponent(
email,
)}&password=${encodeURIComponent(password)}`,
});
const data: any = await loginResponse.json();
// Netlify Identity Verify User – JWT
const headers = req.headers;
const accessToken = headers.get(“authorization“)?.split(“ “)[1];
const verifyResponse = await fetch(netlifyIdentityVerifyUrl, {
method: ‘GET‘,
headers: {
“Authorization“: `Bearer ${accessToken}`
}
});
const data = await verifyResponse.json();
Netlify Blobs
For most applications, You need a location to store media. Gallerify is an image sharing platform so I needed a way to store the image files.
Netlify Blobs provides mechanisms to store the actual file data and then metadata or data that will be normally saved in a database.
I leveraged Netlify Blobs to store images as Blobs and authenticated user information that uploaded the file.
Challenges I Faced:
I did not have much knowledge on Netlify blobs so when I read the basics from the documentation and then I tried to save the image, anytime I try to serve the image to the frontend the image was served as a binary format.
Even though my Cache Control settings were previously set to a max-age of 0 and then no-cache, any time there are updates and deletions my Netlify Function was still serving old data. Fresh data was only served after 60 seconds.
Listing Blobs using await imagesStore.list() does not return data with metadata but keys. I was using for await…of to iterate through the response from await images.list().
Once the data increased my Netlify Functions were crashing due to the function execution duration of 10 seconds.
Decisions Made:
In order the solve the problem of images served as binary, I read multiple blog posts and articles. At some point I even used a third party image storage service until I stumbled upon @mattlewandowski93‘s post on this website.
I later figured out how to serve images properly by reading @mattlewandowski93‘s submission post.
I want to say a big thanks to @mattlewandowski93‘s for explaining how he was able to serve the images properly from Netlify Functions.
@mattlewandowski93‘s post provided amazing insights on how he served images properly how he implemented features for his application.
I was able to create an API endpoint to properly serve images in the right format.
// Code snippet of how to properly serve images
const userIdFromParams = url.searchParams.get(“uid“) as string;
const imageId = url.searchParams.get(“image_id“) as string;
const imageName = url.searchParams.get(“image_name“) as string;
const imageBlobKey = `${userIdFromParams}/${imageId}/${imageName}`;
const currentImageBlob = await imageStore.getWithMetadata(imageBlobKey, { type: “blob“ });
if (currentImageBlob) {
return new Response(currentImageBlob.data, {
status: 200,
statusText: “OK“,
headers: {
“Content-Type“: currentImageBlob?.metadata?.type as any,
},
});
}
return generateAppResponse(“error“, 404, “Image not found“, currentImageBlob);
Also note that when you want to get blob data, you have to specify the type. In my case I used the blob type
The line below is how the image is initially stored so that it can be sent back to the Netlify Functions to properly search based on prefixes. Searching by directory prefix style is faster.
In order serve fresh data when updates and deletions occurred, I had to read the Netlify Blobs Documentation multiple times in order to understand the API better.
I later found out that Netlify Blobs uses an eventual consistency model by default. The data is stored in a single region and cached at the edge for fast access across the globe. Therefore when a blob is added, it becomes globally available immediately.
However updates and deletions were propagated to all edge locations within 60 seconds. This means that it may take up to 60 seconds for a fresh blob to be served after it has been updated or deleted.
In order to prevent this from happening and then serve fresh data, I had to reconfigure the default behavior of the Blobs API and opt-in to strong consistency with the Netlify Blobs API.
After configuring using the strong options when I fetched images after images were added or deleted I was receiving the fresh data.
name: “uploads“,
consistency: “strong“,
siteID,
token,
});
In order to get metadata of blobs when using the await imageStore.list(), I used the technique below.
const images = [];
for await (const blob of entry.blobs) {
const metadata = await store.getMetadata(blob.key);
images.push(metadata);
}
However using the technique above was not efficient because when data begins to increase, once this code runs in your Netlify Function, Your function times out after 10 seconds and an error is thrown.
Another option was to use Background Functions but that was going to be an overkill. Also I needed to serve a response immediately when I fetch for all data.
I wished the Blobs API had an option to list blobs with metadata.
For 2 days I was thinking about how to solve the problem of fetching data without my functions timing out after 10 seconds.
So I read the Documentation once more and decided to use a prefix. The reasoning behind this decision is that when await imagesStore.list() is executed the data is returned fast enough no matter how the size of the data after uploading multiple images and testing it out.
The problem was that the return value was the key and the etag
Therefore by storing blobs using the key format below, I can get data belonging to a particular user. Think of it like a directory structure on your computer.
// The uuidv4 package was used to generate unique id’s for files and images
// Store file in the format – userId/imageId/filename
const imageBlobKey = `${userId}/${imageId}/${imageName}`;
await imageStore.set(imageBlobKey, imageFile, {
metadata: imageMetadata
});
An array of keys in the format above is sent to the frontend and then when I want to fetch images belonging to a particular user I fetch by the prefix. The response from the execution is very fast.
const userIdFromParams = url.searchParams.get(“uid“) as string;
const imageId = url.searchParams.get(“image_id“) as string;
const imageName = url.searchParams.get(“image_name“) as string;
const imageBlobKey = `${userIdFromParams}/${imageId}/${imageName}`;
const currentImageBlob = await imageStore.getWithMetadata(imageBlobKey, { type: “blob“ });
if (currentImageBlob) {
return new Response(currentImageBlob.data, {
status: 200,
statusText: “OK“,
headers: {
“Content-Type“: currentImageBlob?.metadata?.type as any,
},
});
Anytime I want to get image data or image metadata there are API endpoints that receive the userId, imageId, fileName as query parameters and then it’s reconstructed again in the format userId/imageId/filename so that I can fetch by a specific user or any image.
That it, at this point you have the learned how you can leverage Netlify Primitives to build full-stack applications.
I am confident this post will inspire a lot of developers to build the applications like Giphy, Imgur, Or the “next big app”…
I want to add that in case you have any challenges finding information on the Netlify Documentation, try out the Netlify Bot. It’s so awesome and it helped me narrow down information I was looking for.
The ideas are endless. For example a Blog application, Social Media application, Podcast application e.t.c
You can checkout project I work on at my GitHub repo here
Thanks for taking the time out of your day to read this blog post.
Happy building with Netlify.
