Pinniped is a portable, open-source Backend as a Service designed for small, fast-moving teams and front-end engineers. Pinniped streamlines backend development by providing an intuitive CLI tool and admin dashboard for creating, customizing, and deploying flexible backend applications.
Before discussing the definition and benefits of a backend as a service (BaaS), we’ll examine the typical structure of web applications, the challenges associated with building them, and some tools that address those challenges.
Web applications are commonly built by separating their functionality into three tiers. These tiers are the Data Tier, the Application Tier, and the Presentation Tier. This separation isolates different responsibilities of the application and is often reflected in the physical configuration of the hardware as well. Isolating the responsibilities of each tier makes each easier to develop, maintain, and scale.
The Data Tier is responsible for data storage, retrieval, and management within a web application. It typically encompasses a database and caches. The database conventionally lives on its own hardware, isolated from the other tiers.
The Application Tier, or Business Logic Tier, processes user requests, executes business logic, and manages data exchange between the presentation and data tiers. It handles the application's core functionality, including data validation, authorization, and integration with other services. The application tier is often stateless, allowing it to handle varying request volumes by horizontally scaling across multiple physical or virtual machines.
The interface (API) that allows clients to interact with the application tier is crucial. These interfaces are commonly built using tools like GraphQL or following architectural styles like REST, which we’ll focus on here. REST APIs represent entities, like database tables, as "resources" with unique URLs. Each resource instance (e.g., a row in the table) is accessible by appending its identifier to the resource URL.
A web server receives, processes, and responds to HTTP requests, often implementing a REST API to structure these communications. We'll use Express, a popular web framework for Node.js, to illustrate the creation of a REST API and define some terms that will be referenced later.
Building a REST API using Express involves creating and registering unique route handler functions. These functions define how the server responds to specific HTTP requests targeting particular URLs (endpoints) within the API. Route handlers in Express always receive two key arguments:
Within the route handler, you use the request object to access information about the request and the response object to define the content and structure of the response. This allows you to implement the logic for CRUD operations (Create, Read, Update, Delete) on your resources based on the HTTP method (POST, GET, PUT, DELETE) used in the request.
Here’s a simple example showing the registration of route handler functions at specific URL endpoints. Note how each route requires its own function to handle the unique logic of the route.
Middleware in web frameworks are functions that act as preprocessors before a route handler. These functions are well-suited for handling shared tasks such as parsing data, logging requests, and user authentication. They can be applied globally or to specific routes.
Returning to the final tier of the three-tier architecture, the Presentation Tier is the user-facing layer comprising the web application's user interface and user experience components. It displays information to the users and captures their inputs, facilitating interactions between the user and the application tier. The presentation tier usually runs on users’ devices.
While three-tier architecture offers a clear separation of concerns and many associated benefits, it also introduces challenges. We’ll explore some of these challenges below.
Developers must decide how and where data is stored. How should the data be modeled? Does it best fit a relational, document, or graph model? Will the database run on its own machine or be embedded with the rest of the application? After choosing a suitable database, developers must create data structures that fit the data. They also have to configure and maintain their database as their requirements and user base change over time.
Strategies for handling increased traffic in the data tier include upgrading the database's hardware or implementing various techniques to lighten the load on the database, such as setting up caches, queues, and SQL indexes. Depending on the situation, more significant and costly changes to the architecture, like replicating or partitioning the database, could be necessary.
Developing a robust application tier presents several hurdles. One major challenge is efficiently creating and maintaining an API layer, especially for applications with numerous resources. Exposing each resource through a well-designed API can be a repetitive task. Beyond API development, other challenges include implementing strong security measures for authentication and authorization, validating data, and effectively integrating business-specific logic.
Hosting can be a complicated part of running a web application. The decision on where to host the application, whether locally or in the cloud, involves numerous technical details. Additionally, there are security concerns that must be addressed, such as obtaining and renewing a TLS certificate to secure network traffic. If the hardware is located on-site, scaling the physical infrastructure to meet increasing demand can require significant time and effort.
While every web application is unique, the challenges described above are common, leading to the development of tools such as IaaS, PaaS, and BaaS that solve some or all of these challenges with varying levels of control and abstraction. If a developer wants the most control, they would address each of these challenges themselves.
IaaS products remove the need to purchase, physically house, power, and maintain hardware by offering hardware usage over the internet in a pay-as-you-go model. One common IaaS product is a Virtual Private Server (VPS), which is a server and operating system for developers to remotely host their applications.
While IaaS tools handle hardware concerns, developers are responsible for everything else: managing runtime environments, installing application dependencies, and configuring reverse proxies, firewalls, and load balancers.
PaaS products build on what IaaS products offer. In addition to providing physical infrastructure, they offer solutions for challenges associated with deployment, server configuration, security configuration, and maintainability. These solutions allow developers to focus almost exclusively on application development.
While PaaS products can solve many of the challenges of configuring and running a web application, they come with the significant drawback of vendor lock-in. Moving an application away from a PaaS product can be challenging.
BaaS solutions abstract away much of the process of creating an application’s backend by configuring the application and data tiers and auto-generating an API to facilitate communication with the presentation tier. They commonly offer managed cloud hosting services that streamline the hosting process and can automatically scale backend infrastructure to handle varying traffic levels. BaaS solutions generally provide a suite of additional features as well, such as file storage, observability, and real-time data subscriptions.
Most BaaS products build on the features offered by PaaS by abstracting server configuration and the application and data tiers. These features enable developers to focus on their application's business logic and functionality.
The more an application follows norms, such as utilizing a REST API to interact with entities stored in a database, the more likely a BaaS can be helpful. A BaaS generally increases abstraction to increase developers' ease of use. However, if an application’s needs are unique, a BaaS product’s abstractions can be limiting.
Serverless functions, code that executes in cloud environments, are automatically managed and provisioned on demand by BaaS providers. These functions execute custom logic in response to events, extending the auto-generated API's capabilities beyond basic CRUD operations. These functions allow developers to adapt the backend to their specific needs.
While serverless functions offer significant benefits, they also introduce the following challenges.
Each BaaS offers a different feature set. The chart below highlights some of today’s more popular BaaS solutions and the features they offer.
Firebase is a powerful BaaS platform from Google that offers a wealth of development tools. As a closed-source product, Firebase doesn't allow you to self-host your application. This can be a drawback if data ownership and transparency are major priorities for your project. However, if Firebase's features align well with your needs, the convenience of its built-in tools might outweigh this limitation.
Supabase is a popular open-source option that offers more control. Developers can use Supabase’s managed hosting services or self-host with their preferred hosting provider, however, the self-hosting setup process can be quite complex. Like Firebase, Supabase uses serverless functions to extend the BaaS.
PocketBase is a lightweight BaaS that prioritizes developer flexibility. Unlike Firebase’s managed hosting, Pocketbase runs from a single executable, allowing deployment on any Linux server. This ease of deployment comes at the cost of a more manual server configuration process. Its extensibility utilizes local functions in Golang or a limited Javascript runtime. These features cater to developers who value control and customization over managed convenience.
Among the BaaS options, we saw an opportunity to build a BaaS that was easy to self-host and customize with JavaScript, a language familiar to most front-end engineers. One that could streamline application development by providing database management and API server functionality, while prioritizing ease of use. A solution intended for small teams that value flexibility and don’t need all the features offered by a larger BaaS. This is reflected in the design choices we made for Pinniped:
We wanted Pinniped to be as simple as possible to deploy. Because we were targeting small applications, we decided to run Pinniped as a single process with a combined application and data tier. This departs from the typical three-tier architecture and has significant drawbacks when it comes to scalability, as Pinniped cannot scale the server and database separately. We’ll continue exploring this decision's ramifications throughout the case study. The upsides of using a single process are reduced latency and minimal configuration. This allows developers to quickly run a Pinniped application in a local development environment or any production environment with Node.js.
Pinniped’s business logic and API functionality can be customized with locally executed JavaScript, avoiding the increased network latency and configuration complexity of serverless functions. Pinniped’s extensibility functions run in a Node JS runtime, supporting up-to-date JavaScript ES6 features, the Node API, and npm packages familiar to most Javascript engineers. This implementation also affects Pinniped's scalability, but we’ll defer that discussion until later in the case study.
We wanted Pinniped to be easy to use, so we made it simple to interact with and integrate into any application. Simple workflows allow users to create a Pinniped application, deploy it onto a virtual private server (VPS), and manage it through an admin dashboard.
Pinniped fits into the BaaS landscape as an option for developers who want a BaaS to build small applications in Javascript quickly. Features such as user file storage or real-time data subscriptions are left to the developer to implement through Pinniped’s strong extensibility support. Pinniped’s scalability is limited, so it’s unsuited for large-scale or enterprise applications that expect to see high traffic.
In this section, we’ll explain how a BaaS has to be structured differently from a standard web application and how those differences drove Pinniped’s design.
If we were building just one web application, our backend would be designed exclusively for that web application. But because we don’t know how each user of Pinniped will build their backend, we need a general-purpose way of representing the entities they will need in their databases. Users also need user-friendly tools that create and edit database schemas.
Pinniped exposes a special set of endpoints for interacting with table schemas. These routes listen for requests to create, update, and delete table schemas in the database.
We created an admin UI to simplify interacting with these routes and to act as the primary means to modify schema. Instead of writing SQL statements, users fill in a form that defines the schema for a table. Our goal was to make the UI flexible enough to support users’ differing requirements. To make the application work with a dynamic group of database tables, we needed a way to store metadata about each table a user creates.
We keep essential metadata in a table called tablemeta. Each row within this table contains a particular table’s name, a unique tableId, and API access rules. Additionally, the tablemeta row includes details about a table’s columns, such as column names, data types, data validation constraints, and relationships. Whenever we create, update, or drop a table within Pinniped, the corresponding row in tablemeta is adjusted to reflect that change.
Developing an application that interacts with a database can introduce challenges. One of these challenges is the impedance mismatch that occurs when data is represented as objects in the application but is stored as tables and rows in the database. This mismatch leads to the need for a translation layer between the application and the database to manage data flow effectively.
Many applications use a tool called an ORM (Object Relational Mapping), which serves as that translation layer. ORMs have different translation methods, but they usually use a table model that ingests raw table data and makes it usable by the application as an object. These table models also have methods for interacting with the database.
For Pinniped, we needed a similar translation layer to manage the database schema and interact with the metadata stored about each table. Because ORMs aren’t designed for the dynamic needs of a BaaS, we opted to build a custom translation layer. This choice removed the extra layer of abstraction that an ORM would add, which gave us more flexibility.
We utilize Knex, a query builder library, and a Table model class that serves as an in-application representation of a specific database table and its metadata. When Pinniped receives a schema-related request, it instantiates an instance of the Table class, which we then use to build a schema migration file we pass to Knex to execute.
Handling schema changes through migration files is more complicated than simply updating tables directly, but it has benefits. Migration files create a reversible, repeatable record of how schemas are updated. They are self-documenting, easy to roll back, and easy to test in a development environment before being used in production. These benefits of data integrity make up for the complexities of dynamically generating these migration files.
Creating a REST API to interact with an application’s data can be a tedious, repetitive process. BaaS solutions automate this process by generating an API layer that reflects the application's database schema. The challenge of auto-generating an API stems from the application's schema being unique to the developer's needs. So, how can the API accurately create endpoints referencing the right resources when we don’t know what those resources will be beforehand?
In contrast to traditional REST APIs that require pre-defined routes for each resource, Pinniped utilizes a dynamic approach. Rather than representing each table as a top-level resource, as in a standard backend, Pinniped uses a single resource, tables, as the base resource for all CRUD operations. Every table is an instance of tables with an associated ID and name. One level deeper, each table has a sub-resource known as rows. Similar to tables, a single row is an instance of rows. Pinniped mounts routes with dynamic table names, using URL parameters instead of static routes based on the resource.
By taking this approach, Pinniped only needs to register a single set of route handler functions to provide a RESTish API layer for an application’s dynamic database schema.
We developed the Pinniped-SDK npm package to make Pinniped’s API easy to interact with. Instead of learning the API signatures when building their presentation tier, developers can use the simple Javascript SDK methods to interact with the backend.
In a typical backend, using a unique handler function for each resource and operation gives developers fine-grained control over how each route should behave. For example, developers can define authorization and data validation rules on a route-by-route basis to control interactions with each route. A unique handler function also allows them to implement application-specific functionality (extensibility) in addition to the standard REST API operations.
With Pinniped’s dynamic RESTish API layer, each route handler function needs to work in a general-purpose way that supports any underlying database schema and its unique requirements. This complicates the following endpoint-specific concerns:
Pinniped enforces endpoint and operation-specific data validation using the stored metadata about each table and middleware functions. This process ensures incoming data adheres to the expected structure for each table.
The LoadTableContext middleware retrieves the relevant table metadata and uses it to construct a dynamic Table object in memory using the Table model class, as previously described. This model represents the valid data format for the targeted table.
A separate data validation middleware function intercepts the request before it reaches the final route handler. This middleware compares the request’s body (containing a row's new or updated data) against the corresponding Table model generated earlier. If validation rules are violated (e.g., exceeding the maximum length for a field or missing a required value), the middleware sends a 400 Bad Request response back to the client. This error response includes a clear message detailing the constraint violation, allowing the user to rectify the issue and resubmit the request.
For security concerns, developers restrict data access for different users in their applications. For example, an e-commerce site wouldn’t want all users to have access to each customer's payment information. Developers give users access to select data by labeling them with authorization roles. Then, before allowing access to a user, the server verifies their authorization level. Here’s how Pinniped achieves access control with a dynamic set of database tables.
Since Pinniped’s combined application and data layers are not built to scale horizontally, session-based authentication with a local session store was the perfect fit for our needs.
Pinniped implements session-based authentication through the Express-Session npm package. This package provides a middleware that creates a session and cookie for each unique client that interacts with the backend. When a client makes subsequent requests to the backend, their browser automatically sends this cookie in the request headers. The session ID stored inside the cookie allows Express-Session to retrieve the client’s info from the persistent session store and make it available in memory via the req.session object.
Pinniped supports two simple user roles: Admin and User. Upon successful login, a user’s ID, username, and role are added to their session data as a user object on req.session, indicating successful authentication. When the user logs out, this object is removed.
Each table has an API rule associated with the actions that can be performed on it. These rules and their corresponding values, public, user, creator, or admin, are stored in tablemeta and are made available by LoadTableContext, as an instance of the Table model on res.locals. This context is then available for all remaining middleware to utilize.
With the relevant Table model instance, accompanying API rules, and the client’s session information available in memory, Pinniped’s AuthCheck middleware function has all the information it needs. It then either sends an error response to the client or passes the request along to its intended endpoint.
A final authorization check occurs once the request reaches the endpoint route handler function. If the request’s API rule value is creator, the row(s) the client intends to interact with are retrieved and compared to the current session user’s ID. If the current session user’s ID does not match the value of the creator column on the requested row(s), a 403 Forbidden response will be sent to the client, preventing them from performing their intended action.
Beyond the core CRUD operations provided by auto-generated APIs, most BaaS products address the lack of unique route handlers by offering mechanisms for developers to define and execute custom application logic. Serverless functions are a common approach for achieving this functionality, but they come with inherent tradeoffs.
We chose an alternative approach with Pinniped to minimize these common BaaS extensibility pain points for our target developers.
Pinniped's local extensibility framework allows developers to register custom event handlers directly within their backend project. This approach offers several advantages:
However, local extensibility also comes with a key limitation: scalability. Unlike serverless functions, which can scale independently in a traditional backend architecture, local functions share the resource pool of the Pinniped backend. This limits their ability to handle high volumes of concurrent requests compared to serverless solutions.
Despite this limitation, Pinniped’s focus on ease of use and portability justified this tradeoff.
Pinniped’s local extensibility framework allows developers to register Custom Routes and Event Handler Functions.
An index.js file serves as each Pinniped project’s main entry point and is used to configure and run the application. Within this file, the pinniped npm package is imported and used to create an instance of the pinniped class, known as app. This app instance is used to register custom routes and event listeners before it configures and starts an Express web server that will serve the application.
Most API operations trigger events, including CRUD operations, schema changes, and account management actions. These events provide access to the Express request and response objects along with any relevant data from the route that triggered the event. This allows developers to write flexible event handler functions that can manipulate data, interact with external services, or perform any other actions needed by their unique application.
Notably, developers can send custom responses (as shown below) or perform actions unrelated to the response, (such as logging, analytics, server-sent events, etc) and let the route handler that triggered the event send its default response.
In the traditional three-tier architecture, the database lives on its own machine, and the application communicates with it over a network. This kind of architecture has many benefits; however, we wanted to keep the entire Pinniped backend as simple as possible and deliver it as a single npm package. This goal ruled out running the application and database on separate machines and led us to consider using an embedded database.
An embedded database operates on the same machine as the application server, which requires less configuration from the user. Embedding also removes network latency and instability between the database and the application. Removing the need for network protocols also allows for more straightforward communication between the application and the database. These speed increases enable them to handle more read/write operations than an equivalent networked database. An embedded database fit our goals to keep Pinniped portable and easy to use, but it wasn’t without drawbacks.
Embedding the database means that the application can not scale separately from the database, and multiple application servers can’t use the same database. If a traffic bottleneck is reached, traditional three-tier architecture scaling strategies are impossible. Those applications that expect to see such high volumes aren’t the intended users of Pinniped, so we decided the tradeoff for portability and simplicity was worth it. As a result, Pinniped’s architecture resembles two distinct tiers, with the database and application logic sharing one.
Having chosen to embed the database, we had a few primary concerns when deciding which database to use: speed, reliability, documentation, and library support. One option that aligned well with our needs was SQLite, a mature, widely used, speedy SQL database with plenty of documentation and supporting libraries. It met our criteria and was also successfully used as the database for PocketBase, which gave us confidence that it would work for our use case.
After choosing to run our backend as a single process delivered as an npm package, we considered a few different ways of packaging and distributing the project.
We considered Docker because of its portability benefits: we could include all our configurations and dependencies in the Docker image and run Pinniped on any platform that supports Docker. However, because of our single-process architecture, many of Docker's strengths, like orchestrating multiple nodes and scaling them independently, wouldn’t benefit us.
For us, the drawback of requiring users to manage Docker as an additional dependency was significant. The simplicity of an npm-only approach made it particularly accessible to Pinniped’s target audience: front-end developers who likely already use it and know how to configure it in production environments.
We also explored compiling Pinniped into an executable file. This approach's primary benefit was further reducing the dependencies needed to run the application in production. However, building to an executable with Node.js is not fully supported, and the technical challenges accompanying the process made it untenable.
We’ve attempted to make deploying Pinniped as straightforward as possible while retaining flexibility so users can deploy however they want. The default route for deploying Pinniped is to develop the backend locally, set schema and add any customization code, set any configurations, place the project on a VPS, install Node.js, and run the application. Application configurations are contained in a .env file which allows you to specify the domain for your application, the port for the server to listen on, the CORS whitelist, the session secret, and various configurations related to Pinniped’s automated TLS certificate feature.
The application serves the Admin UI and API, at the specified domain and port.
Hypertext Transfer Protocol Secure (HTTPS) is the primary communication protocol for sending information between a web server and a client. It secures the communication between the web server and the client by encrypting the data in transit. Its importance comes from protecting the data packets from on-path attacks—malicious attempts to access the data. Since HTTPS is pivotal to securing communication on the internet, Pinniped automates certifying the application to use HTTPS.
If the VPS IP that Pinniped is running on has a domain name pointing to it, the domain variable in the .env configuration file can be set. Then on start-up, Pinniped will check for a TLS certificate. If there is no certificate or it is expired, Pinniped will attempt to obtain one from Let’s Encrypt. To do this, we start a server to listen on port 80 and initialize the challenge request to Let’s Encrypt. When our server satisfies the challenge, Let’s Encrypt issues the certificate. We bring down the challenge server and start the Pinniped application on port 443 (HTTPS) using the new certificates while redirecting any port 80 (HTTP) traffic to 443.
Pinniped performs a daily check of the certificate. If it is within 30 days of expiring, the challenge server will attempt to renew it. If successful, the server restarts with the new certificate.
While Pinniped will work in any local or cloud environment with NodeJS installed, the Pinniped-CLI provides an easy-to-use deployment pipeline to host a local Pinniped application on an AWS EC2 instance. After installing and configuring Amazon’s AWS CLI tool with AWS credentials, the Pinniped-CLI tool walks users through the provisioning and deployment process. The Pinniped-CLI also automates the process of starting, stopping, and updating a deployed Pinniped application via a secure shell (SSH) connection.
Right now, Pinniped’s CLI tool only supports AWS. We’d like to expand its functionality to include more major cloud providers.
While Pinniped has a process for database backups, it would be ideal to add an option to integrate with a more robust disaster recovery tool such as Litestream which automatically replicates the database to offsite storage.
File Storage
File storage is a commonly provided BaaS offering, and there are many situations when Pinniped users might want native support for it. It would be great to allow for both local storage and integration with cloud storage solutions.
Type Flexibility
An oddity of SQLite databases is that they don’t strictly type columns. It makes SQLite unique in the SQL world and is useful when you don’t know what data types a user may want to store. We’d like to implement these flexible types to increase the range of what users can do with Pinniped.