Beyond the Basics (Part V): Formik, D3, and More!

Once you've finished this post, you'll have a template for easily creating forms using Formik, as well as experience with D3 visualizations!

If you haven't read the first post in the series, this is a step by step guide on building a SaaS app that goes beyond the basics, showing you how to do everything from accept payments to manage users. The example project is a Google rank tracker that we'll build together piece by piece, but you can apply these lessons to any kind of SaaS app.

In the last post, we implemented user authentication in both Flask and React. Now that the basic structure is in place, we're going to implement a full "slice" of the application – we'll build the proxies page, where users can add and delete crawler proxies. I call it a slice because we'll build every part of the functionality in this post, from the data model to the user interface.

Table of Contents

Part I: Building the Google Search Scraper

Part II: Production Ready Deployment with NGINX, Flask, and Postgres

Part III: Flask, SQLAlchemy, and Postgres

Part IV: User Authentication with Flask and React

Building the Proxy Connection Data Model

The proxy model will contain all the details needed for Puppeteer to crawl Google using that connection, such as URL, username, and password. We'll also keep track of some stats, such as a counter of how many times the proxy is blocked, which will come in handy later when we want to visualize proxy performance with D3.

class ProxyConnection(db.Model):
    __tablename__ = "proxyconn"

    id = db.Column(db.Integer, primary_key=True)

    user_id = db.Column(
        db.Integer,
        db.ForeignKey("user.id", ondelete="CASCADE"),
        index=True,
        nullable=False,
    )

    proxy_url = db.Column(db.String, nullable=False)
    username = db.Column(db.String, nullable=False)
    password = db.Column(db.String, nullable=False)

    # Can this proxy support multiple parallel requests?
    allow_parallel = db.Column(
        db.Boolean, default=False, server_default="f", nullable=False
    )

    success_count = db.Column(db.Integer, default=0, server_default="0")
    block_count = db.Column(db.Integer, default=0, server_default="0")
    no_result_count = db.Column(db.Integer, default=0, server_default="0")
    consecutive_fails = db.Column(db.Integer, default=0, server_default="0")

    # Proxy is currently in use (only applicable when allow_parallel = 'f').
    engaged = db.Column(db.Boolean, default=False, server_default="f")

    # Must wait at least this long before allowing another request.
    min_wait_time = db.Column(db.Integer, default=0, server_default="0", nullable=False)

    # Use random delay when proxying with a static IP to avoid blocks.
    random_delay = db.Column(db.Integer, default=0, server_default="0", nullable=False)

    last_used = db.Column(db.DateTime, index=True, nullable=True)

    user = db.relationship("User")

I'll also define a Marshmallow schema as part of the data model. This will make it easier to accept form submissions in JSON format, as well as return data from the API.

from marshmallow_sqlalchemy import SQLAlchemyAutoSchema, auto_field
from app.models.proxyconn import ProxyConnection


class ProxySchema(SQLAlchemyAutoSchema):

    class Meta:
        model = ProxyConnection
        load_instance = True

    # Set password to load_only so that it is accepted during form
    # submissions, but never dumped back into JSON format.
    password = auto_field(load_only=True)

The SQLAlchemyAutoSchema class is a great convenience, because it automatically maps the model class to Marshmallow fields. When we need to treat a certain field differently, such as password here, it's easy enough to override the functionality.

Whenever new models are created in the project, we need those models to exist as actual tables in Postgres. We'll go through performing database migrations later, but for development purposes, it's easy to create new tables in Postgres using the Flask manager script.

docker exec -it openranktracker_app_1 python manage.py shell
>> db.create_all()

Creating and Deleting Proxy Connections

We're going to need GET, POST, and DELETE methods for the proxy model. Fortunately, this is pretty straightforward, especially because we'll be using Marshmallow to handle validation and serialization.

The ProxiesView handles creating new proxies, as well as returning all proxies belonging to a specific user.

from flask import request, g, abort
from marshmallow import ValidationError

from app.api.auth import AuthenticatedView
from app.models.proxyconn import ProxyConnection
from app.serde.proxy import ProxySchema
from app import db


class ProxiesView(AuthenticatedView):
    def get(self):
        return (
            ProxySchema().dump(
                ProxyConnection.query.filter_by(user_id=g.user.id)
                .order_by(ProxyConnection.id)
                .all(),
                many=True,
            ),
            200,
        )

    def post(self):
        try:
            proxy = ProxySchema().load(request.get_json(), session=db.session)
            proxy.user = g.user
        except ValidationError:
            abort(400)

        db.session.add(proxy)
        db.session.commit()

        return ProxySchema().dump(proxy), 201

We use the global Flask context to filter proxies by user, and to assign an owner to new proxies. The POST method simply returns a 400 Bad Request if the Marshmallow validation fails. This should not happen, however, because the front-end form will have its own validations to prevent bad submissions. More complex validations that can only be done on the back-end are sometimes necessary, but in this case we're concerned only with whether required fields are submitted.

The ProxyView will handle deletion of proxy connections.

from flask import g, abort

from app.api.auth import AuthenticatedView
from app.models.proxyconn import ProxyConnection
from app import db


class ProxyView(AuthenticatedView):
    def delete(self, proxy_id):
        proxy = ProxyConnection.query.get(proxy_id)

        if proxy.user_id != g.user.id:
            abort(403)

        db.session.delete(proxy)
        db.session.commit()

        return "", 200

Pretty simple, really! Unless you're trying to delete proxies that don't belong to you. In that case, we abort with a 403.

Finally, we make a quick stop in app/api/__init__.py to associate the new handlers with API routes.

api.add_resource(ProxyView, "/proxies/<int:proxy_id>/")
api.add_resource(ProxiesView, "/proxies/")

Creating the New Proxy Form

Now that the database models and API routes are in place, we need a form for submitting new proxies. This won't be the first form in the app – after all, we already have sign up and login forms. This time around, however, we're going to get a little fancier, and use the Formik library.

The login and signup forms were very simple. The proxy form, however, has five fields and additional validations beyond whether something is required or not. Handling all of that with Formik should cut down the amount of code we'll need to write.

The first step in building the form will be to define default values, as well as the validations we need to perform. Let's look at the first part of the ProxyPopup.js module to see how that's done.

import { Formik, Form, Field } from "formik";
import * as Yup from "yup";

const defaultProxy = {
    proxy_url: "",
    username: "",
    password: "",
    min_wait_time: 60,
    random_delay: 10
};  

const proxySchema = Yup.object().shape({
    proxy_url: Yup.string().required(),
    username: Yup.string().required(),
    password: Yup.string().required(),
    min_wait_time: Yup.number()
        .positive()
        .integer()
        .required(),
    random_delay: Yup.number()
        .positive()
        .integer()
        .required()
});

The Yup library integrates seamlessly with Formik, and allows you to build up different combinations of validators with ease.

Formik itself provides a base Formik component that expects a function as its child. We'll define our form inside that function, and Formik will pass arguments that include a values object, as well as touched and errors objects.

We can use these objects to drive the styling of the form, as you can see below.

The form relies on the touched and errors objects to flag the username field as an error. The password input isn't flagged, even though it's required, because the touched object indicates that it hasn't experienced a blur event yet. The errors object is updated automatically according to the Yup schema we provided. Formik simplifies tracking all of this state information.

I'll include a sample of the above form here, slightly abbreviated for length.

<Formik
    initialValues={defaultProxy}
    onSubmit={onSubmit}
    validationSchema={proxySchema}
    validateOnMount
>
    {({
        values,
        errors,
        touched,
        handleChange,
        handleBlur,
        handleSubmit,
        isValid
    }) => (
        <Form onSubmit={handleSubmit}>
            <div className="formGroup">
                <label className="formLabel">Proxy URL</label>
                <Input
                    name="proxy_url"
                    onChange={handleChange}
                    onBlur={handleBlur}
                    value={values.proxy_url}
                    border={
                        touched.proxy_url &&
                        errors.proxy_url &&
                        `1px solid ${COLORS.warning}`
                    }
                    style={{ width: "100%" }}
                />
            </div>
            <div className="formGroup">
                <label className="formLabel">
                    Proxy Username
                </label>
                <Input
                    name="username"
                    onBlur={handleBlur}
                    onChange={handleChange}
                    value={values.username}
                    border={
                        touched.username &&
                        errors.username &&
                        `1px solid ${COLORS.warning}`
                    }
                    style={{ width: "100%" }}
                />
            </div>
        </Form>
    )}
</Formik>

You may notice that I'm using custom classes such as Input instead of normal HTML inputs. These are simply convenience classes created using styled components. I have created a handful of these commonly required elements in order to avoid redefining their CSS over and over.

The custom form elements and buttons can be found in the util/controls.js module.

import styled from "styled-components";
import { BORDER_RADIUS, COLORS, PAD_XS, PAD_SM } from "./constants";

export const Input = styled.input`
    color: ${COLORS.fg1};
    background-color: ${COLORS.bg4};
    box-sizing: border-box;
    padding: ${PAD_XS} ${PAD_SM};
    outline: none;
    border-radius: ${BORDER_RADIUS};
    border: ${props => props.border || "none"};
`;

export const Button = styled.button`
    background: none;
    border: none;
    border-radius: ${BORDER_RADIUS};
    outline: none;
    cursor: pointer;

    &:disabled {
        filter: brightness(50%);
        cursor: default;
    }
`;

Building the Proxy Dashboard with Flexbox

We can create new proxies now, but we also need a place to view existing proxies, and monitor their performance.

How many proxies are needed depends on how many keywords we'd like to track, but we can assume it's easily possible to have a dozen or more. We'll use flexbox to create a layout that works as a grid, collapsing eventually to a single column when there isn't much space to work with.

First we'll take a look at the JSX that produces the dashboard.

<div className={styles.container}>
    <div className={styles.buttonRow}>
        <PrimaryButton
            style={{ padding: PAD_SM, marginLeft: "auto" }}
            onClick={addProxyServer}
        >
            Add Proxy Server
        </PrimaryButton>
    </div>
    <div className={styles.proxyList}>
        {proxies.map(proxy => (
        <div key={proxy.id} className={styles.proxyContainer}>
            <ProxyConnection proxy={proxy} onDelete={deleteProxy} />
        </div>
        ))}
    </div>
</div>

The buttonRow div is a flex container that houses the Add Proxy button, which is displayed on the right side of the page. Instead of using float: right here, it's possible to use margin-left: auto to achieve the same result. The proxyList class is also a flex container, of course, but with the flex-wrap property added.

The nowrap default of flex-wrap means items spill outside of their container when there isn't enough space. By changing to wrap, the children are instead allowed to break to the next line.

This is the relevant CSS that makes it all happen.

.container {
    padding: var(--pad-md);
    padding-top: var(--pad-sm);
    box-sizing: border-box;
}

.buttonRow {
    display: flex;
    margin-bottom: var(--margin-md);
}

.proxyList {
    display: flex;
    flex-wrap: wrap;
}

.proxyContainer {
    margin-right: var(--margin-sm);
    margin-bottom: var(--margin-sm);
}

The outer container class applies some padding so that the dashboard isn't pressed to the edges of the page. Using box-sizing: border-box prevents that added padding from creating scrollbars.

Adding a Donut Chart Using D3

If you recall the schema of the proxy table, we're keeping track of how many successful and failed requests each proxy has made. We'll display a donut chart for each proxy as an easy way to see performance at a glance.

The three donut slices represent successful and blocked requests, as well as requests that returned no results (in amber).

We'll create a DonutChart component that works with any kind of data having up to 3 categories. The component expects a category prop that has positive, neutral, and negative keys that map to integer values.

Unlike the vast majority of the app, the DonutChart is a class-based component. This is necessary because D3 works directly with the DOM. As a result, we can't rely on the normal rendering cycle. Instead, we'll have to manually watch for prop changes to determine when a re-render is necessary.

Fortunately, for class-based components we can use componentDidUpdate to determine if a re-render is required.

componentDidUpdate(prevProps) {
    if (prevProps.category != this.props.category) {
        this.drawChart();
    }
}

This is a simple example, but in more complex cases, allows us to have fine-grained control over what happens when props are changed.

The drawChart method contains the actual D3 rendering logic.

drawChart() {
    const svg = d3.select(this.svgRef.current).select("g");

    const radius = Math.min(this.width, this.height) / 2;
    const donutWidth = 10;

    const arc = d3
        .arc()
        .padAngle(0.05)
        .innerRadius(radius - donutWidth)
        .outerRadius(radius)
        .cornerRadius(15);

    const data = [
        this.props.category.POSITIVE,
        this.props.category.NEGATIVE,
        this.props.category.NEUTRAL
    ];

    const pie = d3
        .pie()
        .value(d => d)
        .sort(null);

    // Select all existing SVG path elements and associate them with
    // the positive, neutral, and negative sections of the donut
    // chart.
    const path = svg.selectAll("path").data(pie(data));

    // The enter() and append() methods take into account any existing
    // SVG paths (i.e. drawChart was already called) and appends
    // additional path elements if necessary.
    path.enter()
        .append("path")
        .merge(path)
        .attr("d", arc)
        .attr("fill", (d, i) => {
            return [COLORS.success, COLORS.warning, COLORS.caution][i];
        })
        .attr("transform", "translate(0, 0)");

    // The exit() method defines what should happen if there are more
    // SVG path elements than data elements.  In this case, we simply
    // remove the extra path elements, but we can do more here, such
    // as adding transition effects.
    path.exit().remove();
}