Going Static: Episode II — Attack of the Comments

Offloading comments to an external service like Disqus¹ has always felt like a necessary evil to me when building Jekyll sites.

Convenient to embed a small bit of <script> voodoo into your pages, sure. But kiss goodbye to controlling the user experience, look and feel, data, and privacy. The alternatives haven’t been all that great for the statically-minded unless you were willing to make some compromises…

Self-Hosted Comment Systems

Isso² describes itself as “commenting software similar to Disqus.” You host a SQLite database and embed some JavaScript on your pages (just like Disqus and friends) and you’re ready to roll. After freeing my content from Wordpress and “going static” I really didn’t want to manage a database again just to have comments on my site. So these solutions were out.

Static Comments

What I really was searching for was a commenting system to compliment the rest of my Jekyll generated site.

Over the years I’ve come across several solutions that seemed promising:

• Use mailto links (how very retro ) to email comments for inclusion into post.
• Use PHP to do something similar.
• Leverage GitHub’s issue tracker.

Jekyll plugins like Jekyll::StaticComments and Jekyll AWS Comments were pretty close to what I was looking for. A PHP <form> captures a comment, converts into YAML and emails it over to be placed in a prescribed location. Then with the help of a Liquid for loop, comments are displayed on the appropriate pages.

I can’t really explain it, but something about using PHP with Jekyll felt off to me. And so I didn’t pull the trigger on these solutions either.

It wasn’t until I discovered Eduardo Bouças’s blog post “Rethinking the Comment System for My Jekyll Site” and the launch of Staticman 2.0 that I finally decided to ditch Disqus. Starting the process of migrating years of comment data and integrating them into the rest of my site’s statically generated content.

Enter Staticman

On paper and in practice Staticman was just the app I was looking for to power static-based commenting on my site.

• Designed to work with Jekyll and GitHub Pages.
• Free and open source. Run it on your own server as a Node.js app or go the free hosted route.
• Complete control over the data/content, user experience, and user interface.
• Not just for comments! Perfect for any sort of user generated content: reviews, comments, polls, and more.
• User submitted content can be merged in automatically or moderated.

Getting Started

Much like building my first Jekyll site, I found the process of integrating Staticman into my workflow very rewarding. It was nice to get dirty again crafting markup, styling <form>s, and giving comments a feel that fit the rest of my site.

Thankfully I didn’t have to start from scratch as I was able to draw inspiration from the Staticman demo site — Popcorn and Eduardo Bouças’s personal site. The documentation for Staticman does a good job of explaining how to set things up so definitely give that a read first to familiarize yourself with what the app can do.

Building the Form

I set my gaze on squaring away the “Leave a comment” submission form first. Seemed like an easy target as the styling of various form elements like <input>, <label>, <textarea> and buttons were already done as part of my living style guide.

All it really needed for completion was a decision on what fields I wanted to capture, and a little bit of JavaScript for events handling and submission. Arriving at this for my post__comments.html include (class names and Liquid removed for brevity).

<form id="comment-form" method="post" action="https://api.staticman.net/v1/entry/{{ site.repository }}/{{ site.staticman.branch }}">
  <fieldset>
    <label for="comment-form-message">Comment</label>
    <textarea type="text" rows="3" id="comment-form-message" name="fields[message]" tabindex="1"></textarea>
  </fieldset>
  <fieldset>
    <label for="comment-form-name">Name</label>
    <input type="text" id="comment-form-name" name="fields[name]" tabindex="2" />
  </fieldset>
  <fieldset>
    <label for="comment-form-email">Email address</label>
    <input type="email" id="comment-form-email" name="fields[email]" tabindex="3" />
  </fieldset>
  <fieldset>
    <label for="comment-form-url">Website</label>
    <input type="url" id="comment-form-url" name="fields[url]" tabindex="4"/>
  </fieldset>
  <fieldset class="hidden" style="display:none;">
    <!-- used by Staticman to generate filenames for each comment -->
    <input type="hidden" name="options[slug]" value="{{ page.slug }}">
    <!-- honey pot field used to filter out spam -->
    <label for="comment-form-location">Not used. Leave blank if you are a human.</label>
    <input type="text" id="comment-form-location" name="fields[hidden]" autocomplete="off"/>
  </fieldset>
  <fieldset>
    <button type="submit" id="comment-form-submit" tabindex="5">Submit Comment</button>
  </fieldset>
</form>
<!-- End new comment form -->

Staticman’s documentation covers this in more detail, but essentially I’m adding fields[] values to the name attributes. message, name, email, and url fields are then used to generate a .yml similar to this:

message: "![Bill Murray](http://www.fillmurray.com/400/300)\r\n\r\n“It's hard to be an artist. It's hard to be anything. It's hard to be.”"
name: Bill Murray
email: b0caa2a71f5066b3d90711c224578c21
url: ''
hidden: ''
date: '2016-08-11T19:33:25.928Z'

A note on hidden and date fields you may have noticed in the sample comment above:

hidden is used as a spam deterrent in the form of a honeypot. The thought is a human wouldn’t fill out an input they can’t see, but a spam bot may. Adding fields[hidden] to this input and not placing it in the allowedFields array in our Jekyll config, instructs Staticman to reject the entry. Hopefully filtering out bots who are dumb enough to populate it with something.

date is captured when the entry is generated by Staticman. Its format can be changed from iso8601 (default) to timestamp-seconds or timestamp-milliseconds.

Interactions and State

Using Popcorn’s main.js as a guide I added all the AJAX goodness, alert messaging, along with disabled and loading form states.

To avoid disrupting the flow too much I went with inline alert messaging directly above the submit button.

And to improve the user experience upon submission the submit button’s text changes to Loading..., becomes disabled, and an animated SVG icon inserted for bit of extra flare.

$(form).addClass('disabled');
$('#comment-form-submit').html('<svg class="icon spin"><use xlink:href="#icon-loading"></use></svg> Loading...');

If the form is successfully submitted a message appears notifying the user that the comment has been received and is pending moderation. Since my site takes a bit to generate with Jekyll I felt it necessary to convey this to the user, hopefully avoiding duplicate submissions.

With smaller sites hosted with GitHub Pages this becomes less of a problem, as they build much faster. Especially true if you decide to go with the auto merge option and skip moderating comments.

Displaying Comments

There’s a bunch of Staticman settings available to you, but forget all that right now. For this next step all you really need to know is static comment files will live in _data/comments/<post slug>/. By predictably placing them here we will be able to access their contents from the following array: site.data.comments[page.slug].

With this array we’ll be looping through it with for just like you would with site.posts to spit out a list of all posts. But first we’ll use an assign tag to rename the array and apply a sort³ filter on the objects. This will order them by filename, which in our case should be chronological⁴.

{% assign comments = site.data.comments[page.slug] | sort %}
{% for comment in comments %}
  show a comment
{% endfor %}

Since I’m capturing message, name, email, and url in the comment form these will be the same fields I’ll want to pull from to build each comment. Using an assign tag again we’ll cleanup variable names like comment[1].avatar into just avatar. Which will then be used to pass parameters into the comment.html include:

{% assign comments = site.data.comments[page.slug] | sort %}
{% for comment in comments %}
  {% assign avatar = comment[1].avatar %}
  {% assign email = comment[1].email %}
  {% assign name = comment[1].name %}
  {% assign url = comment[1].url %}
  {% assign date = comment[1].date %}
  {% assign message = comment[1].message %}
  {% include comment.html index=forloop.index avatar=avatar email=email name=name url=url date=date message=message %}
{% endfor %}

If done correctly the values and strings in a data file like _data/comments/basics/comment-2014-02-10-040840.yml

---
id: comment-1237690364
date: '2014-02-10 04:08:40 +0000'
updated: '2014-02-10 04:08:40 +0000'
post_id: "/basics"
name: Tamara
url: ''
message: "This? This is freakin' awesome! Thanks so much for sharing your mad skills and expertise with us!"

Should populate _includes/comment.html and spit out as the following HTML:

<article id="comment1" class="js-comment comment" itemprop="comment" itemscope itemtype="http://schema.org/Comment">
  <div class="comment__avatar-wrapper">
    <img class="comment__avatar" src="https://www.gravatar.com/avatar/?d=mm&amp;s=50" srcset="https://www.gravatar.com/avatar/?d=mm&amp;s=100 2x" alt="Tamara" height="50" width="50">
  </div>
  <div class="comment__content-wrapper">
    <h3 class="comment__author" itemprop="author" itemscope itemtype="http://schema.org/Person">
      <span itemprop="name">Tamara</span>
    </h3>
    <div class="comment__date">
      <a href="#comment1" itemprop="url">
      <time datetime="2014-02-09T23:08:40-05:00" itemprop="datePublished">February 09, 2014 at 11:08 PM</time>
      </a>
    </div>
    <div itemprop="text"><p>This? This is freakin’ awesome! Thanks so much for sharing your mad skills and expertise with us!</p></div>
  </div>
</article>

Looking like this when styled with CSS:

There’s not much magic in the comment.html include — some structured data markup sprinkled about and a few Liquid conditionals for displaying author avatars and URLs.

Setting Up Staticman

With the front-end portion of my static-based comment system squared away, it was time to configure Staticman. Because I went with the hosted version, it only took a few quick steps to setup.

Adding Staticman as a Collaborator

First you need to grant Staticman access to your Jekyll repository on GitHub. You don’t have to actually host the site there (I use Media Temple for that), but it does need to be a standard Jekyll site with valid _config.yml.

Following the docs I added GitHub username staticmanapp as a collaborator and then pinged https://api.staticman.net/v1/connect/{your GitHub username}/{your repository name} as instructed to accept the invitation.

Configuring Staticman

Staticman is configured by settings defined in your Jekyll _config.yml under a staticman object. There’s a whole list of stuff you can configure — the important stuff being allowedFields, branch, format, moderation, and path.

There’s also an undocumented generatedFields⁵ setting that is useful for time stamping each file Staticman creates.

I ended up with the following settings in my _config.yml:

staticman:
  allowedFields          : ['name', 'email', 'url', 'message']
  branch                 : "master"
  commitMessage          : "New comment."
  filename               : comment-{@timestamp}
  format                 : "yml"
  moderation             : true
  path                   : "_data/comments/{options.slug}"
  requiredFields         : ['name', 'email', 'message']
  transforms:
    email                : "md5"
  generatedFields:
    date:
      type               : "date"
      options:
        format           : "iso8601"

In case spam makes it through, I’d like another layer of “protection” to block it. Setting moderation: true will make Staticman send a pull request whenever a new comment entry is submitted. At this point you can examine the content inside of the PR and decide if you want to merge or close it.

When hosting with GitHub Pages, a merge will instantly force Jekyll to rebuild the site — publishing the comment. Since I self host I have the extra step of pulling from remote, before building locally and deploying via rsync.

Hooking Up the Form

For your forms to work with Staticman they need to POST to:

https://api.staticman.net/v1/entry/{your GitHub repository}/{your repository name}/{the name of the branch}`

Instead of hard-coding the site repository and branch strings into this endpoint, use site variables defined in _config.yml instead. eg: {{ site.repository }} and {{ site.staticman.branch }} respectively.

# sample _config.yml

repository: "mmistakes/made-mistakes-jekyll"
staticman:
  branch: "master"

Hitting the Staticman endpoint should trigger the success and error messages in our comment <form>. Firing up the console in your browser of choice can also give you some more hints on what’s going on if you encounter any snags.

For example if all of the required fields aren’t filled out an error like this could hit the console:

Object {readyState: 4, responseText: "[{"code":"MISSING_REQUIRED_FIELDS","data":["name","email","message"]}]", responseJSON: Array[1], status: 500, statusText: "error"}

Publishing Comments

If configured correctly you should receive a pull request notification on GitHub each time a comment entry is submitted. Look the commit over (if you’re moderating them) and merge pull request to accept or close to block it.

Migrating Disqus Comments

It was now time to deal with the 500+ Disqus comments I’ve accumulated. A good chunk of them had valuable content worth keeping, so I didn’t exactly want to dump them all.

I came across a Rake task by Patrick Hawks, aptly named jekyll-disqus-comments that downloads Disqus posts as YAML files via the Disqus API.

With some modifications I was able to get it working with my site and _posts files.

Installing

Copy the following files to the root of your Jekyll project folder.

• _rake/disqus_comments.rake
• Rakefile (Not necessary if you already have a Rakefile that loads _rake/*)

Obtain a Disqus API Public Key

To use the plugin, you will need to obtain a public key from the Disqus API and add it to your _config.yml. You can do this by:

Step 1. Register new application.

Step 2. Setup application using suggested configuration below:

Label: <Name of application> eg. Jekyll Disqus importer
Description: Convert comments into static files.
Website:
Domains: disqus.com
Default Access: Read only

Step 3. Add the following lines to your _config.yml:

comments:
  disqus:
    short_name: YOUR-DISQUS-FORUM-SHORTNAME-HERE
    api_key:    YOUR-DISQUS-PUBLIC-KEY-HERE

Run Import Task

Import comments from Disqus by running rake disquscomments from the CLI. If it completes successfully you should find a set of .yml files in _data/comments/<post-slug>/ similar to this:

├── _data
|  └── comments
|      └── 365-days-of-drawing
|      |   └── comment-2013-08-30-162902.yml
|      |   └── comment-2013-08-30-204505.yml
|      └── basics
|          └── comment-2014-02-10-040840.yml

Each with YAML Front Matter data similar to this:

---
id: comment-1237690364
date: '2014-02-10 04:08:40 +0000'
updated: '2014-02-10 04:08:40 +0000'
post_id: "/basics"
name: Tamara
url: ''
message: "This? This is freakin' awesome! Thanks so much for sharing your mad skills and expertise with us!"

Key names correlate with the ones defined earlier with Staticman, along with a few specific to Disqus: id, updated, and post_id that aren’t currently used on the site.

I’m a little obsessive so I went through a ton of old comments adding GitHub Flavored Markdown⁶ backticks to improve the reading experience of code blocks. Having properly formatted code blocks in comments looks so good I couldn’t pass it up.

Pulling this off with Disqus required way more work and didn’t support Markdown.

Troubleshooting

When running rake disquscomments I ran into several warnings like this:

Comments feed not found: <domain.com>/post-slug/

For posts that I knew didn’t have any comments this wasn’t a problem, but for those that did it was a real head scratcher. Eventually I discovered that ident in disqus_comments.rake wasn’t matching the style of post permalinks used on my site.

I was able to determine what Disqus was expecting for id’s and adjust the plugin by:

1. Exporting all my Disqus comments as XML.
2. Opening the Disqus XML file.
3. Looking at the <link> elements eg. <link>https://mademistakes.com/mastering-paper/contour-drawing/</link>

By playing around with the following line in disqus_comments.rake I finally sorted it out:

# site.url + post.id + trailing slash
ident = site['url'] + post.id + '/'

Final Thoughts

Treating comments as content and integrating them into the same build process as the rest of my site has been an informative and rewarding experience. By successfully migrating over 500 comments away from Disqus I was able to:

• Style them consistently and match the rest of the site’s design.
• Improve the appearance of <code> blocks within comments.
• Make it easier for visitors to leave a comment without having to create a Disqus account.

SEO Implications

The comments left on many of my posts often contain corrections, follow-up, and other valuable post content. From earlier tests it did seem as if search engines were able to crawl the embedded Disqus JavaScript comments and partially index them. Time will tell if I’ll see any SEO lift now that comments are part of the HTML and marked up as structured data.

Spam Slipping Through

Seems to only happen on my older posts or ones that rank well in Google and friends. As no one is really adding valuable comments to these I’ve added a comments_locked conditional to disable the comment form on specific pages.

{% unless page.comments_locked == true %}
  <!-- comment form -->
{% else %}
  <p><!-- comments locked messaging --></p>
{% endunless %}

I’ll have to keep an eye on the effectiveness of this method, or possibly find a tastier honeypot to better combat spam bots.

Comment Replies

One thing I miss since leaving Disqus, are comment notifications. Sure you can setup GitHub to notify you of each Staticman pull request, which will in turn clue you in that you have a new comment. What’s missing is a way to notify the commenter that there’s been a reply to their comment.

Less likely a commenter will return to the page to see if a reply was made without the nudge of a notification. Wordpress and friends has the whole “subscribe to comments” feature which could apply here I suppose.