Blogging with Emacs and Org

1. Quick Links
2. Introduction
3. Regex Replace with Elisp
4. Site Structure
5. Include and Generate Content
6. RSS Feed
7. Functions for Publishing
8. Results

1. Quick Links

I read quite a few other blog posts that described setting up org-mode as a static generator. Confession: sometimes I skipped all the words and went right for the elisp examples. To return the favor, here's my up to date code:

2. Introduction

Recently I've been using Emacs for more and more of my workflow. It's great to have the same environment regardless of platform. Packages like evil-mode, helm and magit make Emacs a joy to use.

But I had one major annoyance: this website was created with a hodgepodge of HTML, python, and markdown.

In this post I describe how I generate my site from Org source files.

If you aren't familar with Emacs or org-mode then here's a quick summary: Emacs is a lisp interpreter that comes with functions for editing text. It's a platform that allows you to build your own IDE. Org is an Emacs extension for writing, formatting, and publishing documents.

3. Regex Replace with Elisp

The first step was changing markdown files into org. I took it as an opportunity to practice elisp and regex. Here are a few functions I wrote that do search and replace on buffer.

;;; A macro for moving point to start of buffer and saving the match data.
(defmacro goto-min-save-match-data (body)
  `(progn 
    (goto-char (point-min))
     (save-match-data ,body)))

;;; Replace pattern with new in the current buffer
(defun replace-regex-current-buffer (pattern new)
  (goto-min-save-match-data
   (while (re-search-forward pattern nil t) nil t
          (replace-match new))))

Change some common tags over:

(defun replace-simple-md ()
  (interactive)
  ;; Change all the tabs to two spaces
  (replace-regex-current-buffer "\t" "  ")

  ;; Use 80 dashes for a line instead of 3
  (replace-regex-current-buffer "^---" (s-repeat 80 "-"))

  ;; Change lists
  (replace-regex-current-buffer "*" "+")

  ;; Convert image links from markdown syntax to org syntax
  (replace-regex-current-buffer "!\\[.+(\\(.+\\))" "[[file:../../\\1]]"))

My least favorite feature of markdown is the code syntax. In a markdown file, four spaces before a line means that it will be treated as a code block. This function captures the language tag in markdown code block, and wraps it in an org source block.

;; Replaces a block like this
;;
;;     :::python
;;     print("Begone markdown")
;;
;; With:
;;
;;     #+BEGIN_SRC python
;;     print("Begone markdown")
;;     #+END_SRC
(defun replace-code-section ()
  (goto-min-save-match-data
  (save-match-data
    (while (re-search-forward "^[[:blank:]]+:::\\(\\w+\\)" nil t) nil t
           (replace-match "#+BEGIN_SRC \\1")
           (re-search-forward "^[[:alnum:]]")
           (move-beginning-of-line nil)
           (newline)
           (insert "#+END_SRC")
           (newline)))))

Finally, deindent the old markdown code blocks by 4 spaces.

(defun de-indent-code ()
  (interactive)
  (goto-min-save-match-data
   (while (re-search-forward "^[[:space:]]\\{4\\}" nil t) nil t
          (replace-match "")
          (next-line))))

4. Site Structure

Next I decided on a site structure. Most org-mode publishing examples show the directory containing the HTML separate from the source files.

E.G. ~/projects/nicolasknoebber.com => ~/public_html

I wanted the HTML to be in the same directory as the org source. The trick is a src/ folder that mirrors the parent directory.

.
├── index.html
├── blog.html
├── style.css
├── includes.css
├── images
├── lambda/ <aws lambda handlers>
├── logo/  <svg logos>
├── posts
│   ├── <post_name>.html
│   ├── style.css
│   ├── js/
│   └── old/
├── scripts/ <shell scripts for interacting with aws>
└── src
    ├── site.el
    ├── index.org
    ├── blog.org
    └── posts 
	└── <post_name>.org

org-publish-project-alist is an association list variable that tells org how to publish the site. You can configure it with a list of components that make up the project. I split my website into two components: main, and posts.

(setq org-publish-project-alist
      `(("nicolasknoebber.com"
         :components ("main" "posts"))
        ("main"
         :publishing-directory "~/projects/nicolasknoebber.com"
         :base-directory "~/projects/nicolasknoebber.com/src")
         :publishing-function org-html-publish-to-html
        ("posts"
         :publishing-directory "~/projects/nicolasknoebber.com/posts"
         :base-directory "~/projects/nicolasknoebber.com/src/posts"
         :publishing-function org-html-publish-to-html)))

5. Include and Generate Content

Next I configured the publisher to include a header and footer in every page. I wanted these to look the same on all the pages. You can specify these by setting the html-postamble and html-preamble properties in the project association list.

You can also set the content that is inserted in the <head> tag. I use this to include a CSS stylesheet. I included a header with a few links for navigation.

For the footer I added links to Emacs/Org, their logos, their respective versions that were used, and the date the file was last exported. For the posts component, I also have a noscript tag for comments.

I utilize org's sitemap feature to automatically generate index.org which links to all the posts.

Here are a few snippets from site.el

(require 'ox)
(defconst html-main-head "<link rel=\"stylesheet\" type=\"text/css\" href=\"/style.css\" />")
(defconst html-posts-head "<link rel=\"stylesheet\" type=\"text/css\" href=\"/style.css\" />")

(defconst html-postamble
  (concat
   "   <span id=\"made-with\">
   &nbsp;&nbsp;generated with&nbsp;&nbsp;
   <a
     href=\"https://www.gnu.org/software/emacs\"
   ><img src=\"/logo/emacs.svg\" id=\"emacs-logo\" alt=\"Emacs\"></a>"
   "&nbsp;" emacs-version "&nbsp;"
   "<a href=\"https://orgmode.org\"
    ><img
         src=\"/logo/org-mode.svg\"
         id=\"org-mode-logo\" alt=\"Org\"></a>"
   org-version
   (format " on %s" (format-time-string "%m/%d/%y"))
   "</span>"))


(defconst html-posts-postamble
  (concat
   html-postamble
   "
<noscript>
  <div id=\"no-script-comment-message\">Enable scripts to see and post comments.</div>
</noscript>
<script type=\"text/javascript\" src=\"js/comments.js\"></script>"))

(defconst html-preamble "<a href=\"/\">Home</a>")
(defconst html-posts-preamble
  (concat html-preamble "
<a href=\"/posts/index.html\">Blog</a>
<a href=\"/posts/rss.xml\">
       <img id=\"rss-logo\" src=\"/logo/rss.png\"></a>
"))


(defun generate-posts-sitemap(title list)
  "Default site map, as a string.
TITLE is the title of the site map.  LIST is an internal
representation for the files to include, as returned by
`org-list-to-lisp'.  PROJECT is the current project.  This is
almost identical to the version in the org publish source code.
The only change I made is wrapping it in the .sitemap div."
  (concat
   "#+TITLE: " title
   "\n\n"
   "#+begin_sitemap\n"
   (org-list-to-org list)
   "\n#+end_sitemap"))

(defun format-sitemap-entry (entry _style project)
  "Format ENTRY in PROJECT.
Leaves the rss page out of the main sitemap list."
  (if (equal "rss.org" entry) ""
    (format "[[file:%s][%s]] =%s="
            entry
            (org-publish-find-title entry project)
            (format-time-string "%m/%d/%Y" (org-publish-find-date entry project)))))

(defun format-exported-timestamps(timestamp _backend _channel)
  "Remove <> from exported org TIMESTAMP."
  (print (replace-regexp-in-string "&[lg]t;" "" timestamp))
  (replace-regexp-in-string "&[lg]t;" "" timestamp)
)


(eval-after-load 'ox
  '(add-to-list
    'org-export-filter-timestamp-functions
    'format-exported-timestamps))

(setq org-publish-project-alist
      `(("nicolasknoebber.com"
         :components ("main" "posts"))
        ("main"
         :publishing-directory "~/projects/nicolasknoebber.com"
         :base-directory "~/projects/nicolasknoebber.com/src"
         :publishing-function org-html-publish-to-html
         :section-numbers nil
         :with-toc nil
         :html-head ,html-main-head
         :html-preamble ,html-preamble
         :html-postamble ,html-postamble
         :html-head-include-scripts nil
         :html-head-include-default-style nil
         )
        ("posts"
         :publishing-directory "~/projects/nicolasknoebber.com/posts"
         :base-directory "~/projects/nicolasknoebber.com/src/posts"
         :publishing-function org-html-publish-to-html
         :html-head ,html-posts-head
         :html-head-include-scripts nil
         :html-head-include-default-style nil
         :html-preamble ,html-posts-preamble
         :html-postamble ,html-posts-postamble
         :auto-sitemap t
         :sitemap-title "Blog"
         :sitemap-function generate-posts-sitemap
         :sitemap-format-entry format-sitemap-entry
         :sitemap-style list
         :sitemap-sort-files anti-chronologically
         :sitemap-filename "index.org"
         )))

6. RSS Feed

The final step was to add an RSS feed. This was the trickiest bit as its not a default feature of org-mode. I found ox-rss which is included in the org-contrib package. To load the library I added the following to my init.el:

(use-package org
  :ensure org-plus-contrib)

The problem was that ox-rss expects a single org file to convert into a XML feed. However, my blog is composed of many org files, so this wouldn't work out of the box.

I searched the web found and found this post which outlines a solution: hack the sitemap generator to automatically generate a single file from all the posts and then tell ox-rss to export that. I decided to create another sitemap instead of using the one I already had because the formats are different.

The RSS exporter expects entries formatted like this:

#+TITLE: Example Feed
* Example Post
:properties:
:rss_permalink: example.html
:pubdate: <2020-12-26 Sat>
:ID:       f0ccd140-db92-4af4-9759-831fdf69b447
:END:
* Another Post
:properties:
:rss_permalink: another-post.html
:pubdate: <2020-03-30 Mon>
:ID:       a42371e7-f67a-4445-b4df-000e76bdce86
:END:

To make the RSS sitemap match this format I provided the following functions:

(defun posts-rss-feed (title list)
  "Generate a sitemap of posts that is exported as a RSS feed.
TITLE is the title of the RSS feed.  LIST is an internal
representation for the files to include.  PROJECT is the current
project."
  (concat
   "#+TITLE: " title "\n\n"
          (org-list-to-subtree list)))


(defun format-posts-rss-feed-entry (entry _style project)
  "Format ENTRY for the posts RSS feed in PROJECT."
  (let* (
         (title (org-publish-find-title entry project))
         (link (concat (file-name-sans-extension entry) ".html"))
         (pubdate (format-time-string (car org-time-stamp-formats)
          (org-publish-find-date entry project))))
    (message pubdate)
    (format "%s
:properties:
:rss_permalink: %s
:pubdate: %s
:end:\n"
            title
            link
            pubdate)))

Finally I added a new project to my publish alist:

("posts-rss"
 :publishing-directory "~/projects/nicolasknoebber.com/posts"
 :base-directory "~/projects/nicolasknoebber.com/src/posts"
 :base-extension "org"
 :exclude "index.org"
 :publishing-function publish-posts-rss-feed
 :rss-extension "xml"
 :html-link-home "https://nicolasknoebber.com/posts/"
 :html-link-use-abs-url t
 :html-link-org-files-as-html t
 :auto-sitemap t
 :sitemap-function posts-rss-feed
 :sitemap-title "Nicolas Knoebber's Blog"
 :sitemap-filename "rss.org"
 :sitemap-style list
 :sitemap-sort-files anti-chronologically
 :sitemap-format-entry format-posts-rss-feed-entry)
)

Here's the sitemap generates: rss.org

Then I configured ox-rss to ignore everything but rss.org and export it as rss:

(defun publish-posts-rss-feed (plist filename dir)
  "Publish PLIST to RSS when FILENAME is rss.org.
DIR is the location of the output."
  (if (equal "rss.org" (file-name-nondirectory filename))
      (org-rss-publish-to-rss plist filename dir)))

Final result: https://nicolasknoebber.com/posts/rss.xml

7. Functions for Publishing

I created a key bind to load site.el and publish all of my org files.

Addition to init.el

(defun export-nicolasknoebber ()
  "Build nicolasknoebber.com."
  (interactive)
  (load-file "~/projects/nicolasknoebber.com/src/site.el")
  (org-publish "nicolasknoebber.com" t)) ;; Add t to force all files to republish.

(with-eval-after-load "org"
  (define-key org-mode-map (kbd "C-c c") 'publish-site))

Finally I added a function to publish and upload the current buffer to my site's s3 bucket.

(defun publish-nicolasknoebber-file ()
  "Exports current org file to html and uploads to s3://nicolasknoebber.com."
  (interactive)
  (org-publish-current-file)
  (let* (
         (org-file (buffer-file-name (buffer-base-buffer)))
         (publishing-dir (org-publish-property :publishing-directory
                                               (org-publish-get-project-from-filename org-file)))
         (html-file (replace-regexp-in-string "org$" "html" (buffer-name)))
         (html-file-path (concat publishing-dir "/" html-file))
         (site-path (replace-regexp-in-string ".+nicolasknoebber.com" "" html-file-path))
         (aws-s3-cmd
          (concat "aws s3 cp " html-file-path " s3://nicolasknoebber.com" site-path)))
    (eshell-command aws-s3-cmd)))

8. Results

This is a great improvement from my previous system: it's easy to publish, the writing environment is amazing, I have a RSS feed, and it's easy to setup on a new machine.

In markdown I would often accidentally create dead links and not know until I exported it. Now creating links is a streamlined process:

Type C-c C-l to call org-insert-link
Select file: which opens Helms fuzzy file finder
Find my file and name the link
Click the new link, which opens the contents in a new buffer

Emacs will even display images. Web links are also clickable and open in the default browser.

Another frustration I had was syntax in exported code blocks. I was using pygments, which did a nice job usually. However, it was a pain to change color schemes, and exported code always looked different than it did in my editor.

Now exported code looks the same as it does in Emacs:

Screenshot of Golang code in Emacs

Exported Golang code

// Returns a function that initializes dotfile storage.
// The result function must be ran at the time of a command being run so that
// the user can override default storage configuration with --storage-dir or --storage-name.
func getStorageClosure(home string, dir, name *string) func() (*file.Storage, error) {
        return func() (*file.Storage, error) {
                storage := &file.Storage{}

                if err := storage.Setup(home, *dir, *name); err != nil {
                        return nil, errors.Wrap(err, "failed to setup dotfile storage")
                }
                return storage, nil
        }
}

Overall I'm happy with the results and will most likely continue to use Emacs+Org as a static site generator.