Screenr: Automated Workflow For Your Email - Stuff I Built
I tried Hey, the new e-mail service built by Basecamp. It is great, simple, smart. They’ve put a lot of thoughts behind re-inventing email. There are some pretty interesting features but fundamentally what is different is the opinionated workflow on which it operates.
The only mail that lends into the “Imbox” (how they name the important inbox) is from senders the user allowed. The first time they receive an email from someone, it ends in a staging zone, and then the user need to screen senders and decide if they want to receive emails from them, and where they want these email to lend: 1. in the imbox, 2. in a set of “feeds” (email that is casually read), 3. in the “paper trail” (receipts, …).
I liked the overall experience of Hey, and was about to subscribe. However there are two things that are a no-go for me. It doesn’t support custom domains for email addresses (yet)1, and there is no “archive” feature2. So I decided against that.
But I really liked the workflow, and I figured it shouldn’t be that big of a deal to re-create something similar in a plain-old mailbox.
the features
Here’s what I need:
- Senders need to be screened for their spam to be allowed in my inbox. I figured this can probably be done by redirecting mail from senders I never screened into a sub-folder and reviewing that occasionally.
- Screening senders need to be dead easy. That’s the beauty of Hey: screening is done by clicking on a thumbs-up or thumbs-down button. That’s why it works, no non-sense of creating or editing rules.
- Support the same kind of classifying structure. I already had a newsletter folder with a few manually created rules, and a reference folder in which I would occasionally put a mail that I wanted to retrieve easily - this is just automating that.
- Works from my computer or my phone, using existing apps. There is no point in starting a multi-month project and implement a working email UI, this needs to fit within my limited evenings and kids naps.
- Low maintenance, for the same reason.
the solution
So I built Screenr.
Screenr plugs to an email server3, monitors email in folders, and classifies email into a set of folders.
Incoming emails from unknown senders (i.e. with an email address that has not been seen yet) are moved to a “For screening” folder. Emails from screened senders are moved based on configuration: either leave in the inbox, or move it to a particular folder.
graph TD
A[Inbound mail] --> B(Inbox)
B --> C["Sender screened?"]
C -->|No| D("Move to 'For Screening'")
C -->|Yes| E(Move to configured folder)
To configure the tool, I just need to screen emails in the “For screening” folder occasionally, and move them based on where I want these to land: move emails from senders I still want in my inbox to a Screened folder (which then gets moved back into the inbox once Screenr registered the decision). Move emails I want to reject to a Rejected folder. Move emails I barely care about, such as newsletters, promotions, etc. into a newsletter folder. Move receipts, booking confirmations, order tracking info, tax documents, etc. into a Reference folder. If I move a single email between two monitored folder, then Screenr will move all of the mail from that sender into the new folder.
graph TD
A[For screening] --> C[Decide what to do with this sender]
C -->|Allow in inbox| D("Move to 'Screened'")
C -->|Ok but classify elsewhere| E(Move to the folder you want)
C -->|Don't want mail from them| F(Move to 'Rejected')
Screenr supports two configurations (at the folder level) to classify email: either move the email directly to the target folder, or through an intermediary “classification” folder (like the “Screened” folder for email for inbox). The reason for that is that you might want to occasionally classify email without creating a rule: e.g. if my spouse forwards me a booking confirmation it’s nice to file it into the Reference folder without automatically classifying everything she sends me as reference4.
Bonus points: now I default my behavior for processed email based on the folder I’m looking at. I archive email in my inbox - the reduced amount of email there makes inbox 0 much easier to attain - and delete email in the “newsletter” folder. Less thinking, more better.
how I use it
There are 4 folders I’m watching:
- Inbox is the email that I (probably) care about. Once I read it, I archive it.
- Newsletter is the stuff that I’m occasionally browsing, but it’s mostly composed of automatically sent emails and newsletters and there’s likely no required action. Once I read it, I (most likely) delete it.
- Papertrail receives all my order confirmation, plane tickets, vouchers, etc. It usually just stays there. I used to call it Reference but I like Hey’s name better. Some of the mail I manually move there, mostly when my parents send me a scan of a bank record, or my wife sends me some reservation, because I don’t want to classify them there.
- For screening is all the mail that I’m receiving from senders I haven’t reviewed yet.
When something arrives in For screening, first I get excited because that’s proof that Screenr still does its job. Then I decide what to do with it by moving it in one of the folders:
- Screened is for emails I want in my inbox, like people I know, notifications I care about, etc. Screener registers my decision then move it (and potentially other emails by that sender) back into the inbox
- File in papertrail is the same as screened, but for my papertrail folder.
- Rejected is for what’s not exactly a spam, but I don’t actually want. Examples are emails from my car dealership, which is mostly spam, but which also sends me service confirmation once in a while, which I then go fishing in that folder and redirect to reference. Or companies who want me to wish them happy birthday:
at work
I’m using an additional folder called notifications for all the spam generated by tools, that I either quickly browse or simply delete when no time:
the technology
I wanted reasonable portability, my email provider recently released a bare-bones nodejs hosting and I figured I could give it a shot, so I settled for typescript. That host was hardly controllable, so I ended giving up and re-hosting onto the Kubernetes cluster from which you retrieved this static HTML page in a vastly over-complicated setup instead (more on this in this blog post).
The application itself is decomposed into 4 layers following the onion architecture:
domain
, contains the actual classification logiccontracts
, contains interfaces and data exchange structures for the dependencies that the domain will need an implementation for, such asIMailbox
which allows retrieving emails or moving them around,Mails
which are those mails, orIScreeningStorage
which provides interfaces to store screening behavior or retrieve it based on sender.infrastructure
, which are the particular implementations for those dependencies: anIMAPMailbox
implementingIMailbox
which provides said features for an IMAP server.application
which binds all the stuff together: loads configuration, instantiate infrastructure, inject in domain, then kick off the process.
For the storage of screening results, I settled on a file-based system which stores everything in memory and serializes it back to a file in storage when new rules are detected.
Building the IMAP connector was the most painful. The client libraries in the node ecosystem are not super well maintained and documented5, I tried two, and diagnosing some issues I was having6 was a straight pain that almost made me abandon. In retrospect I would probably pick .Net Core instead, given that the IMAP libraries are much more solid. If I want to make evolutions on that, I don’t exclude a rewrite ; the languages are sufficiently close that this wouldn’t be a big thing to do.
As mentioned, the hosting is done on a k8s cluster. A Helm chart specifies a single deployment with no service, and mounting a file share inside the container to store and retrieve the settings. Here is the extent of the infrastructure:
graph TD
A[Pod] --> C[File share]
where to get it
To run Screenr, you need a computer that can run node ; ideally a server that you don’t turn off.
- Using the package on npm: install node.js, then Screenr using command
npm install -g screenr
. Download this config file asconfig.json
and set it up following this instructions on npm. Run commandscreenr
in the same directory as the config file. - Source is on GitHub and the Readme explains how to configure it. Clone the repo locally, then run
npm install
andnpm run build
. To start it, go withnpm run start
. - Docker image is on DockerHub (or
docker pull cfe84/screenr
). Kubernetes Helm Chart are available in the git repo.
where I left it
There are some optimizations that could be done. For starters, this could use IMAP notifications to actually listen for new mail, rather than reconnecting and checking every 20s. I went for the straight forward thing right now, not sure if it’ll be a problem. Also this could be extended to support more rules, but it might require some more UI work which I don’t want to do, so that’s it for now.
notes
-
like sending mail from at feval.ca for example. They announced it for end of 2020, and it seems that it will be supported at an additional cost. I don’t want my digital identity to be coupled with an email provider, so I can’t use it till then. ↩
-
this one is by design: the manifesto specifies that they don’t believe in “Inbox Zero” and won’t implement it. It’s a matter of taste I guess, but I’ve tried for a couple of weeks and I just can’t. ↩
-
using IMAP, although it could be extended to support whatever email provider you want it to ↩
-
although that would settle certain debates on what should be considered as reference within my couple. ↩
-
but still Kudos for creating them in the first place, I suspect this is not an easy task. ↩
-
some mails didn’t want to move and made the remote server close the connection. It might still happen, i’m still not sure what’s happening. It goes away after a while. ↩