And now for something completely different

Druid is known for being a Drupal-house. And we are. Most of the projects for us are still in the category of Drupal where a CMS is needed. But Drupal is not always the right tool for the job. Like Dries Buytaert stated in DrupalCon Vienna, “Drupal is no longer for simple sites”, and generally not for all use cases. That said, Drupal is still a very good choice for the right jobs. 

We have recently had some non-Drupal projects, and now I’m going to tell about one of those. Non-Drupal meaning, however, that the tech stack wasn’t completely new to us, as we are already very familiar with the programming languages themselves (PHP and JavaScript).

The need – say no more, say no more

Our customer had a need which was not totally structured at the moment as they targeted a market which is under a major disruption. The digital landscape for this particular market is evolving, and as is often the case with evolution, you need to adapt or perish. What we already knew from the start was that the application needed to be mobile-friendly and its main function should be to facilitate communication between users. Create a backlog from that!

Mobile? So we thought PWA is the way to go. At Druid we have been fans of PWA for a while, and we have written about this exciting concept before. Progressive Web Applications are the future and show much promise. PWA is also a very good choice when you want to distribute your apps without App Store or Play Store. For example, some internal tools could be delivered from intranets. I’ll tell more about PWA later. Anyway, we thought that PWA would suit the need very well.

Communication? Wink wink nudge nudge. Say no more, say no more. This raised ideas for data structuring and for the UI.

Tech stack – And Now for Something Completely Different

We decided to go bold and try totally new components for this project: frontend, backend, database and infrastructure. Of course, there was knowledge and learning behind these things with some PoCs and such. And like I said, we were familiar with the tech. Or at least someone on the team was.

If we start from the bottom, we chose Docker for being the glue between components and between environments. We quickly drafted an open source version of this Docker thingie and released it as a separate thing. Stonehenge is a multi-project local development environment and toolset on Docker. Now the project uses Stonehenge as a developer’s tool to run it. As the local development environment is quite a common problem for developers, we thought this could be beneficial for others too. So that’s why we extracted this functionality from the project and released it as a separate tool.

Basically, Stonehenge provides us with local URLs and a proxy to handle the traffic to our projects. Proxy is made with Traefik which is just a breeze of fresh air compared to any previous tech used for the same purpose. I can say it works and performs very well on production too!

The project itself defines the services for our application. The basic stuff like Nginx, PHP, database and CLI and their relationships. We use Docker Compose for this.

For the application itself, we chose Symfony 4 for the backend and React for the frontend. Basically, Symfony creates a standard JSON API which the React application then uses. One reason we chose Symfony was the support for our database (what could it be?) via Doctrine. When evaluating different backend (PHP) frameworks, we studied the experiences of other developers, and there seemed to be nothing but praise for Symfony 4. And it really was a pleasure I can say. Some of us already had experience with Symfony as Drupal 8 is built on top of Symfony 3. Still, there were many new things for us to learn.

The database. MySQL, MariaDB, PostgreSQL or something else? We ended up choosing MongoDB to complete our jump into the unknown. MongoDB is a so-called NoSQL database where data is stored as objects instead of rows. This is very useful with data that can be very different by structure (read: document might have data which the other documents of the same type do not have). Also, the schema does not need to be defined beforehand (and updated) but it just lives by how you use your document entities.

React is something which I personally cannot write very much about, however my colleague Kristian has written a small recap of its use in our project below. Beware though: the Dependency Hell in Javascript world and the vast amount of same-but-different kind of tools available might sometimes make a developer’s life not-so-easy.

Quote from Kristian:

“It was pretty great to work with React on a large-scale project. In general, it was surprisingly easy. We did make some mistakes early on, which ended up forcing us to refactor. The original plan was to render the majority of the application with Twig, and only the communication aspect would be controlled by React. Eventually, the majority of the application was ported to our React app.

This means that we initially didn’t implement any sort of routing system and we didn’t think enough about the architecture of our store. Luckily we were able to refactor with relative ease once these problems presented themselves. 

Probably the most fun thing during this application for me was working with MobX. This is something I’ve wanted to do for a long time, and I’m glad I got a chance to finally use it in a commercial application. Essentially MobX is a state management tool built on the observer pattern. All observers who are watching an observable variable will magically update whenever the observable value changes. If there is one thing I’d do differently next time, I’d probably use MobX State Tree, which is a more opinionated version of MobX with some Redux-like behavior, without the overhead of Redux.”

Well, there you have it.

As we jumped on the PWA bandwagon, React helped us there to basically create a single page app which helps on some aspects on PWA. What does a PWA do, one might ask? You can think of it as an app which is basically a webpage. That means no App Store or Play Store for distribution. The app is updated when the web application is updated. There are a few distinct features to make it an “app like” experience: caching of assets for speed, standalone mode (to make your own UI without browser components), access to some mobile APIs and offline capabilities. There are also push notifications which currently work only on Android. Check the 2018 State of Progressive Web Apps for more info.

Progressive Web Applications is funny enough something Apple envisioned already in 2007 when they announced/released the original iPhone. Currently they are very strongly driven by Google. So this means basically that PWA works better on Android phones at the moment.

Fortune favors the bold, and I strongly believe the PWA is the future of apps and especially useful when creating tools for organizations and business.

Well, that was quite a long rant about technical stuff. Let’s take a breath for a moment.

“And now a film about a man with a tape recorder up his brother’s nose”

GDPR aspect

Yeah, the infamous GDPR. At this point we don’t have actual user data, so we’re nicely covered. But we designed the whole development on dummy data which started to be very fruitful during the project. We programmatically added data fixtures, which filled our database with dummy data. So in that sense, we’re totally GDPR-compliant when we do development. We don’t need production data.

This gave us some unexpected benefits too. In testing for example, we quickly noticed that we can use this generated data within our tests. How cool is that! Well, it is quite cool I can say.

“In this picture, there are 47 people. None of them can be seen.”

Next steps

The project is now in MVP state and moves to “field-testing” phase. Hopefully, end-users will like what we have done. The testing will be done with small groups and in a controlled way. This means we use a certain set of generated data suited for that test group. Also, we control the PWA aspect of the MVP. At the moment it works best on Android, so that will be the platform used in testing in testing.

“Now, what’s to be done? Tell me sir, have you confused your cat recently?”

About the coconut

We have learned a lot! This means that our learnings are already influencing our new projects and somewhat of how we handle older projects. We’re about to copy the backend stuff for a new API-only project and the Docker stuff has evolved into a very usable state. And releasing an open source project (Stonehenge) was a definite plus!

The most exciting part at least for me has been to share all this with more and more people inside Druid so that all the learnings can be put into practice on a wider scale.

“Wait a minute — supposing two swallows carried it together?”

Marko Korhonen
CTO, The Ministry of Silly Walks

Working with Progressive Web Apps

At Druid we are always keeping our eyes open for new technologies, which can benefit both us and our clients. Recently we have been experimenting with Progressive Web Apps. 

TL;DR

• Progressive Web Apps provide handy app-like functionality for web applications
• The two major requirements are the manifest.json and a service worker
• PWAs can be the answer to user’s unwillingness to install apps on their phone

So without further ado, let’s delve into the world of manifests and service workers…

What are Progressive Web Apps?

PWA is a term for web apps that share certain features with mobile applications (i. e. phone apps).  Essentially a PWA is a web app with several modern web capabilities designed to give users a browsing experience similar to using a mobile app.

Before we get started, it should be mentioned that some of these features will not work on all phones. As of writing this blog post, iOS does not yet fully support PWAs.

The requirements

In order to be considered by browsers to be a Progressive Web App, your app must meet a list of requirements.

The most important points are these: 

• Site is served over HTTPS
• Pages are responsive and mobile-friendly
• Site must work offline
• A manifest must be provided
• A service worker must be registered
• Site must load fast enough for 3G

Now, most of these features that you would include anyway if you were building a web app. However, there are two important aspects, which make PWAs different than normal web apps, which I will give a brief introduction to here:

The Manifest

A manifest is a simple file which essentially tells browsers that this is a PWA. The specification can be read in details here: https://w3c.github.io/manifest/

It’s possible to define a lot of functionality in the manifest, such as a visual theme, orientation (landscape or portrait), basic configuring of colours, etc. The only mandatory fields of your manifest are name and short_name. These describe the name of your app. 

The manifest must be referred from your HTML head, like so

<link rel=“manifest” href=“/manifest.json”>

Service Worker

This is kind of the body of your PWA. It is able to process all requests to and from your site from the client’s browser. Every time the user makes a request to your site, the ‘fetch’ event is triggered and you can handle that request however you wish. 

A service worker runs in your browser (it does not have access to the DOM). The service worker is a “special” JavaScript file, which runs in a different scope than regular frontend JS. The browser can execute this script without the page being open. It can even be executed while the browser is closed. This is extremely handy for several app-like features, such as push notifications. In this article we will only scratch the basics, so we will not really be using much of this advanced functionality. 

Unlike frontend JavaScript, the service worker needs to be registered to the browser navigator.

I’ve included the script app.js in my HTML and from there the following code is executed:

if ('serviceWorker' in navigator) {
  navigator.serviceWorker
    .register('/sw.js')
    .then((registration) => {
      console.log('Service worker registered', registration);
    })
    .catch((error) => {
      console.error('Something went wrong with registering service worker', error);
    });
}

Then, in the ServiceWorker, you can add eventListeners.The most important events are install and fetch. 

install

This event runs the first time the user visits your site. Since it will only run once, it is expected that page load might take slightly longer than normal during this request.

const CACHE_NAME = 'static_cache';

const urlsToCache = [
  '.',
  '/js/script.js',
  '/css/style.css',
  '/images/myCat.png',
  '/images/myDog.png'
];

self.addEventListener('install', (event) => {
  event.waitUntil(
    caches.open(CACHE_NAME)
      .then(cache => cache.addAll(urlsToCache))
  );
});

The previous code defined the name of the cache we will be using and an array of URLs that should be cached immediately. Your cache will be created if it does not exist. You can add any internal URL here. Just keep in mind that the ServiceWorker has a certain scope. The scope is equivalent to its URL. So for example in the previous code, the service worker is located at /sw.js, which means that its scope spans the entire site. If the ServiceWorker was located at /swdir/sw.js, It would only be able to handle requests within the /swdir/ url.

fetch

Now comes the exciting part – fetch. As I mentioned earlier, this event is triggered every time (except the first) a user makes a request to your site. 

self.addEventListener('fetch', (event) => {
  console.log('Fetching data for', event.request.url);
  event.respondWith(
    caches.match(event.request).then((response) => {
      if (response) {
        console.log('Returning ' + event.request.url + ' from cache');
        return response;
      } else {
        console.log('Fetching ' + event.request.url + ' from network');
        return fetch(event.request);
        // TODO Add fetched file to cache
      }
    }).catch((error) => {
      // TODO Handle error
    });
  );
});

Remember, fetch is triggered every time a request is made for an individual file on your server. So for example, if the user visits index.html, it’s possible that a fetch event will be triggered individually for the urls index.html, /js/script.js, /css/style.css and /images/myCat.png. So in that case it will run 4 times. 

The previous code is a simple way of serving offline content. For each request, the ServiceWorker checks if the file already exists in cache. If it does, it is served to the user. Otherwise, the file is fetched from the server.

With this simple code, it is possible to have an offline PWA up and running. 

Some advice

While the concept of PWA isn’t extremely daunting or anything, there are definitely certain aspects that can be confusing and frustrating. Listed below are some of the more useful tools I used when learning the basics of PWA:

Lighthouse

Lighthouse is a built-in tool for Google Chrome designed specifically for testing Progressive Web Apps. Just go to your page, open Chrome DevTools and press the tab “Audits”. Lastly, press the blue button called “Perform an audit…”. Chrome will then run all the tests and give you the results in an easy-to-digest list. In case of failed audits, the reason for failure should be pretty self-explanatory. If you’re like me and like the shotgun approach, just keep modifying your code and re-run Lighthouse until it’s all green. Do keep in mind that your site needs to be served in HTTPS, so in case you’re testing on localhost, this audit will invariably fail. 

Google Developers

Google has some very good and up-to-date resources on PWA. One nice thing about their docs is that it will automatically warn you if the article is old and therefore more likely to be deprecated. 

Conclusion

Progressive Web Apps are a new exciting technology and it is definitely worth it to invest some resources into learning this technology. In an age where users no longer want to install apps on their mobile devices, PWAs serve as a nice middleground between the modern web applications and more traditional mobile applications. 

Design process matters – here are three reasons why

Have you ever paid attention to how your web service is being designed? How do the design and the technical implementation mesh together? I can promise that it’s not a given that they do. In this industry it is surprisingly common to see clumsy and unclear design conventions in use to the benefit of neither the client nor the service provider.

A good design process helps you create a consistent user experience throughout the web service, and tackles many challenges that project management faces. It also saves time and money. That’s why we here at Druid have made our design process a priority in the last couple of years. 

“What should a design process be like?” you ask. Agile, efficient, and optimized – a process that can achieve the following three powerful advantages:

1. Scalability

A good design process makes it possible to create scalable design work. What scalability basically means is that instead of designing separate web pages, you design the components that web pages are built with.

What’s the use of scalability? For one, the user experience of your web design will be consistent when the same components repeat throughout the service. A seamless and effortless user experience is a crucial factor in the digital world when you try to turn users into customers. Secondly, scalable design saves time and energy (and therefore costs), because you don’t have to design everything separately from scratch. This means more results with less trouble.

What about when you want to update the styles of your site? That is where a style guide comes into play. Style guides allow you to make site-wide style updates and to easily modify the styles of your components. And since all the changes are updated into all of the components at the same time, the consistency of the site’s styles can easily be maintained.

2. Cost-effectiveness

A good design process ensures that your service is being built cost-effectively. Just the scalability of the design work alone creates considerable savings. Another aspect that factors in on the costs is agility of the design work and its synchronization with the technical implementation of the project. 

When the design and the implementation of a project move ahead concurrently in cycles, possible issues and solutions can be considered ahead of time and from several points of view. When using the right tools, design solutions can be tested and iterated at an early stage. This way you can avoid many potentially burdensome and laborious modifications at a later stage.

If, however, you’re using a straight production pipeline where the design and implementation follow each other chronologically, you can easily find yourself in a situation, where you’ve used too heavy a toolset prematurely and spent money designing something “too” fancy before the client’s actual needs are even known and understood.

We tend to create lightweight prototypes of the components we design before creating them for the actual system. This way you, as our client, get to give us your input at an early stage  – before modifications and changes are laborious and expensive.

3. Better basis for success

A good design process provides excellent tools for managing the expectations of the customer and helps the customer be a part of the process from the get-go.

When the design process is on point and clearly communicated, it helps the customer to better understand the roles of both parties and the division of work between them. The client will also know when to give input and what kind of input is expected from them. When the client is an integral part of the team, unfortunate surprises can be avoided by both parties, and it becomes more transparent how fast things can realistically be achieved. 

At Druid, every project starts with a design kickoff. During the kickoff we and the client go through how we carry out our projects. We agree the goals and the metrics with which we measure success. And as is fitting with the agile methodology, the goals and metrics may refocus or reform as the project progresses; the further the project develops, the more we learn and the wiser we become. 

As you can see, flexible and clearly defined design process can provide considerable advantages. How do we handle this in practice? Contact us and we’ll tell you more!
 

Making super fast virtual machines with passthrough

Reader beware: this text is highly technical! It’s meant for fellow developers and tech enthusiasts. We’ll take a look at using virtualization as a no-compromise replacement for dual booting between operating systems, emphasis on the word no-compromise. Many tasks are quite feasible even with a basic VM, for example testing websites in a legacy browser not available on current operating systems. For more demanding tasks, booting to another natively running OS and then back just for one specific app or a gaming break is cumbersome. So what can be done?

Getting up to speed

Emulating a CPU, graphics and I/O has a heavy performance cost. We can get away with it when running old applications on modern hardware (think of DOSBox and video game console emulators). We need something faster than emulation for a snappy VM running new operating systems and apps. To do this, the virtual machine monitor has to bring the guest system closer to bare metal. Hardware assisted virtualization has been on server and consumer CPUs for years (as Intel VT-x and AMD-V). It allows executing guest instructions on the real CPU with far less overhead than in emulation. It is crucial for running x86 virtual machines at nearly native performance. It’s also widely supported nowadays.

On the I/O side of things, various paravirtualization methods have been used to boost performance compared to emulation. What this means is the virtual machine manager exposes special devices that allow somewhat direct access to actual host hardware through an API. This provides faster disk, networking and timing support in virtualized environments. Even limited hardware accelerated graphics support exists in various virtual machine managers, where OpenGL and Direct3D up to version 9 via an API translation similar to Wine are supported. Paravirtualized devices often need specific device drivers that have to be installed on the guest OS.

Another step further is to provide a guest system with dedicated, exclusive access to actual devices on the host machine. We call this method PCI passthrough. Since the guest has direct access to the device, it is controlled by the same device drivers and has the potential to provide the same performance and device specific functionality as on bare metal. For instance, TRIM commands can be sent directly to an SSD connected to a passed through disk controller. Dedicated storage and networking hardware can be useful for demanding server use cases where the best performance is needed without giving up the benefits of virtualization. On the desktop, an interesting use case is dedicating a graphics card (VGA passthrough) to a guest OS and running graphics-intensive applications with high performance.

The software side

Support for PCI passthrough exists in various virtualization software. However for VGA passthrough specifically the common and well documented approach is to run a VM using the Linux kernel’s KVM as its hypervisor, QEMU as the userspace emulator and OVMF as its UEFI component. Trying out QEMU is relatively straightforward. Virtual machines can be fired up from the command line and all the needed configuration options are given as arguments. Then, host devices can be given to a VM using a helper driver called vfio-pci.

If all goes well, you’ll have a VM with direct access to the piece of hardware, with minimal overhead. Pretty much any PCI-E device can be passed through, with caveats (we’ll get back to these in a moment). Many motherboards have their SATA and USB ports spread out on more than one controller. Then, one of them could be dedicated to a VM. My own VM setup has its own graphics card, USB controller (mouse and keyboard can be switched between the host and guest with a switch), an add-on SATA controller and the onboard audio passed through. After some tinkering, optimization and figuring out what works best, I’ve given up on dual booting because the VM is simply free of compromises.

The fine print

Let’s look at the caveats, then. Doing this needs obviously appropriate hardware. Most importantly both the CPU and the motherboard need IOMMU virtualization support (Intel VT-d and AMD-Vi). Luckily, they have been available in many if not most consumer platforms for years. However, a working implementation on a motherboard is not a given even if VT-d or AMD-Vi support is advertised. Though in many cases fixes have been provided via BIOS/UEFI updates, virtualization features are hardly a priority on mainstream consumer hardware.

Another common issue is something known is IOMMU grouping which deals with the separation between devices. To put it simply, devices cannot be passed through if other devices belong to the same group, unless you pass all of them. Otherwise they could interfere with each other and nasty things could happen. How your onboard devices and add-on card slots are grouped depends on the motherboard and the chipset it’s based on.

Finally, the PCI-E devices themselves can have firmware bugs that cause them to behave badly when passed to a VM. Some hardware vendors are also known to implement VM detection in their (consumer hardware) drivers and prevent them from working if they are running inside one. With server grade, and to some extent enthusiast consumer hardware you might avoid these issues. Still, it doesn’t hurt to do some research on your prospective components before building a setup like this.

All in all, we have just scratched the surface on this matter and this is only intended to provide an introduction. If you’re interested in this kind of thing, I recommend checking out the links below for more details. You might also ask, what’s the point? Considering the amount of time spent tinkering and possibly the cost of additional extension cards for VMs, one could just buy a whole extra machine and be done with it. But where’s the challenge and fun in that? 😉

Passthrough and virtualization in general will quite probably remain in the niche for regular desktop/laptop users but time will tell what happens in the business and server world.

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

https://vfio.blogspot.com/

A static content layout lifesaver: Paragraphs module

As Drupal website users, we all know how boring static basic pages and detail pages are. As Drupal developers, we all know how frustrating those boring basic pages and detail pages become, when someone (indeed, the content editor) tries to change the layout by messing with the source code of the WYSIWYG.

Frustrate no more. The Paragraphs module is here to save the day (and your content layout)!

What is it, this Paragraphs module?

Imagine you’re a content editor and you want to create an article with some text, a few images, maybe a caption underneath the image. You also would like to have a quote, which stands out with a full width background image. How would you do that with the WYSIWYG? You try to modify the HTML in the source code and hope it turns out exactly like you had in mind. 

Of course, that’s wishful thinking. But there is a solution. One that doesn’t require a big WYSIWYG field, but a solution that lets the content editor create every section of the article separately.

Curious how it works? Let’s dive right into it, shall we.

** For examples I am referring to a paragraph demo site I recently created for Druid. **

The Paragraphs module simply creates a new field type, Paragraph, which you can use in your content type. And for each type, you can add field types. Sounds familiar? Well… it kind of works the same way as a content type.

Now you’re probably thinking, “haha, what’s so life-saving about this” and you have a point. The field type is nothing life-saving at all, but the paragraph types behind it are.

Paragraph types

A paragraph type consists of fields, and these fields are the same types as those of a content type (you even get an extra reference option to another paragraph).

After you’ve created your paragraph types, you can add them to your content type – and this is the nice thing about it. Basically, you create a field with a reference to paragraph types. You can define no paragraph types, which will let you use them all, or you can define only those you would like to use in a specific content type. When you create a new node you can use as many paragraphs as desired, and you can mix them as much as you like. There’s also a handy little drag and drop so you can change the order of the paragraphs really quickly and easily.

Each paragraph type has a matching template file, where you can add the needed HTML markup. This allows you to theme per paragraph and that way, you don’t have to worry about the layout of your node. It doesn’t matter if you add a certain paragraph type to the top of the content or the bottom; if the theming has been done properly, it will look good.
 

Looking pretty good, right? Try that with one huge WYSIWYG body field. Yup, didn’t think so!

Why you should only use it for static content 

The purpose of this module is to create nicer and cleaner content. As in content created by the client. Overview pages, feeds, or any other item that needs dynamically generated content, should probably be handled by something that is able to do this automatically, like views.

If you want to give the content editor a bit more freedom layout-wise, but at the same time you don’t want to worry about your pretty layout getting messed up, the Paragraphs module is exactly what you need. Dying to try it out? You can find the module here. I truly hope you enjoy using it as much as I do!