How Drupal.org is built

>> NEIL DRUMM:  Yeah, I'm Neil Drumm.  I work for the Drupal Association on the Engineering Team.  I'm in Brooklyn.  And left my second and third displays at home.  So hopefully I'll get through this without speaker notes here.  

So I work on this site.  Drupal.org.  And I got involved with it some time around 2006 or 2007 as far as I can tell.  

Starting with working on API.Drupal.org as a volunteer.  

And at some point after that they were going on vacation and messaged me and asked if I wanted access to the one server I think it was actually not even a whole server.  And to cover anything while he was offline.  

And I've been working on Drupal.org in various roles and responsibilities since then.  

And started at the association more officially in 2010.  The Board of Directors at the time got a redesign together and tried having volunteers implement it but of course 10 companies volunteering 15 people didn't quite bring a Website together so a few of us went on contract to get that done and launched.  And yeah, since then, I've been contracting with the association and went full time eventually.  

And today it's not just me working on the site, it's my three co-workers, my boss, Tim on Drupal.org, Brendan who is a developer and he also the one that handles most of the email help at Drupal.org.  Mixologic, Ryan, he is the developer services engineer so stuff like Drupal CI and Composer and Composer Endpoint and I do a bit of anything, anywhere from I did a bit of themeing this week, CSS work.  Sometimes all go as deep as getting into our configuration management system and reconfiguring things.  And we get a lot of help from Ryan Newton and the rest of tag1 consulting.  They are our contractor for infrastructure services handling stuff like actually managing the configuration management and provisioning servers, VMs, making sure everything is -- all the operating system software services are up to date, secure.  And Michael Hess is super volunteer who mainly does like proof of concepts for stuff like Drupal Steward and some of the automatic update stuff.  So yeah altogether us and a few volunteers build this site.  But it's not just one site, it's really eight Drupal sites in addition to www.Drupal.org it's association.Drupal.org mainly that's for electing the association Board nowadays, API.Drupal.org, the API reference.  Events for Drupal cons.  

And groups.Drupal.org that's still an Drupal 6 so that site definitely could use some more maintenance.  Drupal jobs.  We have a job board for hiring Drupal talent and getting your resume out there.  

Localize for translating the Drupal software and security.Drupal.org which handles coordinating with people reporting security issues and us as the security people coordinating the containers, fixing those issues and doing a coordinated disclosure.  

Yeah, real quick, someone asked about single sign-on we are currently using a module called Bakery.  

It's probably -- I don't know if I would use the same system if we were starting from scratch today.  

But basically it allows Drupal.org to act as the www.Drupal.org to authenticate all of the sites with single sign-on.  

And as we move to Drupal 8, we'll be looking at other options or maybe we'll stick with Bakery and get it updated to Drupal 8.  I believe it's somehow about halfway ported, last I knew.  

And it's not just Websites that we do.  We provide services.  The localized.Drupal.org that provides downloadable translation files for sites.  And Drupal downloads actually downloading Drupal in modules, the downloads of Drupal has.  The Composer Endpoint that is the ultimate install Drupal with Composer and Drupal projects with Composer and we host that -- the package that's equivalent for Drupal.org ourselves.  So as a direct connection to all of the modules that exist and other projects.  

And the developer tools.  So stuff like Drupal CI and GitLab.  

And we've added -- we are adding a couple new services.  Automatic updates.  That actually is in production.  As of January or December.  I'll talk a little bit about that later.  

And Drupal Steward will be coming up I believe later this year to add a layer of protection for highly Drupal core security issues as they are announced.  So you don't necessarily have to have your site updated right away.  With a little bit less risk.  

And automatic updates.  That is something, a service, that I was deeply involved in for the Drupal.org side of things.  

First there is a Public Service Announcement API that I built.  

So you want to -- your site wants to know when it will be updating.  You don't necessarily want your site to do automatic update for our Drupal for all security releases.  

It's the highly critical ones that the security team knows will be exploited against a lot of sites pretty quickly.  Lots of other of course security issues.  You want to get your site updated quickly and efficiently.  But a lot can come with things like you know it's only for admins or only for authenticated users.  And will affect different sites in different ways.  

But the highly critical ones affect pretty much all sites that we want automatic updates to handle.  So this API will let sites know in a machine readable way that one of these is coming up.  And the main work of that, building this, was done by Michael Hess and I took that kind of proof of concept and reviewed it and revised it and deployed it and on our end it's basically a thresh command that let's you go into this feed and then will purge the file into our CDN.  

And there's also a -- so once the site knows that it might want to update, it has to know if it's in a good condition to be updated so that's the Drupal.org side of that is the checksums, release contents checksums API.  So this reverses automatic updates.  It works pretty much by replacing chunks of code as if they were downloaded as a new thing off of Drupal.org.  

So it's not patching or anything, it's literally replacing files and if you've patched your codebase, if you have tax to core on your site that's not a good spot to do automatic updates because it will get the site into an inconsistent state where some of your local changes might go away, some might stay, and it's -- it will just get your site to a position it probably won't work or runs a high risk of not working.  

So we have an API which has the checksum of every single file in every single release of Drupal and every single release of any Drupal module main distribution.  

So the client can check for if any of the files have been changed in any way.  

And at the top you'll see a few things untrusted comment and then some of the strings that start with RW.  That is package signing.  Or data signing.  

It is a -- since auto mec updates are a pretty high stakes API, we want to be able to verify that this data is not tampered with.  As it goes from our servers to your sites.  

So that stuff is based on BST signify it's a change signature format that David Strauss Peter Valanen and Mike Maten (phonetic) put together that allows us to have intermediate keys.  And basically we have a VM that is separated from the rest of our infrastructure that its only job is to hold onto that intermediate key and add that bit of data to the top of these files.  

And then Drupal sites can verify that with a static root key.  

And we don't have to have the private version of the root key on a live server in fact it's on a hardware security module a little USB stick type thing that four of us have.  And there's actually no way to get that off of the hardware security module in an unencrypted form so we can't even view it.  We just use that HSM to assign new intermediate keys and update those to the signing Oracle VM.  

And the way this first iteration of automatic updates works is it -- it is replacing files.  It's not working with Composer or anything like that.  So it's not compatible with Composer if your site is built that way.  

And it does -- our target here was the kind of lowest common denominator.  Sites that you put up somewhere and forget about.  And maybe they are on in underpallored shared hosting, stuff like that.  

So let's see.  Extracting a full copy of Drupal that might be -- that more stuff could go wrong, what this client is doing is just replacing the files that have changed.  So the file switches have been removed, if any.  

So that's -- what we produce is a zip file of the deletion manifest that's the list of files that no longer exist.  And then a full copy of any files that have changed.  And we'll generate this for any combination of releases for any pod.  So the claim can get into -- I believe that's done module updating now but the main goal is Drupal core updating.  But the future proofs are the Drupal.org side of it.  

And the whole file itself that gets assigned as a separate thing that the client downloads.  

And that client is a Drupal module.  And I was doing all the work on the server side along with the other volunteers.  And the client that was sponsored by the European Commission and Lucas heading did most of the implementation.  And it's out of beta.  It's a stable release now.  And again, it's for simple sites installed via this downloading files -- just downloading files and putting them on a server.  And what works for those are managed sites, if you have a deployment workflow already, that's even better.  If you have -- it's even better than using automatic updates if you have a CI pipeline and testing for your specific site, that's really where you like to be.  But this is here for sites which don't have that.  

And Composer support is coming in the next phase of work and Lucas has been tinkering with that and we're looking for sponsors if anyone would like that to go faster.  

One of the big projects I worked on a couple years back was Drupal.org's issue crediting system.  

That that -- we've been crediting contributors for a long time.  This is the earliest example I could find.  Them putting something in the commit message.  And the system really grew out of mentions peoples' names and commit comments.  

And what we were mentioning people in commit messages, it's great for code contributions but it doesn't tell the whole story.  We wanted to be able to credit organizations, too.  

And find out if people are spending their own time on things or if organizations were sponsoring their work.  

And find out which organizations were leading contributors.  

So internally this is entity references.  Projects maintainers can credit people on issues.  And people can attribute their effort to organizations.  

Each comment is a good place for the -- for you to attribute work as you go.  

So the comments happened to be references to the organizations.  

Since organization data is on comments, it made sense to make the underlying storage for the credit to be entity references to those comments.  And as a project maintainer when you took that check box to credit someone underneath it actually stores references to every comment by one of the credited people.  

So if someone switches the attribution halfway through an issue, everything gets aggregated together for the issue.  There's a couple more things in the field storage so it's not quite as simple as two entity references.  There's a check box to explicitly mark volunteer contributions and another entity referenced to fields to call out organizations to customers.  

They make some fun queries when doing reports this is just one of the ones I found quickly looking through my history.  It's one of the smaller queries for this data that I've done.  

Since this is issue credit rather than something in a git commit it's being used for crediting other important non-code work.  We didn't initially plan on exactly this use but it's helping fill a good gap since of course code contributions are not the only way of contributing.  

The rest of the organizing, communicating, documenting, designing deserves just as much recognition.  

And there's a page on Drupal.org which is not well advertised.  It has some of the graphs on here on that page are good.  It's Drupal.org/metrics.  And some of the graphs on there like number of users created per month, that's not actually all that useful because most of the users on the site are spammers or unused accounts.  And it's really just -- it's not really measuring anything.  

But commit -- getting into commit credit, how many people were credited per month, that is a more useful metric of someone doing something tangible to help the project.  

So that page has a graph of how many people were contributing credited per month broken up in a few different ways.  

You know, Drupal is a global project.  So we have it broken out by region.  And I believe last year or maybe the year before last we added a -- well, actually we added this a little before 2016 because that's when the graph starts.  We added the ability for people to self-identify as being a member of an underrepresented community.  Such as the race, minority in your field or location, or woman in a male-dominated industry.  

And actually I have it in my notes we added this in 2018.  That's why part of why the not specified is so big.  You can go to your user profile and update this and this graph updates once a day as people fill in their data.  

It's all self-reported.  And kept confidential except for as we get reports like this.  

And the spike in 2017, that is when the multilingual initiative completed.  They decided to do a single issue to stay kind of a party thanking everyone who had contributed throughout the years.  

And that was quite a few people.  

And the third way it's broken out right now is whether you're contributing explicitly as a volunteer or for an organization or a bit of both.  

And you know, we get to use -- we also get to use the commit credit data as the ranking in the marketplace.  So that used to be Drupal.org/marketplace.  That used to be alphabetical.  And now it's ranked by how much rough metric of how much organizations are giving back.  

But it's still not covering everything, all the types of contribution.  It's still pretty focused on code contribution, things into issues.  We'll see stuff like that example before of people making issue to credit, someone speaking at a conference or speaking at a meetup or attending a meeting.  

And we have the contribution recognition committee who is working on making recommendations for the next round of and future rounds of stuff that gets recognized on user profiles and organization profiles.  

And they are meeting now.  They have a survey out.  The best way to find it I think it's -- there's a link to it in the slides that are linked from my session on the MidCamp site.  You can also go to the association's blog.  And it's two or three blog posts down.  That's the blog post title, Drupal contribution culture.  And they have a survey looking for what people are doing for Drupal and what they would like to be recognized for.  

And what else are we working on?  So that's a couple of big examples of things that I've been working on in the past few years.  We have other projects in the pipeline right now.  

A lot of stuff motivated by the upcoming Drupal 9 release.  I think the beta release that's coming out later today.  And also for Drupal 9, improving Composer support in Drupal core and making the packaged downloads of Drupal Composer ready.  So in the past, if you tried finding Drupal Composer commands on regularly downloaded Drupal, it would not get you into a good spot.  But now with the recent versions of I believe it's Drupal 8.8 and later, you can download Drupal and then it works the same as if you would run Composer with an 8 project because that's exactly what it's doing on our servers.  

I am working on semantic versions for contrib for modules and themes.  So getting rid of the first bit, the 8.x and -- because Drupal projects, they can be compatible with Drupal 8 and 9 simultaneously.  It wouldn't make sense to have a 9.x in front of your version numbers because there's no such thing as a Drupal 9 module.  It's -- that minimum probably works with Drupal 8, 9 or later.  There's some things that truly is 9 only.  But there's no way to have projects be Drupal 8 projects or Drupal 9 projects now.  Thanks to Core's semantic versioning and the way they are handling the applications and adding new features every six months.  

And since we're changing how versions work we're going all the way to semantic versions so adding the third component, the .0 at the end.  So that's a lot of going through auditing the site, in some cases auditing other sites like localized.Drupal.  org or other places where version numbers are handled and making -- localized.Drupal.org and making sure those work with the new formating.  

And we're getting towards the end of that project.  It's in a place where it's ready to be enable when Core supports it.  It turns out core, since -- since it's been before we used semantic versions it doesn't really have great support for semantic versioning.  But that's in the more recently releases.  

And I'm moving on the parts of it that are more frontend like UI related.  We can change how the Project SEARCH works at any time.  You know before we need to get the data model correct.  

So the thing I'm literally working on right now is changing the core compatibility filter when you surge a module from it used to be just that prefix of the version number to using dependency data to figure out if it works with some version of Drupal 9.  If it works with some version of Drupal 8 or some version of Drupal 7.  

So that will be deployed hopefully later today.  Maybe on Monday.  

One of my co-workers, Brendan, is working on event listings.  

So taking some of the weight off of Drupalcal and groups.Drupal.org.  What Drupalcal is doing right now is when they see an event on groups.Drupal.org they manually figure out where it is, like where it actually is in the world.  There's no geocoding on groups.Drupal.org.  

So taking some weight off of the old Drupal 6 site groups.Drupal.org.  

And something else I'll be working on in the near future, probably in the next couple months, maybe six months, is issue forks.  

So we migrated all of our repositories to GitLab.  We use it for something interesting now.  

The unique thing about how we'll be doing issue forks on Drupal.org is they will be collaborative.  So if you fork a project on GitHub or GitLab.com, it will just be for your access.  And the way -- and you can add individual collaborators one by one.  

But the way a lot of Drupal issues work is there's multiple people who just show up and help out with an issue.  It's not something you would want to do through an invitation process.  We don't know who is going to work on what issue ahead of time.  

So we'll have issue forks that anyone can get access to contribute to.  And really the main work is going to be figuring out does everyone have access right away with their git account?  Or does -- do they have to click an additional button to get access to a specific issue?  Or do we want to allow force pushes?  Do we want to have branches that are person specific and then people can work from each other's branches.  Stuff like -- you know, if it's a free-for-all, git is not the easiest -- doesn't have the best UI.  It's easy to force push and not know that you're deleting something someone else worked on so getting the restrictions in there so that people can collaborate and there's some guide rails so people don't overwrite each other's work accidentally.  And hopefully not intentionally.  

And we have to update all of these sites to Drupal 8 or Drupal 9 at some point.  And we have the old site groups that's the Drupal 6 site.  

So we're working on technical debt cleanup right now.  Stuff like you know the events listing.  If we can get that to a good enough spot, that's one less content type on groups.Drupal.org and we won't have to migrate that.  

The association site, it's really for elections now.  But there are some old blog posts on there, which we can migrate and then we don't have to update the blog functionality on the association site to Drupal 8 or 9.  

Creating PHP, so you know we're not doing a PHP upgrade at the same time as a Drupal upgrade.  

So getting everything organized before actually updating.  

And for Drupal.org itself, the last time we did a migration, updating to Drupal 7, we actually turned the site off for 24 hours.  And yeah, actually stayed up probably for about 30 hours or probably more than that with all of the premigration and post migration cleanup.  

And we probably can't get away with that nowadays.  I think people will expect Drupal.org to be up 24/7.  So we'll have to figure out an incremental migration for www.Drupal.org.  

Let's see.  I'll get through the last few slides and then handle all of the questions.  

So supporting all these works -- all this work.  

You know, that's -- those are a few examples of stuff we do.  There's lots of day-to-day things.  This week the things I can remember doing are cleaning up the spam.  Helping with the security releases around CKEditor on Wednesday.  And actually did the security updates for our three sites that use CKEditor on Drupal.org.  Fixed a data lost bug and CSS issue.  

And a bunch of other kind of random stuff I can't remember.  

And you know, it doesn't all come free.  My co-workers and I are paid employees.  Of the association.  

And we do use various services which charge various amounts.  They are not all free.  Although a lot of these companies do give us generous discounts and some of these are Open Source projects there's no monetary cost but we do actually have to spend our time to actually use them.  

So we use a lot of third party services nowadays like Slack and Zoom for communicating around our team, Datadog for monitoring and logging, Zendesk if you have emailed [email protected].  

And we have actual servers.  There's 11 -- at least 11 physical servers at the Oregon State University lab.  We haven't bought new servers in a few years, just replacement parts that are needed.  Except for the media servers.  And we're moving the media servers.  Everything is just a host for virtual machines.  

So we're -- we get the most out of the hardware we have and can stay pretty flexible.  We have to move virtual machines around as needed.  

And we do have a contract with the Oregon State University Open Source Lab for their piece.  

And then the Engineering Team, we're only 4 people out of the 18 people on staff at the association.  My co-workers, they produce Drupal Cons and do something like coordinating Drupal brand and marketing with the Promote Drupal initiative and much more.  

So membership is the easiest way to support us financially, either as an individual or organization.  

And if you're an organization who has a bit more of a budget, supporting partners, they directly benefit Drupal.org.  They are the supporting -- their supporting partner of revenue is the earmarks for the Engineering Team and building Drupal.org.  And in addition to helping out benefits include increased visibility and boost in Drupal.org marketplace ranking.  

And especially this week, we're organizing a conversation in May and figuring how that's going to be affected and the whole industry is affected by the COVID-19 situation.  So these are some of the people we have -- who have reached out and given concrete support in the past few days around all of that.  And you know, we're seeing what we can do for Minneapolis and seeing if that can go on or if it needs to be moved.  And there's updates on the events Drupal.org Website for what we can do there.  

And that list will also be updated as people -- more people step up on the events site.  

And that is all I have for the regular presentation.  Come to the Contribution Day on Saturday, tomorrow.  There's information on the Website.  

And I should get Slack out of there.  I'll just do the Zoom chat.  I was initially thinking this would be on Slack.  

So first question is --

>> DWAYNE McDANIEL:  Actually before we get to that, Neil, I just want to do something really quick.  I want to unmute everybody at the same time, and let's all give Neil a round of applause like we're in the same room as him.  

(Applause).

>> Whoo!  

>> DWAYNE McDANIEL:  All right now I'll unmute you again Neil all right let's give you those questions we just wanted to make you feel -- say a big thank you.

>> NEIL DRUMM:  Thanks.  So how do we track the number of downloads of a module or sites reporting using a module?  

So right now we are not actually tracking the number of downloads.  That's something that broke a few months ago for -- because of some SSL issues, something like that, it just stopped working.  And we haven't gotten to fixing it yet partly because download count is not really a meaningful statistic.  There's a lot of CI systems and automated things that are downloading modules that it's not really a meaningful metric.  So I think in all likelihood we'll probably take downloads off of projects pages at some point.  But if you really look at them, they haven't updated since I believe November.  

Sites reporting using a module, that is a more interesting trick since it is -- when your site checks update status, it is -- we're counting that.  That's an individual vitamin going to Drupal.org.  And there's -- an individual site going to Drupal.org and there's a unique identifier each site generates so we can count how many of those we've seen each week.  

It's a little bit of -- there's some ways it overcounts and some ways it undercounts.  With my co-worker, Ryan, he has looked at the data in a lot of detail.  And there's stuff like sites that don't have cron running that I'll check in once a month and we don't count that because we are looking for sites that check in at least once a week.  So that might be an undercount.  And there's also sites which have update status turned on they don't want that running in production so that's an undercount.  In the future we're looking at doing a telemetry initiative to get even more data, stuff like what version of PHP are we running on?  What version of database are you running on?  All making it clear what your site is sharing and isn't sharing.  And that would give the core maintainers and module maintainers a lot more around like making decisions like do we need to support PHP 7.2 or maybe everyone is upgraded and we can stop supporting.  

And the actual way we count all of those numbers technically is most of those requests go to our CDN vastly.  And that handles like 99.5% of the requests.  They never hit our servers.  

Except that we have the logs from that, HTTP logs.  And we aggregate those through mock scripts and that's used for stats.  

Who selects which -- next question.  Who selects which modules are used on Drupal.org?  

That is mainly me.  And my co-workers.  

We're the ones who will have to look at maintaining something long term and making sure it's a good module that's secure, has a Drupal 8 or 9 equivalent or upgrade, to a new upgrade.  Yeah, it's us, we're running the site to get all of the modules.  

And Marco asks, will the pull request related feature be happening before the start of the Drupal 8 upgrade?  Yes.  So the where pull requests are right now is they work on staging.  And -- except for giving people request to push them out.  So it's probably already like a third done.  

I think the -- who gets access to what and any restrictions around who can push to what, that's going to be the hard thing to figure out.  

I think -- let me see if I can switch to a browser.  

The way to get involved and to see much of the progress of pull requests is git.Drupalcode.org/issue.  And that's where the issue forks will live.  And it is the shortest URL that has a link to this longer URL that has where the documentation will live for issue forks and merge requests.  And we have a couple metaissues around that.  And there's a lot of issues like to allow force pushes or deny force pushes and figuring out all of these access ideas.  

And yeah that all is pretty straightforward once we figure out -- implementing it will be straightforward once we decide on how everyone works together.  So that's something we can knock out before the Drupal 8 upgrade.  

And one last question.  

How can we help you to make less burden to maintain the sites?  Do you have some good tips to us to help your team?  Um, let's see.  

I mean, the big project on the horizon is the Drupal 8 or 9 upgrade.  

So looking at modules we use and making sure they have a good Drupal 8 or Drupal 9 version.  

Let's see.  The codebase for all the sites is on Bitbucket so it's just the cheap option at the time.  And we don't want to host the stuff to build our site on our site itself.  That would be circular dependency.  

But the way we actually deploy the sites and make changes to the sites is we have public repositories for each site with a thresh make file since this is Drupal 7.  

And yeah, you will see us doing a lot of work this week.  Just CKEditor upgrades.  And we do three or four deployments on an average day.  Well, probably between one and four deployments on an average day.  These are a lot of quick things that are a lot of deployments.  But other days it will be just one thing or one thing every couple of days.  

So that's the place to see what's getting deployed right now.  And the full listing of -- this is all the modules that are running on Drupal.org.  What versions of them if we have any patches.  So I mean that would be a concrete way to help remove -- remote technical -- to get patches reviewed into their parent projects.  

And did anyone else have questions.

>> DWAYNE McDANIEL:  There were a couple a little further up and I think the asker of that first one around SEO and if you could talk a little bit about SEO a little bit I think that would be interesting.

>> NEIL DRUMM:  Yeah I covered those two as we went.  Single sign on.

>> DWAYNE McDANIEL:  Oh, single sign on.  I'm sorry; I read that as SEO.

>> NEIL DRUMM:  No worries.

>> DWAYNE McDANIEL:  And then you had a slide that showed usage.  And there was a big Skype in 2017.

>> NEIL DRUMM:  Yeah that was the multilingual initiative completing.

>> DWAYNE McDANIEL:  There you go.  

>> NEIL DRUMM:  And yeah, SEO I don't personally pay attention to that very much nowadays.  We'll occasionally do a round of getting the social share previews like the screenshot that the image that shows up with sharing something, getting those in like a better condition.  But we're -- yeah, we're in a good spot SEO-wise.  I don't know if people even track page rank anymore but I know we probably have a pretty high page rank.  And if you're -- it's a site that's been around for a while, Google likes well enough, and yeah, it's master tools, Google webmaster tools sends you an email saying like there's -- click on this page, -- there's a fix on this page we'll take a quick look and fix it if it's actionable.