Automate Drupal 9 upgrades: Addressing deprecated code as a community

>> Okay. Yeah. Thanks, everyone for joining. So this is automate Drupal 9 upgrades, addressing deprecated code as a community at Midcamp 2020 where we're all doing this digitally. If you want, the slides are on the Midcamp page right now. They have links to a bunch of stuff here.

I might post some links in the chat as we go along. I'm Dan Montgomery. I work at Palantir.net. It is a digital consultancy that provides web strategy and design services to clients that include higher education, health care, and government. So this talk is ‑‑ we're going to kind of cover a lot of what is going to be focused on Drupal rector. We will do context editing, updates from Drupal 8 to 9, what that process typically involves. Michael, who is on the call here had a session just before this, and he went into more detail around that. So that's a good one to look at as well. If you want to find more information, and then we're going to go to the Drupal rector tool, talk about what it is, how it works, how you can develop with it, and kind of some resources that if you want to start development, you know, what are the first stages that you need to kind of do that? And then we'll finish with our goals for that project and some kind of broader community goals. 

So context. Drupal 8 code will mostly continue to work in Drupal 9. This is different from previous upgrades where people have gone from 6 to 7 or 7 to 8, and there's been a lot of changes, particularly 7 to 8. And in Drupal 9, most of this stuff will continue to work as your Drupal 8. 

The deprecated core functionality is the key piece you might need to pay attention to. And that's probably going to necessitate updates to custom code or contributed code. There is quite a number of deprecations over the life of Drupal 8, but we can cover about two‑thirds of those by addressing custom and contributed code. This is a little bit out of day. This is that list of that top 15. And even amongst those top 15, the biggest one by far covers quite a lot of that. It looks like, I don't know. I'm just guessing, but it looks like 40% or something of those deprecations are here. There is more ‑‑ this graph is updated on acquia's page. There is more information on that section of Acquia's site. 

So, what's the process of updating your custom and contributed code from 8 to 9? The general thing that you're going to have to do is identify what code needs to be updated. You will need to understand the new method that exists. So, you know, if it's a replacement, if there's kind of a new pattern.

Some of you are making jokes. You will apply the update and then you will need to test the update. In terms of identifying the affected code, there are a few different ways. One of the ways that's available is on ‑‑ it's kind of sitting beside the link that I had sent before. And Acquia has a tool that you can scan contributed code. So it uses PHPStan which is a tool that analyzes code. And it will scan all of the contributed projects on Drupal.org and provide a summary of those deprecations. Here is an example. It gives you a good starting point for what you need to do. Here is a call to a deprecated message. I think all the 8 point whatevers are going away. So that part is not needed. And it tells you you need to use addMessage. That's a starting point.  I think they might be linked. Maybe not. I don't remember. But that is kind of a good starting place if you don't want to spin something up locally. It's an online tool. You can run this as a module. This is a project that uses the same PHPStan and checks for deprecations among other things. 

This is available as a composer package, and it is packaged up as this Phar package, which is a PHP method for making an executable‑like thing to avoid dependencies. So Composer has packages when you include things as Composer packages, they can include other things and they live all in the same place. But this is an attempt to remove some of that. So it's not ‑‑ it's just something to be aware of, maybe. So yeah. Those are two tools. We're not going to talk too much about them any more, because we want to get into kind of what you can do to replace them. 

So understanding the replacement can be kind of challenging, depending on your familiarity with the different systems and your familiarity withdrew pal.org and the change notice. So, one thing you can do is go to Drupal.org and in their API documentation, they kind of have these different places that you can see on the pages. If you are looking up the Drupal message page, there is a peach‑colored notification that says it's the same message that you saw before. This thing is not being used any more. You should use this other one. That's a link to a new system, which is nice. The other thing that you'll see, which is the one on the bottom here is that the change record is also linked. It's not called out as the change record, but that node, etc., etc., is the change record. And that change record has a lot more information. So it basically tells you, you know, here's a bunch of issues if you want the backstory. And it includes a general description. I haven't looked at all of the change records, but most of them follow this pattern where they have a general recommendation and walk you through examples. In this case, it says hey, you're going to need to get that messenger service, and then you're going to need to call any number of methods on that service, because there's not ‑‑ it's not passed as an argument any more. You have to find the right one and you have to pass in the message. It's not super complicated, but it's not as simple as find and replace a text string with another one. And you might not ‑‑ you know, you might want to kind of read through this and understand what's going on. 

Another way you could do this is if you're doing local development with your IDE, this is PHPStorm, it will cross out things for you or indicate while you are using this code that you shouldn't be using this. This PHPStorm puts a strike through it. It says if you have this, it is deprecated. You will see something like this, which has a lot of the stuff that you just saw before. This is in the Drupal code base. It says, you know, see that node? It's not called out as a change record, but it does point you to that thing. Then it says this is deprecated. So, that's kind of, if you're developing your stuff, you will see it there. Or if you're looking through your old code. You could basically copy and paste the examples, modify them as you need to. The code completion tools in the IDEs can be really helpful. You will find out right away if you are using methods that actually exist or if you are not. The parameters are not lining up the way you expected. 

Another option is to try to do this automated. Because such a large number of the deprecations. This is a command line tool and available as a Composer package. So you can install it using Drupal.net, Drupal rector. It's on packages. So that's why it has the effect. So when you use that tool, it looks like this. Basically you call that binary file, which is rector. Rector is actually just a broad php tool, which I'll talk about in a minute. So you're calling that tool, and then you say the task that you want. I forget what they're called here. But you want to call the binary and you want to process your code. You link to a module or a category. It does everything within that. In this example, we're doing dry‑run as an argument, which means it won't update it. It will just tell you what's going on. So it's an easy tool to use once you've got it configured. 

The output from this looks like a Git message or any other diff. It basically tells you we want to remove Drupal_set_message and we will preplace ‑‑ replace it with the hackie way of we're not going to use dependency injection. We're just going to do a static call. We're going to do add status. And it will list this for all of that code that you've just listed. If it doesn't find it, it won't do it. It does the rules you have included in your configuration. Rector itself is a tool that can be used for a lot of things. It supports other frameworks. It supports upgrades from PHP7 to 7.1, I think. I don't remember which versions. But it basically will attempt to do these updates for PHP things, for other libraries, for custom rules that you might write. It's a framework for doing this. There are frameworks that come packaged with it and we are trying to add Drupal specifics with it. Drupal rector itself is basically an easy way to use that tool. That's the first part. The second part is it comes with default configuration. It was tricky for us to connect the dots and make sure that all the auto loading was working the way Drupal needs it to work so that it can kind of connect things. I guess this is maybe a time to take a step back. So I mentioned that Drupal rector isn't doing ‑‑ replacing one string with another. It attempts to analyze the code within the context. And it knows that, like, this is a method or this is a cleanse. These are the arguments for it. And it kind of understands the inheritance of the classes and what's there and does sort of a versioning for this. But I don't remember off the top of my head. But it kind of figuring all of that stuff out. Because it does that, it needs a little bit of configuration, and that configuration is included and the Drupal rector project is a starting point.  

And the last thing ‑‑ I forgot. Sorry, Jason, are you asking if it does reflection? Or is that the name for it? Yeah. It includes a bunch of deprecation examples. What is the before example and what's the after example? If it says, like, go use this new method, that's helpful, but it's not the existing ‑‑ for me to learn how to do the stuff. The other thing that the examples help us with is you can download this tool and run them on the examples and see how they work, the updates. 

So, kind of got sidetracked there for a bit. But the last stage is that you need some way to test this. You can either manually test it by creating examples and testing the code. Just going to putter around and see if anything breaks. Or you can use automated testing like any of the tools that people commonly use. I'm not going to go any any of that too much, because that's not the focus of this discussion. But it's probably something that you will need to figure out. 

So kind of to review, in order to update your code, you're going to need to identify the affected code. Drupal check is a good tool. And then you need to understand the replacement. Drupal.org and blog posts are good sources. And apply the update withdrew pal rector, and then test the updates. I think the take‑away is there's a lot of information and there's a lot of tools out there. I think it's an optimistic view. Like, the situation where going from 7 to 8 was pretty challenging, but going from 8 to 9 has already been less of a challenge because of the way that the deprecations have been introduced, the update path. And then these tools are helping to fill in the remaining gaps there, you know, of doing those last bits. 

So what's the current state of Drupal rector? It currently supports Drupal_set_message in the most common use cases or what we imagine the most common use cases are. It supports them in an initial way, which doesn't necessarily do the ‑‑ it doesn't right now do the dependency injection or some of the other ways that might be better long‑term, but it gets your code in a working state. It does partial support for entityManager and a pull request for getMock, which is part of the Php unit. I think those are the top three. I know Drupal_set_message is the top one. These are already very widely used deprecations. And I think entityManager has been there ‑‑ oh. Database query are almost ready for a pull request as well. Entity manager was pulled in Drupal 8. It was decided ‑‑ I'm pretty sure it was entityManager. But because it was in the controller base, I think myself and everybody else just used it everywhere. So there are some of these things that are very old. But, you know, they are deprecating not just new things. 

Cool. So Drupal 8 rector, which has the number 8 in there is another project that this kind of came from. There was an original project called Drupal 8 Rector, and it with us very similar. The Drupal rector project was a fourth of that project. We took inspiration from that. It's not directly on GitHub, but it's related to that. We took a lot of information from that. It includes a couple more rectors or things that might cover additional Drupal deprecations. It also attempts to cover things beyond deprecations. Drupal rector has not seen a lot of updates lately. And it isn't using the power package or newer versions of Drupal Rector itself, change to use a package to eliminate the conflicts. Arthur did a lot of work to make sure it was working with the set‑up. It uses symphony. We're using symphony. The versions are not the same. We can't have those things conflicting and it was a problem that everybody was having here. So that's addressed in a lot of those issues. 

So, developing rectors. So this is kind of the process that we're going through in the project. We're creating examples, and then we're creating the rectors themselves, which includes creating a class, registering the class, and then developing the refactor code. 

So the first thing I have been doing when we have been making these is we have an example module in the repository, and we're adding a bunch of deprecated code in there. This is an example of a class in a custom module. It says hey, we're calling DrupalSetMessage the deprecated way. And then we create a class. So this is in the source directory. And this is very similar to how Drupal does things. So, we're creating a Php class that extends abstractRector, and we give it a name and it has these three primary things. Basically you give it some definition. You define what this applies to and how it applies the update. 

‑‑ Chris is asking what kind of test coverage you have. We're going to get into what nodes are, because nodes are a different thing in this case. Test coverage is ‑‑ we don't have it running the test and confirming that the things work. So the answer to that might be kind of no automated test coverage, but the examples are meant for that. 

Yeah. I'm going to explain the node thing in a second here. Cool. For those ‑‑ those are great questions, because it is very confusing. So you define this class, and then you register this class. And the registering is very simple to Drupal as well. There is a yaml file. In this case, there are configurations, and so we're say something we have bundled them by what minor version of Drupal they applied to or were introduced in. We have 8.5 deprecations, and they are all bundled up into Drupal all deprecations. This is is the full file for that, because it only does DrupalSetMessage. All you need to do is list the class and that's it. There are many other things you can do to make it complicated, and we tried to use the most simple example, because we anticipate people using the tool are not going to be super familiar with the tool, and we wanted to make it easy. The tilde in yml means null. It means this is the key for that thing, and it's an empty object. That is the definition for it. We're not defining anything for these keys. That was something that was new to me. 

All right. So yeah. We're getting into those class of the actual thing. So the getDefinition, I think it's required. It's useful that are looking at it to understand what you're trying to do at the least. So you have a string that says, you know, this is a description of what I'm trying to do. And then you can create this object which is called a code sample. And you can say this is the old example and the new example is the simplest thing. I have seen that in a lot of these Drupal rectors. They call them rectors, I guess. This is a paired down version to make it fit on the slide, but there are other examples in the Drupal rector project. So get node type. Nodes are not Drupal nodes. So that AST concept, I don't know if nodes is a thing in that. The concept is just that you have things that your php code is made up of, classes, statements, variables, arguments, methods, method calls. Everything is a node. Nodes can have other nodes within them. I think ‑‑ or at least they're connected to other nodes. Yeah. And so, that's what a node is. A node is literally like anything in php. And whenever you see that in rector, that's what it is referring to. None of this is Drupal‑specific. 

In this case, there's a bunch of classes that define these things. A function call is a type of an expression. It's like a subtype of expression. And the ::class says it will take the string and say this rector applies to anything that is a function call. So it only runs on those. 

So the refactor method is the main thing you will be doing in most of your work. You have already defined that this is a thing you want to run. You define what things it applies to. And then you figure out how do I actually convert it? So there's a lot going on here, and I have actually removed a bunch. But more or less, we're creating a node that's a static call. So the thing that we want is that ‑‑ I don't know if people can see my mouse. But the thing that we want is at the bottom here. We want to have a static call to the messenger service. And then that's going to have a call ‑‑ we're ‑‑ I don't know if I'm using the right terms. But we're chaining another call on that call to addStatus. Every single thing is a node in rector ideas. So we need a node called Drupal. And this is a thing that's a fully qualified node. That's the name of it. Yeah. That's just called Drupal, and that gives you this/Drupal. And then we need a static call that knows that it needs the ::. And method is a simple identifier, which is this string which is addStatus. And we're combining them in the last step. We're creating a node that is a method call, which is this whole thing that is going to be the method, which is add status on the messenger service, which is Drupal messenger, and the arguments that it's getting is example message. I didn't show you how to define arguments here, but it's in the code if you want to take a look. 

So the benefits and draw backs of this, the draw‑back is that it's very, very complicated for somebody to onboard to. The benefits are that you can swap out pieces of this and maintain all of the other nodes.  So if you needed to change addStatus, you could change addStatus and change all of the arguments. You can modify the list of arguments and remove one of them. So it's pretty intelligent for knowing, like, what are all the pieces and what are they connected to? 

All right. So, that being said, what are the resources you have in terms of developing these things? We'll walk through some of them, and then yeah. So one is that we kind of have this ‑‑ how are we doing the development? We have the repository on GitHub. It's a public repository. It has the example modules, it has the rector set up. It has a read.me, which is helpful. There are pull requests on GitHub. There is discussion on Drupal.org. If you want to sign up for stuff, that's the place to go. We have ‑‑ rector itself has documentation. So there is a training repository which has a bunch of examples. When we were learning, it didn't have that. So this is very helpful. The next one is this is the most helpful documentation page that I found is this nodes overview. This is a giant list of all of the different things a node can be. And the way I have used it is saying, like, okay, I don't necessarily know what the name of this thing is in PHP, but I know it has an arrow to it, and then it has the name of the method and it has the brackets. And I scroll through that page until I see that, and that is the name of the thing that I need to edit or create or look for, etc. So it's a really good reference. 

All right. Yeah. Besides that, Arthur, who is on the call here, I believe, has added a blog post, which has even more information on the palantir.net blog. In terms of development efforts, I found that the most helpful thing is to use Xdebug. You can explore the node objects, what they have, what they look like, how the connections are maintained within those. The other thing you can do is that nodes have a lot of methods that help you do those things, like to add something to it, to change it, to find the connection. There are also classes.  So each rector is a class, and that class has certain methods. It's not always clear to me why some things are on the method or on the class and some things are on the nodes. So, I found that using Xdebug and seeing the state of the code and what exists is the best way for me to figure out, like, how do I do that? 

All right. Lastly, we're using the same box on that's another repository on the palantir.net. It's kind of a quick start for developing new rectors. It has a redme in it. They are not really required ‑‑ they are not required for Drupal rector. They are just to help us install something, run something, see that it works, and then start developing for it. So it exists in that GitHub repository. It includes a Drupal site that is installed. It has a readme that is like set‑up instructions. We might be changing some of this soon. But instructions on how to set up a subrepository within that. It's not using submodules right now. But then run ‑‑ what commands you can use to run a tool against all of the other modules. It includes a vagrant or DDEV VMs and instructions on how to set those up. And those both support XDebug. So there are instructions in the same box that tell you, you know, hey, this ‑‑ if you log into the VM and turn XDebug on with this command, you can do that. And if you add this string before you run the rector command, it'll trigger the stuff in your IDE to kind of let you listen own XDebug. It doesn't tell you how to set XDebug with PhpStorm, but there is documentation out there already. 

What are the next steps? We as Palantir have goals and there are community goals as well. So Palantir's goal is to maintain the Drupal rector package. We want to develop the rectors themselves. We're focusing on just deprecations for this project. So we want to really cover most of the common deprecations and see how far we can get with that. We also want to potentially cover future deprecations as those happen to Drupal 9. We want to provide documentation. There is existing I don't want to read through them. Can I run it and see what the tool is telling me it should probably look like. And maybe I can try it and it just works. Those examples are helpful. So learning how to do those and expanding the way that the rectors can apply the deprecations would be really good. Those are some goals. Provide documentation around them. Potentially add ‑‑ providing a way to ‑‑ sorry. That's redundant. 

It's not trivial to do these things. If a deprecation came out and it continues to be a good tool to have those deprecations available right away as rectors that people can apply if they want to. 

I want to thank a lot of people. They are listed on this slide. The original project creator did a lot of the surfacing of the information to us. We had a lot of additional contributors on the original project and the project that we have. Ofer is on the call. He has done a lot of work. He is the maintainer for this project. He also works at Palantir.net as a senior frontend developer. 

And lastly, thank everyone else who is joining here. I have like a quick review and then open it up to questions. The quick thing is that we're going to have a ‑‑ I thinkly be around and Ofer will be around on Saturday during the contribution day to kind of help people get started with this or to talk about how they might work with Rector if you're interested in using the tool or if you're interested in developing things for the tool.

This is going to be the same for all the projects. Here is the code, 6364. I think we have about ten minutes for questions. 

>> Can you hear me? 

>> Yeah, I can hear you. 

>> Does rector's abstract syntax tree here, do the nodes allow me to navigate up from maybe the given node that I identified? So, like, in the case of Drupal set message as our example, when I get that node, can I navigate up the chain to say the wrapping method or the wrapping class that's holding the deprecated code? So that I could do things like identify whether or not it implements or extends some base class? 

>> Yeah. You can. So, let me try to do a sharing. Can you see my ‑‑ 

>> Yes. 

>> Okay. 

>> Thank you in advance for winging it. 

>> Let me find out where it is. The answer to your question is yes, it can do that. That's the benefit of having that node tree. For this case, I'm looking for the static calls. You can also look for the static calls for things that are within it. It will know that this static call is within ‑‑ I think Drupal set message is the better example. I think we're doing exactly what you're saying. Yeah. The quicker solution that we have landed on is we're just going to call it statically. 

>> Nice. Thank you. 

>> Yeah. That's a great question. I tried to look at some of this stuff. I think one of the things that came up in chat, which I'll just mention here, since it's not ‑‑ I don't know if this will be recorded or the chat comments. There was a question about the ? Node as a return type. And I think Christian said it means node or null. I don't know if anyone verified that. It looks like you did. 

>> Yeah. It's nullable types. 

>> Okay. So I'm not sure why that was exactly that way, but I think that was in there. Right now we don't have tests running as part of the project. As we're developing things, we test them that way. I don't ‑‑ the way that it would fail that it has failed in the past is that rector itself ‑‑ not Drupal rector ‑‑ has had changes, and those changes have caused issues and the tool just produces a bunch of error messages. So that's the way that while we're testing up, it's pretty easy to see that those just don't work anymore, and rectors themselves need to be updated based on the tools changing. 

>> So, I want to verify something that we're still staring at here on your screen, because it's implied, but I want to make sure I'm reading it correctly. So when I get the class name here, and then I get traits by class, so rector is actually navigating not only the individual class that I'm looking at, but also any super classes that they might extend and traits that those things might have as well. So I can tell if that trait is implemented on any class I might have extended from? 

>> Right. In this case, let me look at the source code. This was something that came from pretty much copy and pasted I think exactly from the Drupal rector project. This was a helper trait that somebody added to do that navigation. And so I don't know exactly how it's working. The first is get parent class, get traits. There's a whole bunch of these methods that exist. 

>> Cool. 

>> And not only to look at them. But you can modify them as well. So if you wanted to do something like go up the tree, add the trait to the class, it's possible. I haven't done it because it's ‑‑ I haven't figured out how to do that yet. And I have been trying to focus on ‑‑ like I know how to do the static stuff, and I want to get the why before jumping back and doing some of the more complicated cases. But it could be good if people are interested in it.

I can call it if you want to take a look. And if anyone has any questions, if you're able to, just jump in and ask the questions.

What's that? 

>> So, it would be obviously nice if one tool would do it all and if it could handle the proper dependency injection instead of just doing a static call. But in the short‑term, we can combine it with our other tools. So Drupal console can add the dependency injection for us. And then it looks like PHP rector, if it can find traits it can maybe also see what services are injected? And if the service is injected, then use that. So if we could combine the tools, then that would probably give us a really satisfactory upgrade path in two steps. As it is now, if I'm replacing Drupal set message with these calls, then I have to go back and fix those. It doesn't seem satisfactory to me. 

>> Uh‑huh. Yeah. That's interesting. I haven't used the tools for the Drupal console. You said it was only for Drupal console or part of Drush? 

>> Drush10 also includes code generation. 

>> Hi. This is Ofer. So we're still trying to also learn rector itself, because it can do so many things.. I do believe that a lot of what we need for dependency injection, we just need to explore more. And a message from Chris regarding twig. I do know that rector itself is knows already how to do upgrades of twig. 

>> There were other questions. One of them is how are we keeping track of deprecations? Right now there are some issues open in the issue queue for the larger ones. And I think we kind of just had the list. But that's a good question. So maybe the answer to that is that we will have a better way of listing the ones that are covered. 

>> The documentation page? 

>> Right now there's just a readme in the project. And so, you know, maybe they're adding a documentation page or some other thing that would be helpful. The way we have been trying to do development is to look at examples. Because reck store is used, I think there was some variable stuff in there. They have done similar things that we could potentially use as examples. Personally it's kind of impossible figuring out how it works. So we wanted to do this simple examples with a couple of these. And as we learn how to do these more advanced things, we could go back and update the ones that we have or use those as sort of a copy and paste examples of how to do the stuff. 

The instructions here, let me see here. This is a command that I need a copy. All right. So that's what is on if you ‑‑ yeah. So you can kind of see the output there. I did a dry run so it's not replacing them. If I run this command, which is basically figuring out what port I'm on or sorry, network address I'm on, and then telling it to do that. I don't know how other string things that work in Linux, unless I look it up again. Essentially that's what it's doing. Configure XDebug on the fly and then running it. And rector has this there. And hopefully it will stop. It did not. That's why demos are not always a great idea. It was working yesterday. We need to look at that. Yeah.

If you want to stick around for a minute, I will try to start this up and show you what XDebug looks like. If you have questions that jump out while we're doing this, otherwise this is kind of ‑‑ yeah. We'll have our discussions on Saturday for anyone that's interested. And we're always looking for help and suggestions.

>> That was a good discussion, folks.

>> Thanks, Dan. This was great. 

>> Thanks.

>> So just a quick clarification. We said that it could be used, but you're not currently doing that. Did I understand that properly? 

>> Well, I know that we aren't doing it now. So it sounds like it is supported in general. But we have tried it withdrew pal. 

>> Right. We didn't do anything specifically withdrew pal, because I believe most of the twig changes happened to happened according to change. I don't think that things ended up using twig changes. But Rector itself has what they call sets, and these are the packages to upgrade PHP and twig. So if you just run rector sets, you'll get a list of all of these things. And then you can run, for example, rector classes ‑‑ ‑‑ set, I believe. And twig 241 is one of the names and you can see what rector can do with upgrading twig to the latest version. 

>> Great. Thanks.

>> Yeah. So while we're waiting on this, maybe this is what the configuration file looks like. We wanted to make a configuration file that you could use out of the box or you could modify if you wanted to. It's a yml file. It lists other configuration files so you can aggregate them up. By default, we're loading all the deprecations, but we also split them up by minor releases. And what we updated to kind of get this working, as I mentioned before, was this autoload_paths to tell it to look in core or core/modules if you're using Docker, you would have to modify that here. And also to tell it that php files in Drupal are modules, theme, profiles. And the services key, this could be defined here, but this is where we're loading all of the other ones that are defining the services objects.

I'm having issues with my VM, which I think may be because I'm doing screen shares so I'm not going to be able to do the XDebug. I will get it sorted out and if people are interested in taking a look on Saturday, that will be the time, I think.

>> Very good.

>> Cool. Well thanks, everyone. I'll post ‑‑ yeah, I guess otherwise the discussion ‑‑ the page has the slides. Michael had his presentation earlier, preparing custom and contributed code for Drupal 9. I encourage everyone to check that out and look at the GitHub page. It has links to the sandbox and other stuff and blog posts. Thanks, everyone. I'm going to stop the screen recording on my end.