Friday, June 8, 2018

Upgrading Drupal from 8.3 to 8.5

Background

I am the sysadmin and developer for Art & Object, a Drupal 8 website built with the Drupal Composer project. The version pin in composer for Drupal was 8, which in hindsight was too broad for our usage. Meaning Drupal point releases (8.3 to 8.4) require study to ensure you understand all the implications, which wasn't something I did. I just blindly did a composer update, thinking everything would be handled automatically.

This really bit me when 8.4 came out because my server was running Debian Jessie, which runs PHP 5.6 and my composer didn't have a platform PHP configuration, so a lot of the underlying Symfony code updated to PHP 7. So I ended up doing a backgrade until I figured it out.

Then there were the critical security Drupal updates (SA-CORE-2018-002 and SA-CORE-2018-004) earlier this year that would not be released for 8.3, so I had to upgrade (or at least, at the time, I felt I had to, though I see now they have a patch for older 8.x releases). By that time, 8.5 was released, so I updated the composer to 8.5 and ran update and after some basic testing, moved on.

Then a few months later, I noticed the status error messages about running the contrib media module alongside the core and I knew I missed something and there was a problem.

I then started down a wicked rabbit hole of getting a local copy running and following the upgrade instructions, running into problem and going back to getting a local copy running fresh again and trying again. Lots of trail and error (mostly errors) and head-banging-on-the-desk. I looked for help on the #media IRC channel, but the best advice came from posting on Stack Overflow, where @sonfd pointed out that the media module needs to be uninstalled first. I thought I had tried that and ran into an error message that mentioned you can't uninstall the media module with media items already created.

The Fix

So after lots and lots (and lots) of local refreshes and trials and errors, here's the list I finally followed when it came time to upgrade production:
  1. First, put the site in maintenance mode. Then take a database backup and make a tarball of your project directory. Don't skip over this.
  2. drush pmu media crop_media_entity: pmu = pm-uninstall. Remove the media module (and crop_media_entity, if you have that, too). This was the tip from @sonfd that opened the rest of this process for me.
  3. composer remove drupal/media: Remove the contrib media module from the filesystem. I should add that I prefixed all my composer commands with /usr/bin/php -d memory_limit=-1 /usr/local/bin/ because I often ran into memory limits when running composer.
  4. composer require drupal/inline_entity_form drupal/crop:1.x-dev drupal/media_entity_instagram:2.x-dev drupal/media_entity:2.x-dev drupal/media_entity_slideshow:2.x-dev drupal/media_entity_twitter:2.x-dev drupal/slick_media:2.x-dev drupal/media_entity_actions: These modules are temporary to help upgrade the database records.
  5. composer remove drupal/video_embed_field: For some reason, I couldn't require video_embed_field:2.x-dev, so I removed it and then...
  6. composer update: When I ran this, it updated video_embed_field to 2.x-dev.
  7. composer require drupal/media_entity_image drupal/media_entity_document drupal/image_widget_crop: More temporary modules to help the upgrade process.
  8. drush cr: Clear cache to make sure Drupal picks up new modules and paths.
  9. drush updb: Run the database updates.
  10. drush pmu entity media_entity: Uninstall these modules (these were the old contrib modules)
  11. composer remove drupal/media_entity drupal/media_entity_image drupal/media_entity_document drupal/crop drupal/image_widget_crop
    /usr/bin/php -d memory_limit=-1 /usr/local/bin/composer require drupal/crop:2.x-dev drupal/image_widget_crop drupal/empty_page:2
    : Clean out the temporary modules from the filesystem.
  12. drush cr: Clear caches
  13. drush updb: Run database updates
  14. drush cex: Export the configuration (so you can commit it later).
  15. The blazy module had an error with the core media and hasn't been updated (as of this writing), but there is a patch to fix that. So I learned how to add patches to a composer file - turned out pretty simple. Add this to composer.json in the extra section:
            "patches": {
                "drupal/blazy": {
                    "Gets Blazy to work with Drupal Core Media": "https://www.drupal.org/files/issues/2881849-8.patch"
                }
            }
  16. composer update: This was odd, but I had to do an update, which picked up the patch, but didn't really install it. I can't remember exactly now, but I believe this actually deleted the blazy folder.
  17. composer remove drupal/blazy: So removing this actually installed it. Who knew? Whatever ... it's still in my composer.json and now the filesystem has the module and the patch.
  18. drush cr: Clear caches!
  19. For some reason, this upgrade created a new field called field_media_image_1 and assigned that as the source for the image media type, which broke some of the images on the site. So I edited media.type.image.yml file to revert source_field back to my original field_image.
  20. drush cim: Import my hack to get my media image type to work.
  21. I had a custom field formatter that I had to edit to change the namespace from media_entity to media.
  22. drush cr: Final cache clear!
  23. Test and make sure all is well. If so, take the site out of maintenance mode and commit your repo changes.

Advice / Conclusion

A lot of this pain could be negated by studying the release notes better. I own that and this counts as one of my many scars of lessons learned. I hope others can learn from my lesson, too. Someone may end up writing a meta post about this post to point out the high cost of maintaining a Drupal site and I don't think they'd be wrong about that, but that's the price you pay for running servers that are publicly accessible.

Tuesday, October 24, 2017

Apex November 2017 Municipal Elections

It's that time of year again for municipal elections and we have two items coming up on the ballot, a parks bond and three seats on the town council.

First, let's talk about the Parks Bond. If you're unfamiliar with it, the town has put together a great FAQ that should address a lot of your questions. I asked John Brown, the Parks director, why the growth itself isn't enough to fund the initiatives and he pointed out that as we grow, those additional tax revenues are typically needed to offset the cost of providing services to meet the demands of the larger community (public safety, streets, infrastructure, etc). He also pointed out that if the funds are there, the council can opt to not issue some of the debt or increase taxes to repay it. If passed, the bond will add additional greenways (connecting 17 more neighborhoods!), provide for a senior center, and build out Pleasant Park (which could be useful for tournaments, meaning soccer parents, etc may not have to travel as much). In my opinion, this will have a great impact on our quality of life and we should all vote Yes for this.

Second, there are three spots opening up for the town council, Nicole Dozier, Gene Schulze, and Denise Wilkie's seats. Ms. Dozier is the only one running for re-election. There are five candidates running for these three seats:


I went to the candidate forum last night to hear from the candidates. If you were unable to make it, they will be posting a video on the Chamber website within a few days. Overall, the candidates seemed very similar to each other with very little disagreements.

So who has my votes? Well, I will be voting NDG (play on the NBA signs around town, encouraging people to vote for the block of Nicole, Brett, and Audra). Ms. Dozier impressed me with her experience, advocacy, and her drive to get a senior center in place. Mr. Hooks impressed me with his town experience (being on the planning board and a lifelong resident of Apex), poise, and volunteering with kids. Mr. Wilson impressed me with an emphasis on economic development.

Finally, if you want to see what the ballot will actually look like so you're prepared, you can see it here. Short and sweet!

Voting is November 7th! You can find election information here, including how to get an absentee ballot, your polling location, and more.


Wednesday, October 11, 2017

Watch out for Drupal 8.4!

I was using a Drupal composer template and a few days ago, it upgraded from 8.3 to 8.4 automatically. I noticed and didn't really think much of it, so I applied the database updates and went on my way. Today, two days afterwards, I ran across this article which pointed out that PHP 7 is required for the underlying Symfony framework. Our site is running on Debian 8, which has PHP 5.6 and powers other PHP applications, so I wasn't looking to update the underlying OS and possibly break my other PHP things.

So I downloaded a db snap from 8.3 and then re-applied the 8.4 update and took another database snapshot. Then I diff'd the database snapshots. The biggest changes seemed to be in the cache tables and removing and adding some revision columns. So I reverse-engineered a backgrade SQL script. With that, I updated the composer.json file from this:
        "drupal/core": "~8.0",

to this:
        "drupal/core": "8.3.*",

Then I took a precautionary database backup and then did a composer update, which took care of the code backgrade. Then I ran my script (drush sqlc < backgrade.sql) and then I did a drush entity-updates to actually update the database schemas to match the backgraded code.

Now I just need to ignore Drupal telling me about 8.4 until I'm ready to fully embrace PHP 7. I feel like something bigger needed to get my attention to the update in system requirements when I ran the initial composer update.

Thursday, September 14, 2017

Ode to Sendgrid

I recently have been playing around with Sendgrid and it is pretty amazing. I have had some experience with sending email programmatically with AWS' SES product and it's nice, especially with its SNS integration, but Sendgrid's killer feature is its speed. I had about 6,000 emails to send, each one personalized with an individual URL and the person's first name. With AWS and a multi-threaded PHP program, I can send about 15 emails per second, which would have taken almost 7 minutes to process. With Sendgrid, I create a template in their system with template tags and then I create a massive data structure with up to 1,000 entries and then post it to the Sendgrid API. Rinse/repeat. I was able to send my 6,000 emails in 6 seconds! Sendgrid took care of the personalization on their end and I was seeing emails being delivered very soon after. Wow. Here's my script code:

I've only scratched the surface - there's lots more Sendgrid offers. One of the nice things is that you can get an activity feed and some nice stats after you send an email to see how people respond:

Activity Log, showing opens, clicks, delivered, etc.

Stat report, showing email performance.



If you're a Google Cloud customer, they offer a free trial with up to 12,000 transactional emails per month. Definitely worth a checking out!

Monday, June 5, 2017

Global Climate

Climate change has been an ongoing issue for about 20 years (maybe more). I was influenced early on by Michael Crichton's book, State of Fear, which posits that there's no real scientific evidence of global warming. Plus, it was just another thing for humanity to focus on, that we need something to stay fearful of. It's an interesting theory, especially if you consider that the current growing fear of terrorism has probably reduced the fear of global warming, mitigating any resistance to progress. A few years ago, Casey shared this punnet-like square video of all the various perspectives of are we going to do anything about it and if global warming is caused by humans or not. So regardless of what you believe and if anything can be done, the risk of doing nothing seems a lot worse than the risk of doing something and it ending up not being a thing.

So with that out of the way, the current news of Trump and the Paris Climate Agreement has me more aware of local efforts. If the US government isn't going to lead the way, then we need to do what we can on a local level. As a North Carolina citizen, we have an opportunity to keep our state moving forward towards more renewable energy sources. There's a bill that was just filed with the NC House that rolls back some of the goals (the minimum stays at 6% vs. growing to 12.5% by 2021). I'm not a lawyer, but it also looks like nuclear energy will be considered renewable, which probably means that Duke Energy could just buy energy from its own nuclear plants vs. other renewable energy sources.

So, if you're a NC resident and you want to do something, you can find your representative and express your opinion. You can also look at what your town is up to.

Thursday, January 12, 2017

Drupal 8 project killed

I realize I haven't updated my blog in awhile, because I've been in the thick of developing and rolling out our new site (check it out!). This was taking our existing Drupal 7 site for FSR Magazine and migrating it to this new site, which is responsive and better catered to deal with all food service news. Unfortunately, we ended up killing the Drupal 8 upgrade because it was just too complex, given our custom module codebase. D8 has a mindset change, where everything is an Entity and it takes a lot of OOP and YAML files to get things done.

I figure it would cost us hundreds of hours to upgrade to D8 and it was tempting to upgrade to D8 for the configuration management, Big Pipe, and better caching features. But that's time and money we didn't want to spend on the upgrade.

I've talked with some folks at my local Drupal User Group (shoutout TriDUG!) and several others are in the same boat, where existing sites aren't upgrading, but rather, they're doing new sites in D8.

At some point, we'll need to do something, but we're able to do what we need with D7. If pressed, it may be easier to port to Backdrop vs. upgrade to D8.

Monday, June 20, 2016

Entities (Porting Drupal 7 modules to Drupal 8)

Preface

Wow. Entities. What a mess. I really struggled with this. There's an example of a database-based module not using them and one that does use them. I think it boiled down to me that what I was representing were reports, which seemed very similar to nodes (in hindsight, perhaps this should have been a content type all along), so I should use entities.

Entities

As I look through examples, trying to figure out how to get my menu routing to work with my database entries, it appears that I need to set them up as entities. I say that, looking at the xmlsitemap and dbtng_example and content_entity_example from the examples module. So, let me back up and learn about D8 Entities. It looks like entities were introduced in Drupal 7 and there is a guide for that, but it's based on D7, so the concepts stuff is ok, but the implementation guidance is outdated. There's a link at the top to take you to a D8-specific guide. The guide says it's just a holder, but I'm going to give it a go. If nothing else, perhaps this post will help others.

First, entity types. Either you're setting up a configuration entity or a content entity. I think content entity fits for me. Next, looking at requirements section of the content type doc, I need to setup src/Entity/GatedContentEntity.php file with docblock stuff that will give the loader the information it needs. Hey, checkout this helpful hint about double quotes in the annotations doc (post #1 reference)! That annotations doc doesn't provide what all you need, but I'll take those 3 lines to start off with. I think this doc may list all of the properties you can use with ContentEntityType, but not sure which ones I really need and how to use them. This is when the examples/content_entity_example/src/Entity/Contact.php file helped. I first just used the core form & access handlers for now.

Random Aside: Sublime Text 3

I use Sublime Text 3 as my IDE and there is a PHPDoc plugin that makes life editing these annotations easier. Package Manager makes it easy to install plugins: just follow the installation instructions (and restart ST), then follow the usage instructions to pull up the Install Package option and then type phpdoc and select it to install. Then restart ST again and you'll be much happier. I think the same applies for Sublime Text 2, too. This Drupal doc has some other helpful Sublime Text tips.

Back to it...

So I found this great doc on how to create your own content entity. And whoa ... this comment pointed me to the Drupal Console project, which can jumpstart this a LOT. So I tucked my handcrafted code away and ran this command:
drupal --target="fnmd8.local" gect --module="gated_content" --entity-class="GatedContent" --entity-name="gated_content" --label="Gated Content"
The end result is about 10 files with lots of boilerplate fields that don't match up with what I'm using. Maybe I'm going about this all wrong. Maybe I don't need Entities? Time to take a break...

After the weekend

So I went back and added the preface above and I feel I need to stick the course with Entities. Perhaps this post will be totally rewritten after I totally understand them and and write a better guide that flows better. I need to just release this post because I'm about to flip back to some D7 work and will be taking another break. I'll pick this post thread back up again shortly.