mybb Blueprinting Automatic Updates for PHP Applications

[Guest Devilshakerz]

Keeping MyBB boards secure is a team effort. Security issues discovered and reported by external researchers and our core developers are analysed, fixed and included in final packages. The process doesn’t end there however: it is essential that administrators are notified to update their forums as soon as possible in order to prevent the addressed vulnerabilities from being exploited in an attack on their boards and users.

 

 

Our recently published summaries, recommendations and links to reviewed guides in the SECURITY.md file contain many resources forum administrators can use to secure their boards against both opportunist and experienced digital criminals. First and foremost though, we always recommend that users keep their MyBB installs up to date. We also suggest using the new subscription feature for all used plugins and themes on Extend.

 

Based on our experience, even large discussion boards that don’t have dedicated technicians tend to use outdated versions of MyBB and the situation in the area of extensions might be equally concerning. Not unlike other software, periodical updates are the main method of delivery for security patches — most MyBB releases contain fixes plugging security holes ranging from theoretical risks to critical vulnerabilities.

 

 

The need for continuous response to vulnerability reports is a strong argument for making the reduction of manual effort needed to keep our packages up to date a long-term goal.

 

In this post we’ll explore what keeps our developers up at night that also affects MyBB’s ability to introduce automated updates, and how the mechanism might be actually implemented once the system — currently being rebuilt for version 1.9 and subsequent branches — is ready.

 

 

Paying off the Technical Debt

 

 

Like most major features, unsupervised updates are difficult to introduce due to the existence of some pain points related to how MyBB up to and including 1.8.x was designed.

 

• Self-Check
   
   
  While MyBB can already (naively) verify the integrity of its core files using mybb.com-hosted checksums (upgraded to sha512 not long ago), with custom modifications or intensive development more detailed information could come in handy. The waters of side-by-side file content comparison and database schema difference report have been tested with an external plugin and the mechanism is being considered for implementation in the core package, aided with digital signatures and extended to pre-install verification.
   
   
  
• Development Flow in Code Repositories
   
   
  At this point all code changes are merged into a single feature branch in the repository, forcing a linear development & release cycle that results in withholding from merging new Pull Requests until the release and makes preparation of out-of-band releases (like security hot-fixes) more difficult, given our aim to make the code of all releases available for inspection and reproduction.
   
  The planned switch to gitflow, a git branching model used by large projects at different capacities, will make preparation and publication of new MyBB releases easier as well as allow more automation without sacrificing resources to maintain project-specific packaging tools. The consistency of discussed self-checks will also improve as maintainers will always be able to create a version tag for any release without blocking the process.
   
  Complete decoupling of maintenance/feature and security updates will additionally allow for simpler upgrade process for the latter; potential regressions resulting from non-critical changes will not impact security updates, where friction should be reduced to minimum.
   
   
  
• Database Schema Migrations
   
   
  MyBB development and usage is currently negatively affected by lack of version control for the database. After a code change is tested, the developers often have to manually reset the installations to revert the database changes. Moreover, when something goes wrong, whether during testing or in production, it may be difficult to investigate the issue due to the lack of the ability to apply some changes selectively, re-run certain operations once they’re fixed, or confirm that all were successfully completed.
   
  In the case of the filesystem, we can compare the same file between distinct versions and allow users to update it by providing a changed files package — within the realm of databases, the equivalents are the comparison of schema (like table definitions, without the data) and migrations. Each database-related change will contain operations for applying and reversing it, which should prove immensely useful whenever more complex updates are handled.
   
   
  
• Contained Update Logic
   
   
  The 1.8.x update script, usually run to synchronise database schema and content, is built with the expectation of being walked through by the administrator using a web browser and includes additional headers and HTML output. This, in the case of the whole process happening in the background, is not only unnecessary but can often make non-interactive execution problematic. It’s also lead to bugs with the HTML output in past releases.
   
  All operations will need to correctly fetch custom settings, avoid assumptions for conditional behaviour and provide fallbacks and precise feedback for tasks failed and succeeded. The layers on top — HTTP-based installer, CLI manager or the MyBB engine itself — will call subroutines containing such operations and interpret the results as needed.
   
   
  
• Merged Code Stability
   
   
  New or modified 1.8 code often introduces regressions that break existing functionality. The two underpinning reasons are a dated codebase that makes some mechanisms difficult to understand for developers — even for those with a lot of experience — and a lack of automated tests. Automated testing is one of many things that require the code to be restructured in order to use modern tools, and the ground work for unit testing and continuous integration is being introduced in MyBB 1.9.
   
  

Composer — We’re Hooked

 

 

A strategically important extension of our toolset is the integration of Composer, introduced in our post on building MyBB packages. This importance comes from the ability to delegate version management of third-party modules and the application itself, and what’s equally anticipated is the possible usage of its event hooks, taking place of MyBB 2.0 CLI database migration and seeding. The installation, and especially upgrading, can execute MyBB’s own scripts or CLI commands to take care of all changes that have to be applied after the new package is fetched and put it place.

 

 

Composer, along with git repositories, is also considered for plugins and themes. Besides avoiding maintenance overhead — by publishing them in repositories, with only minimal data provided on the Extend section — extensions could be updated the same way MyBB would be. Simplification for extension developers is crucial as MyBB installations communicate with Extend to check for updates, including those addressing security issues in plugins.

 

Left: Updates even for some popular extensions are not always pushed to Extend, but at least this one doesn’t come from an influential developer…

 

Shared and Cared

 

 

With the adoption of Composer — the flagship improvement during the refactoring of MyBB 1.x — and plans to further encompass developer relations (the Project takes great pride in the concept of extensibility after all), we always signalled that MyBB will be ready for download and installation without a stack other than that available on modern shared hosts (the script will require, among others, much more recent versions of PHP, but we consider it a fair game with the added benefit of excluding unsafe web hosts that don’t update their software). So how will it affect the update process?

 

In the big picture the application will be tasked with updating its own code, rather than being managed externally using the administrator’s CLI or run by scheduled Composer package updates. A self-update feature that works out-of-the-box equates to creation of logic within MyBB itself to mimic the process:

 

• update checks, relatively easy and already present, however MyBB installations only query the mybb.com website and upon manual request — more central or decentralised locations (through opt-in communication between installations) can be added, to go with periodical checks,
   
  

• 
  downloads, more tricky and prone to network and system problems,
  

• 
  verification, non-trivial to implement correctly and requiring strict procedures on the part of project maintainers (release signing awaits implementation), and
  

• 
  execution, risky in case of serious regressions in updates — although those will likely decrease in frequency, a recovery plan should be available.
  

 

While the mechanism could be executed differently, some tasks will likely be shared, for instance creating backups. MyBB 1.8 comes with a weekly run database backup task, which could be extended to cover files (like uploads and avatars) and securely integrate with external archiving services.

 

Updating improvements, CLI, asset managers, transition to Markdown for simple formatting or how extensions could be built and shared are only some of changes considered for implementation while the roadblocks are being removed. Take a look at MyBB’s Development Roadmap for future branches and join our Community Forums and Discord server to discuss how MyBB should look like.

 

 

 

Continue reading...