Automate everything! – The power of puPHPet

As you readers probably know, I can’t stand XAMPP and MAMP, being two steaming piles of crap, and have long advocated that you set up VirtualBox & Vagrant, then head over to http://www.puphpet.com, fill in the forms to configure your VM,  generate the config.yaml, and then unzip it and run ‘vagrant up’ to install it. Brilliant so far.

Yesterday I had a total downer of a day, trying to run an old legacy PHP 5.3 app. PuPHPet doesn’t have the EOL PHP 5.3, so at first I settled as a one off for MAMP, but it was slow and horrible.

Then I thought, wait! If I don’t configure Apache or PHP in puphpet, I could get a box up and install 5.3 myself. That’s when I discovered the awesomeness of the puphpet/files folder.

The only thing I used in there was the ssh keys. But there are empty folders waiting for .sh files (shell scripts) to be dropped in.

So for this box, I created exec-once/install-stuff.sh which contained the following:

#!/bin/bash
yum -y install httpd php
yum -y install php-mysql php-devel php-gd php-pecl-memcache php-pspell php-snmp php-xmlrpc php-xml

Then upon running vagrant provision, it not only looked for changes in config.yaml, but it checks for changes in these files too!

I then made set-vhosts.sh, and import-database.sh, which look like these:

#!/bin/bash
echo "
===========================================
Adding vhosts to /etc/httpd/conf/httpd.conf
===========================================
"
echo "
<VirtualHost *:80>

   DocumentRoot /var/www/fife/web
   ServerName fife
   ErrorLog /var/www/fife/log/error.log

   <Directory "/var/www/fife">
      Options -Indexes +FollowSymLinks
      Order allow,deny
      Allow from all
      AllowOverride All
  </Directory>

</VirtualHost>
" >> /etc/httpd/conf/httpd.conf

And …

#!/bin/bash
mysql -u root --password=123 --database=fortdev < /var/www/fife/data/sql_scripts/symf_fortdev.sql

I take it by now you get the idea! So now you can totally destroy your VM, and put any customisations in these shell scripts, so your full setup can be back up in 5 minutes flat with a vagrant up and vagrant provision!!!

You can then also start thinking about using puPHPet for deploying your setup to your production server 🙂 There’s a vagrant plugin called Vagrant Managed Servers, which will take care of that for you. https://github.com/tknerr/vagrant-managed-servers . I haven’t looked at it yet, but of course you can expect a blog post on it here when I figure it all out!!

Advertisements

Setup Doctrine2 Migrations and cli-config.php

After composer requiring doctrine/migrations, you need to edit your cli-config.php. Here’s mine, with the new stuff added in bold.

<?php
use Doctrine\ORM\Tools\Console\ConsoleRunner;
use XYZ\XYZService;
use XYZ\Test\XYZTesting;

// This is just a dependency injection container
$container = XYZTesting::getContainer();

/* Edit these details to suit

$container['db.credentials'] = array(
    'driver' => 'pdo_mysql',
    'dbname' => 'twg',
    'user' => 'dbuser',
    'password' => '123',
];
 */

// this is just my service which returns the entitymanager (requires the DIC above)
$svc = new XYZService($container);
$em = $svc->getEntityManager();
$helperSet = ConsoleRunner::createHelperSet($em);

// Add Doctrine Migration commands
$cli = ConsoleRunner::createApplication($helperSet,[
    new \Doctrine\DBAL\Migrations\Tools\Console\Command\DiffCommand(),
    new \Doctrine\DBAL\Migrations\Tools\Console\Command\ExecuteCommand(),
    new \Doctrine\DBAL\Migrations\Tools\Console\Command\GenerateCommand(),
    new \Doctrine\DBAL\Migrations\Tools\Console\Command\MigrateCommand(),
    new \Doctrine\DBAL\Migrations\Tools\Console\Command\StatusCommand(),
    new \Doctrine\DBAL\Migrations\Tools\Console\Command\VersionCommand(),
]);

return $cli->run();

Note that I have removed the last line, return ConsoleRunner::createHelperSet($entityManager); and replaced it with return $cli->run(); Now if you type doctrine:

$ doctrine
Doctrine Command Line Interface version 2.5.1

Usage:
command [options] [arguments]

Options:
-h, --help            Display this help message
-q, --quiet           Do not output any message
-V, --version         Display this application version
--ansi            Force ANSI output
--no-ansi         Disable ANSI output
-n, --no-interaction  Do not ask any interactive question
-v|vv|vvv, --verbose  Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug

Available commands:
help                            Displays help for a command
list                            Lists commands
dbal
dbal:import                     Import SQL file(s) directly to Database.
dbal:run-sql                    Executes arbitrary SQL directly from the command line.
migrations
migrations:diff                 Generate a migration by comparing your current database to your mapping information.
migrations:execute              Execute a single migration version up or down manually.
migrations:generate             Generate a blank migration class.
migrations:migrate              Execute a migration to a specified version or the latest available version.
migrations:status               View the status of a set of migrations.
migrations:version              Manually add and delete migration versions from the version table.
orm
orm:clear-cache:metadata        Clear all metadata cache of the various cache drivers.
orm:clear-cache:query           Clear all query cache of the various cache drivers.
orm:clear-cache:result          Clear all result cache of the various cache drivers.
orm:convert-d1-schema           Converts Doctrine 1.X schema into a Doctrine 2.X schema.
orm:convert-mapping             Convert mapping information between supported formats.
orm:convert:d1-schema           Converts Doctrine 1.X schema into a Doctrine 2.X schema.
orm:convert:mapping             Convert mapping information between supported formats.
orm:ensure-production-settings  Verify that Doctrine is properly configured for a production environment.
orm:generate-entities           Generate entity classes and method stubs from your mapping information.
orm:generate-proxies            Generates proxy classes for entity classes.
orm:generate-repositories       Generate repository classes from your mapping information.
orm:generate:entities           Generate entity classes and method stubs from your mapping information.
orm:generate:proxies            Generates proxy classes for entity classes.
orm:generate:repositories       Generate repository classes from your mapping information.
orm:info                        Show basic information about all mapped entities
orm:mapping:describe            Display information about mapped objects
orm:run-dql                     Executes arbitrary DQL directly from the command line.
orm:schema-tool:create          Processes the schema and either create it directly on EntityManager Storage Connection or generate the SQL output.
orm:schema-tool:drop            Drop the complete database schema of EntityManager Storage Connection or generate the corresponding SQL output.
orm:schema-tool:update          Executes (or dumps) the SQL needed to update the database schema to match the current mapping metadata.
orm:validate-schema             Validate the mapping files.

Loads more commands! Now you can start migrating your database properly and safely 🙂 Have fun!

Update
After having run my migrations, it turns out it dumps the migration files in your doc root! After a bit of faffing about, I got the config working, so here is the full cli-config.php:

<?php

use Doctrine\DBAL\Migrations\Configuration\Configuration;
use Doctrine\DBAL\Migrations\Tools\Console\Command\DiffCommand;
use Doctrine\DBAL\Migrations\Tools\Console\Command\ExecuteCommand;
use Doctrine\DBAL\Migrations\Tools\Console\Command\GenerateCommand;
use Doctrine\DBAL\Migrations\Tools\Console\Command\MigrateCommand;
use Doctrine\DBAL\Migrations\Tools\Console\Command\StatusCommand;
use Doctrine\DBAL\Migrations\Tools\Console\Command\VersionCommand;
use Doctrine\ORM\Tools\Console\ConsoleRunner;
use XYZ\XYZService;
use XYZ\Test\XYZTesting;

$container = XYZTesting::getContainer();

/* Edit these details to suit

$container['db.credentials'] = array(
    'driver' => 'pdo_mysql',
    'dbname' => 'XYZ',
    'user' => 'dbuser',
    'password' => '123',
];
 */

// Fetch the entity Manager
$svc = new XYZService($container);
$em = $svc->getEntityManager();

// Create the helperset
$helperSet = ConsoleRunner::createHelperSet($em);

/** Migrations setup */

$configuration = new Configuration($em->getConnection());
$configuration->setMigrationsDirectory('migrations');

$diff = new DiffCommand();
$exec = new ExecuteCommand();
$gen = new GenerateCommand();
$migrate = new MigrateCommand();
$status = new StatusCommand();
$ver = new VersionCommand();

$diff->setMigrationConfiguration($configuration);


$cli = ConsoleRunner::createApplication($helperSet,[
    $diff, $exec, $gen, $migrate, $status, $ver
]);

return $cli->run();

puPHPet Box MySQL backup

As you all know by now, I love puPHPet. And I like tinkering with things. Which of course leads to the occasional breaking of things. And then I realised, okay, your sites files are being mounted into this virtual machine, what about the databases though?

The great thing about puPHPet and Vagrant is you can build a full server up from scratch in minutes. The one downside is that it cant be smart enough to uninstall things. If you add lines to your config.yaml it will install something, but removing those lines means puPHPet won’t even know it was there, so how would it know to uninstall it? The exceptions of course being the ones which have their own config.yaml entry, with an install: ‘1’ or install: ‘0’

Anyway, I generated me a new config and was about to blitz my old VM when the usual gut instinct checks kicked in, and the DB sprung to mind immediately. So I made a quick backupdbs.sh shell script (courtesy of a StackOverflow post) which makes individual sql files for each of your DB’s. I decided I would keep the script in my mounted sites folder in a bin directory. So hence it lives in /var/www/bin/backupdbs.sh

#!/bin/bash

USER="root"
PASSWORD="YOURPASSWORDHERE"

databases=`mysql -u $USER -p$PASSWORD -e "SHOW DATABASES;" | tr -d "| " | grep -v Database`

for db in $databases; do
    if [[ "$db" != "information_schema" ]] && [[ "$db" != "performance_schema" ]] && [[ "$db" != "mysql" ]] && [[ "$db" != _* ]] ; then
        echo "Dumping database: $db"
        mysqldump -u $USER -p$PASSWORD --databases $db > `date +%Y%m%d`.$db.sql
    fi
done

To back everything up, vagrant ssh into the machine, then (you’ll probably need to switch user to www-data) do the following:

sudo su www-data
cd /var/www/bin
./backupdbs.sh

Now you’ll find all your DB’s SQL files date stamped in the folder! You can continue to break, tweak, fix, and run your VM to your hearts content 😀

Export WordPress post URLs

I’m making some Apache redirects from an old domain to a new domain. I need the old domains blog posts to redirect to the new domain. So I need the WordPress permalinks.

In MySQL, you can say the following to get your permalinks.

SELECT wpp.post_title, wpp.guid,wpp.post_date,
CONCAT(wpo_su.option_value,
REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(
    wpo.option_value,'%year%', date_format(wpp.post_date,'%Y'))
    ,'%monthnum%',date_format(wpp.post_date,'%m'))
    ,'%day%',date_format(wpp.post_date,'%d'))
    ,'%postname%',wpp.post_name )
    ,'%category%',wpc.slug )
    ,'%post_id%', wpp.id)
) as permalink
FROM wp_posts wpp
INNER JOIN wp_options wpo ON wpo.option_name='permalink_structure'
INNER JOIN wp_options wpo_su ON wpo_su.option_name='siteurl'
INNER JOIN (
    SELECT wtr.object_id ID, max(wpt.slug) slug
    FROM wp_term_relationships wtr
    INNER JOIN wp_term_taxonomy wtt ON wtt.term_taxonomy_id=wtr.term_taxonomy_id AND wtt.taxonomy='category'
    INNER JOIN wp_terms wpt on wpt.term_id=wtt.term_id
    GROUP BY  wtr.object_id
) wpc ON wpc.ID=wpp.ID
WHERE wpp.post_type = 'post'
AND wpp.post_status = 'publish'
ORDER BY wpp.post_date DESC

Tail every MySQL query in your terminal

It’s nice to see what the actual queries that run are on your site. Its pretty simple to set up! In your home folder, edit your .bashrc (or equivalent) and slap in the following:

alias tmysql='mysql -u root -e '\''SET GLOBAL general_log = "ON"'\'';sudo tail -f /var/log/mysql/general.log;mysql -u root -e '\''SET GLOBAL general_log = "OFF"'\''; sudo rm /var/log/mysql/general.log; echo '\''\nExiting Tail, Log Deleted.'\'';

Tweaking paths where required, obviously. Now close and open your terminal again, and when you type tmysql, you’ll start tailing MySQL! Sample output below!

          453 Query    SELECT `main_table`.* FROM `enterprise_customersegment_segment` AS `main_table`
          453 Query    SELECT `op_imagecdn_cache`.* FROM `op_imagecdn_cache` WHERE (`op_imagecdn_cache`.`url`='http:blah.com')
          453 Query    SELECT `main_table`.* FROM `madskull_autoblocks` AS `main_table` WHERE (identifier = 'trust-banners')
          453 Query    SELECT `madskull_autoblocks_definitions`.* FROM `madskull_autoblocks_definitions` WHERE (`madskull_autoblocks_definitions`.`id`='39')

MySQL socket DSN connection syntax

Usually I just bung in 127 or localhost in order to connect up to my DB. However, if you aren’t allowing remote connections, then running on a unix socket is much faster and skips all the networking overhead involved with TCP/IP.

So! To use a socket, you use a DSN for the connection. I’ve done this a few times, but I keep forgetting the syntax, sop here it is, solely for my own memory’s sake:

mysql:unix_socket=/tmp/mysql.sock;dbname=my_db_name_here

Lost connection to MySQL server at ‘reading initial communication packet’

Here’s one you might have had. “Lost connection to MySQL server at ‘reading initial communication packet’ “.

If  you get this, then I assume that you are connecting to another machine on the network, a virtual machine, or a remote server.

It’s a pretty simple fix. Edit your /etc/mysql/my.cnf (path may differ depending on your setup), and change the line that says bind-address = 127.0.0.1 to:

bind-address = 0.0.0.0

Save it, then restart mysql.

sudo service mysql restart

Job done!