Replace Symfony reverse proxy with Varnish

In my last post I've presented how to start caching pages with the reverse proxy shipped with Symfony Standard Edition. But as it is mentioned in the documentation the built in PHP proxy is not as fast as Varnish for example. So today I will do a small test with Varnish instead of the Symfony reverse proxy (AppCache).

First install Varnish on virtual machine with Ubuntu (I am using https://box.scotch.io ):

    sudo apt-get install varnish -y

Edit Varnish configuration file found at: /etc/varnish/default.vcl. I have a virtual host called myshop.dev, I will put that as host.

Example of Varnish configuration file for Symfony:

backend default {
    .host = "myshop.dev";
    .port = "80";
}

sub vcl_fetch {
    /* By default, Varnish3 ignores Cache-Control: no-cache and private
      https://www.varnish-cache.org/docs/3.0/tutorial/increasing_your_hitrate.html#cache-control
     */
    if (beresp.http.Cache-Control ~ "private" ||
        beresp.http.Cache-Control ~ "no-cache" ||
        beresp.http.Cache-Control ~ "no-store"
    ) {
    return (hit_for_pass);
    }

}
sub vcl_recv {
     unset req.http.Forwarded;

   if (req.http.Cookie) {
        set req.http.Cookie = ";" + req.http.Cookie;
        set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
        set req.http.Cookie = regsuball(req.http.Cookie, ";(PHPSESSID)=", "; \1=");
        set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");
        if (req.http.Cookie == "") {
            unset req.http.Cookie;
        }
    }


    // Add a Surrogate-Capability header to announce ESI support.
    set req.http.Surrogate-Capability = "abc=ESI/1.0";
}

sub vcl_fetch {
    if (beresp.http.Surrogate-Control ~ "ESI/1.0") {
        unset beresp.http.Surrogate-Control;
        set beresp.do_esi = true;
    }
}

sub vcl_deliver {
  if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS
    set resp.http.X-Cache = "HIT";
  } else {
    set resp.http.X-Cache = "MISS";
  }
  # show how ofthe the object created a hit so far (reset on miss)
  set resp.http.X-Cache-Hits = obj.hits;
}

Configure Symfony config.yml file:

framework:
    esi:   { enabled: true }
    trusted_proxies: [127.0.0.1]
    fragments:  { path: /_fragment }

With these configuration in place you can open the browser and  call your page via Varnish on port 6081:

http://myshop.dev:6081/

If is not working you can try to restart Varnish service:  sudo service varnish restart


One source of information:  http://by-examples.net/2014/12/19/speeding-up-symfony-with-varnish.html

Introduction to Symfony HTTP Cache

You can start using the HTTP Gateway Cache that is shipped with Symfony by editing app.php file:

    $kernel = new AppKernel('prod', true);
    $kernel->loadClassCache();
    // wrap the default AppKernel with the AppCache one
    $kernel = new AppCache($kernel);

I've put the debug mode to be able to see the X-Symfony-Cache header in Firebug. You may want to delete the cache for prod environment before  refreshing the page. When loading the page, you can see in Firebug the Response headers:

    Cache-Control       no-cache
    X-Symfony-Cache   GET /: miss

So we do not have any Cache control directives set and our reverse proxy says it cannot find the path "/" in the list of cached information.

Let's make the response cacheable using Cache Expiration strategy.

In controller:

    $response =  $this->render('default/index.html.twig');
    $response->setSharedMaxAge(30);

    // (optional) set a custom Cache-Control directive
    $response->headers->addCacheControlDirective('must-revalidate', true);
    return $response;

Delete the cache and reload the page:

 First time: loading time is 4.28s according to FireBug
 Response headers:

     Age 0
     X-Symfony-Cache GET / : miss, store

 Reload second time: loading time is 82ms

     Age 9
     X-Symfony-Cache GET/ : fresh

 Reload third time loading time is 1.86 s

    Age 0
    X-Symfony-Cache GET / : stale, invalid, store

Things work as expected, the response time lowered to 82ms.


Now I will be testing the HTTP Cache Validation strategy.

     /**
     * @Route("/", name="homepage_symfony")
     */    
    public function indexAction(Request $request)
    {
        $text = "Welcome!";
        $response = new Response(); 
        $response->setETag(md5($text)); 
        $response->setPublic(); // make sure the response is public/cacheable
        if ($response->isNotModified($request)) { 
            return $response; 
        };

        $response->setContent($this->renderView('default/index.html.twig', array('content'         => $text)));
        return $response;
   }


First delete cache. Than reload page in browser:

First time:
X-Symfony-Cache  GET /: miss, store

Second time:
X-Symfony-Cache  GET /: stale, valid, store

At this moment there is not a big improvement as we are not avoiding any complex logic, just that instead of rendering the page with Twig we are serving it from cache.


Expiration and Validation 

You can of course use both validation and expiration within the same Response. As expiration wins over validation, you can easily benefit from the best of both worlds. In other words, by using both expiration and validation, you can instruct the cache to serve the cached content, while checking back at some interval (the expiration) to verify that the content is still valid. (from Symfony doc)

I will add setSharedMaxAge(30)to the controller resulting:

public function indexAction(Request $request)
{
    $text = "Bun venit pe pagina";
    $response = new Response();
    $response->setETag(md5($text));
    $response->setPublic(); // make sure the response is public/cacheable
    $response->setSharedMaxAge(30);
    if ($response->isNotModified($request)) {  
          return $response;
    }; 
   $response->setContent($this->renderView('default/index.html.twig', array('content' => $text)));
    return $response;
}

For 30 seconds the cache will be fresh and the page will be served from cache. After that using the ETag the freshness of the cache will checked and if it is still fresh the controller will return a 304 status with an empty content to the HTTP Cache which will serve again the response from cache.

First time loading the page:

Age               0
Cache-Control  public, s-maxage=30
Etag    "9c8751de832e5a472655e87731c49419-gzip"
X-Symfony-Cache  GET /: miss, store

Second Time:

Age   5
X-Symfony-Cache   GET /: fresh

After more than 30s

Age     5 
X-Symfony-Cache GET /: stale, valid, store

Symfony magic: generated PHP files

1.  bootstrap.php.cache

In versions 3.x is found at var/bootstrap.php.cache before was in app/ directory.

Loading all of classes from separate files on each request can result in some overhead. To reduce this overhead, the Symfony Standard Edition provides a script to generate a so-called bootstrap file, consisting of multiple classes definitions in a single file. By including this file (which contains a copy of many of the core classes), Symfony no longer needs to include any of the source files containing those classes. This will reduce disc IO quite a bit.

If you're using the Symfony Standard Edition, then you're probably already using the bootstrap file. To be sure, open your front controller (usually app.php) and check to make sure that the following line exists:

include_once __DIR__.'/../var/bootstrap.php.cache'

Note that there are two disadvantages when using a bootstrap file:

 - the file needs to be regenerated whenever any of the original sources change (i.e. when you update the Symfony source or vendor libraries)
 - when debugging, your breakpoints are never reached because the code is executed from bootstrap.php.cache. I wrote in this article about how to handle this.

If you're using the Symfony Standard Edition, the bootstrap file is automatically rebuilt after updating the vendor libraries via the composer install command.

See: http://symfony.com/doc/master/performance.html#use-bootstrap-files

2.  classes.php and classes.map

From Alan Storm blog:  "The generation of classes.php is a bit complicated. First, this file is not generated as part of Symfony's cache warmup. You can call

php app/console cache:clear

all you like, but that won't regenerate classes.php. Instead, you need to load an actual application page, or otherwise boot your application kernel.

#File: vendor/symfony/symfony/src/Symfony/Component/HttpKernel/Kernel.php

public function boot()

{

    //...

    if ($this->loadClassCache) {

        $this->doLoadClassCache($this->loadClassCache[0], $this->loadClassCache[1]);

    }

    //...

}    

It's the call to doLoadClassCache which triggers a call to ClassCollectionLoader::load

#File: vendor/symfony/symfony/src/Symfony/Component/HttpKernel/Kernel.php

protected function doLoadClassCache($name, $extension)

{

    if (!$this->booted && is_file($this->getCacheDir().'/classes.map')) {

        ClassCollectionLoader::load(include($this->getCacheDir().'/classes.map'), $this->getCacheDir(), $name, $this->debug, false, $extension);

    }

}

The work of generating classes.php is done in ClassCollectionLoader::load. Simple enough — except you need to pass in a list of classes you want to combine. These classes come from the classes.map file.

app/cache/dev/classes.map

app/cache/prod/classes.map

Of course, now our question is "where does classes.map come from?". The classes.map file is generated at the same time as the app container file, right after the call to $container->compile() in initializeContainer.

#File: vendor/symfony/symfony/src/Symfony/Component/HttpKernel/Kernel.php

protected function initializeContainer()

{

    //...

    if (!$cache->isFresh()) {

        //...

        $container->compile();

        //...

    }

    //...

}

3.  The Dependency Injection Container(DIC) class file

In a simplified way you can think about the  DIC as an array, having as keys the names of the services, and as values, anonymous functions that create a new instance of the needed class ( have a look at Pimple ). Symfony DIC is more complex, it allows services to interact using tagged services, and this interaction is implemented using some classes called Compiler Passes. Compiler Passes manipulate the services, and they are executed when the DIC is compiled.

In Symfony you do not edit a class to register your services directly into an array, instead you edit different configuration files (YML, XML). Each extra bundle that you install may add its own services that need to be registered. Also Compiler Passes need to be executed. 

All these haveto be "compiled" in order to result a PHP class file containing a big array of services and anonymous functions that instantiate new objects. The compilation process has loaded the services from the configuration, extensions and the compiler passes, the result is dumped so that the cache can be used next time. The dumped version is then used during subsequent requests as it is more efficient. The generated file can be found in the following locations for the dev and prod environments:

var/cache/dev/appDevDebugProjectContainer.php
var/cache/prod/appProdProjectContainer.php

If you're creating new services in Symfony, you'll often need to re-generate you container file. Doing so is as simple as clearing your Symfony cache — the Symfony kernel will regenerate these files automatically. If you're curious, the code that does this starts here

#File: vendor/symfony/symfony/src/Symfony/Component/HttpKernel/Kernel.php

protected function initializeContainer()

{

    $class = $this->getContainerClass();

    $cache = new ConfigCache($this->getCacheDir().'/'.$class.'.php', $this->debug);

    $fresh = true;

    if (!$cache->isFresh()) {

        $container = $this->buildContainer();

        $container->compile();

        $this->dumpContainer($cache, $container, $class, $this->getContainerBaseClass());

        $fresh = false;

    }

    require_once $cache;

    $this->container = new $class();

    $this->container->set('kernel', $this);

    if (!$fresh && $this->container->has('cache_warmer')) {

        $this->container->get('cache_warmer')->warmUp($this->container->getParameter('kernel.cache_dir'));

    }

}

4.  Twig templates

Before inspecting how Twig files are generated, you can read this article from Fabien Potencier on why do we need Twig in first place.

How does Twig work?

   1. Load the template: If the template is already compiled, load it and go to the evaluation step, otherwise:

        1.1. First, the lexer tokenizes the template source code into small pieces for easier processing;

        1.2. Then, the parser converts the token stream into a meaningful tree of nodes (the Abstract Syntax Tree);

        1.3. Eventually, the compiler transforms the AST into PHP code. This step is generating PHP  c    classes that extends Twig_Template class and contain a function called "doDisplay()".

The files are found under cache/twig.

    2. Evaluate the template: It basically means calling the display() method of the compiled template and passing it the context.

Compilation is a costly process, so the result is cached in the directory defined by this configuration option. You can set option "cache" to null to disable Twig template compilation. However, this is not recommended; not even in the dev environment, because the auto_reload option ensures that cached templates which have changed get compiled again.

auto_reload

If true, whenever a template is rendered, Symfony checks first if its source code has changed since it was compiled. If it has changed, the template is compiled again automatically. (source: http://symfony.com/doc/current/reference/configuration/twig.html).

5. Routes

In Symfony you can declare routes in different ways YAML, XML, Annotation, all these are in the end compiled to PHP. You can have a look at the file appDevUrlMatcher.php under /app/cache directory for Symfony 2. In dev environment this file regenerated if you are doing changes to routes, in prod environment you need to clear the cache in order to regenerate it. Have a look at this presentation for more info: http://davidbu.ch/slides/2015-12-03-symfony-routing.html

Consuming a SOAP web service with PHP

I started programming with PHP at the beginning of 2015, and when is about web services, everything I've read is about RESTful services.

But guess what, is 2016 I will have to interact with a SOAP web service. Doing a research I've discovered this very informative article from Benjamin Eberlei:

http://www.whitewashing.de/2014/01/31/soap_and_php_in_2014.html

Take your time to read it.

OK, now that we know what are we talking about, short example on how to consume a SOAP service. I will use a free SOAP web services for testing:  http://www.webservicex.net

I will test a simple request to this particular service: http://www.webservicex.net/New/Home/ServiceDetail/19 called Periodic table.

The link to the WSDL file is: http://www.webservicex.net/periodictable.asmx?WSDL

With this link and using the SoapClient class from PHP I will write:

<?php

$url = "http://www.webservicex.net/periodictable.asmx?WSDL";
$client = new SoapClient($url);

$fcs = $client->__getFunctions();
$types = $client->__getTypes();
var_dump($fcs);
//var_dump($types);

I do not really need to call __getFunctions() and __getTypes() but is useful for learning purpose.

The output should look like this:

array(8) {
  [3]=>
  string(71) "GetElementSymbolResponse GetElementSymbol(GetElementSymbol $parameters)

Where the types of data GetElementSymbolResponse and GetElementSymbol are data types defined in the WSDL file given as parameter to SoapClient constructor.
Let's look to the GetElementSymbol which needs to be passed to the SOAP server.

<s:element name="GetElementSymbol"> 
    <s:complexType>
        <s:sequence>
            <s:element minOccurs="0" maxOccurs="1" name="ElementName" type="s:string"/>
        </s:sequence>
    </s:complexType>
</s:element>

It expect an item called "ElementName". Make sense, let's say I want to find out the symbol for Iron in the periodic table.
Back to my PHP program:

<?php

$res = $client->GetElementSymbol(array('ElementName' => 'Iron'));
//var_dump($res);
echo "Symbol for Iron: ";
echo $res->GetElementSymbolResult;

Evrika! The server answered to our question, the symbol for Iron is "Fe". If you uncomment //var_dump($res); you can see that the response is an object from (stdClass) with one property.

Using the Symfony Validator component as a standalone library

Sometimes I need to build some scripts without using a full framework like Symfony. The good news is that you can use individual components from Symfony. Most often I will be using the HTTPFoundation component, but for today I want to build a CLI script so no need for HTTPFoundation.
I  will be writing a small command line script to read and validate a  CSV file. For the  validation part I will be using the Symfony Validator Component,

Source of inspiration: https://blog.tinned-software.net/using-the-symfony-validator-as-a-standalone-component/


First step, install Symfony Validator using Composer:

                          composer require symfony/validator

and require the autoloader created by Composer.

<?php

require_once 'vendor/autoload.php';
require_once 'Loader.php';

use Symfony\Component\Validator\Validation;

$file = "someFile.csv";


$outputFile = "errorsLog" . "_" . time() . ".txt";
  
$validator = Validation::createValidatorBuilder()
 ->addMethodMapping('loadValidatorMetadata')
 ->getValidator()
;

$loader = new Loader($file, $outputFile, $validator);
$loader->load();

I will pass to my Loader class constructor the CSV file to be read, the file where I want to save the output and an instance of the Symfony validator.

The CSV file is read line by line, and from each line I will be creating a Row object. The validator will validate the Row object against the rules (constraints).
In the Row class I will add the method "loadValidatorMetada" mentioned when instantiating the Validator. I've added several validations for better examplification:

<?php

use Symfony\Component\Validator\Constraints as Assert;
use Symfony\Component\Validator\Mapping\ClassMetadata;


class Row
{
    private $id;         
    private $name;   
    private $personalNumber;    
    private $CountryCode;  

    public function __construct($rowNumber, $data)
    {
        $this->id = $rowNumber;
        $this->name= $data[0];
        $this->personalNumber = $data[2];
        $this->countryCode = $data[1];
    }
 
 
    /**
     * This method is where you define your validation rules.
    */
    public static function loadValidatorMetadata(ClassMetadata $metadata)
    {
        //Name 

        $metadata->addPropertyConstraint('name', new Assert\NotBlank());

        //Personal number
        $metadata->addPropertyConstraint('personalNumber', new Assert\NotBlank());

        $metadata->addPropertyConstraint('personalNumber', new Assert\Type(array(
            'type'    => 'digit',
            'message' => 'The value {{ value }} is not a valid personal number.',
         )));
  
         //Country code
         $metadata->addPropertyConstraint('countryCode', new Assert\NotBlank());
         $metadata->addPropertyConstraint('countryCode', new Assert\Choice(array(
            'choices' => array('DE', 'AT', 'LU', 'ES', 'FR', 'BE', 'NO', 'SI', 'SE', 'IT', 'DK'),
            'message' => '{{ value }} is not a valid country code!',
          )));
  
     }
}

When validating a Row object against these constraints an array of Errors will be returned by the Validator. Below is the Loader class.

<?php

require_once 'Row.php';


class Loader
{
 private $fileName;
 private $validator;
 private $outputFile;
 
 public function __construct($fileName, $outputFile, $validator)
 {
  $this->fileName = $fileName;
  $this->outputFile = $outputFile;
  $this->validator = $validator;

 }
 
 public function load()
 {
  $file_handle = fopen($this->fileName, "r");
  $fileOutputHandle = fopen($this->outputFile, "w");
  
  /* 
   * Keep track of the current row in .csv file
   */
  $rowNumber=1;
  
  /* 
   * Read the file line by line
   */
  while (!feof($file_handle) ) {

   $line_of_text = fgetcsv($file_handle, 0);
   
   if ($rowNumber === 1) {
    
      /*
       * Ignore the first row from file as it contains headers
       */
     
   } else {
       $row = new Row($rowNumber, $line_of_text);    
       $this->validateRow($row, $fileOutputHandle); 
   }
   
   $rowNumber++;
  }
  fclose($file_handle);
  fclose($fileOutputHandle);
 }
 
 
 public function validateRow(Row $row, $fileOutputHandle)
 {
  $errors = $this->validator->validate($row);
  
  foreach ($errors as $error) {
   $errorMsg = "At row:" 
      . $error->getRoot()->getId() 
      . "- Property: " 
      . $error->getPropertyPath()
      . ' - Message: ' 
      . $error->getMessage()
      . "\n";
   
   fwrite($fileOutputHandle, $errorMsg);

  }
 }
}

Symfony/Sonata ACL - search for the object owner

Lately I am wrestling with the ACL in a project using Symfony with Sonata Admin Bundle and Sonata User Bundle. When editing objects from the admin panel offered by Sonata Admin Bundle things go smoothly, but the issues is how to set ACL permissions for objects I create in my own controllers.

Situation:

 An Admin1 user --> creates a Regular1 user --> which creates Objects and save them to database.

Problem:

Make Regular1 user and Admin1 user  owners of the newly created Object.

Note: Admin user is not ROLE_SUPER_ADMIN, just a custom role I have in my app

Solution:

First step, in the createAction controller method, after persisting my object, set the current user (Regular user) as owner of the object.

<?php 

    public function createAction(Request $request)
    {

        .....
        $em->persist($myObject);
        $em->flush();
            
        // retrieve services and get current user
        $adminSecurityHandler = $this->container->get('sonata.admin.security.handler');
        $modelAdmin = $this->container->get('admin.sites');
        $user = $this->getUser();

        $securityIdentity = UserSecurityIdentity::fromAccount($user);

        $objectIdentity = ObjectIdentity::fromDomainObject($myObject);
        $acl = $adminSecurityHandler->getObjectAcl($objectIdentity);
        
        if (is_null($acl)) {
            $acl = $adminSecurityHandler->createAcl($objectIdentity);
        }
        $adminSecurityHandler->addObjectClassAces($acl, $adminSecurityHandler->buildSecurityInformation($modelAdmin));
        $adminSecurityHandler->addObjectOwner($acl,$securityIdentity);  // set current user as owner in ACL

        $adminSecurityHandler->updateAcl($acl);

Second part is to search for the Admin user, owner of the regular user (which is currently logged).
Looking into the list of ACE associated to the object identity, I am searching for the one with  Mask equal to 128 (owner mask).

<?php

        //search for the owner (admin user) of the current user and give him privileges on the "myObject" object
        $userObjectIdentity = ObjectIdentity::fromDomainObject($user);
        $userObjectACL = $adminSecurityHandler->getObjectAcl($userObjectIdentity);

        $aces= $userObjectACL->getObjectAces();    
       
        /*
         *  $aces is an array containing ACEs, objects from this class: 
         *  http://api.symfony.com/2.7/Symfony/Component/Security/Acl/Domain/Entry.html
         */
        foreach($aces as $ace){
            if(128 === $ace->getMask()){
                $adminSecurityIdentity = $ace->getSecurityIdentity();

                $adminSecurityHandler->addObjectOwner($acl,$adminSecurityIdentity);  
                $adminSecurityHandler->updateAcl($acl);

            }
        }

Symfony DIC: setter injection

Most of the time, when defining a service I inject into constructor parameters or other services. But it is possible to use the Symfony DIC to inject parameters/services into setter methods:

     newsletter_manager:
         class:     NewsletterManager
         calls:
             - [setMailer, ['@my_mailer']]

http://symfony.com/doc/current/components/dependency_injection/types.html

Where:

setMailer  - is the name of the method
['@my_mailer'] - is the name of the service being injected

Practical example with Sonata

If you are using Sonata Admin Bundle and Sonata User Bundle, you may want to access the Entity Manager for some reasons. For this we will inject the service ''@doctrine.orm.entity_manager" using a setter method.

I will create a bundle named ApplicationSonataAdminUserBundle where I will place my new Admin class. In the project's services.yml I will add the new service definition:


sonata.user.admin.user:
        class: Application\Sonata\UserBundle\Admin\Model\UserAdmin
        arguments: [~,"%sonata.user.admin.user.entity%","SonataAdminBundle:CRUD"]
        calls:
            - [setUserManager, ['@fos_user.user_manager']]
            - [setTranslationDomain, ['%sonata.user.admin.user.translation_domain%']]
             - [setEntityManager, ["@doctrine.orm.entity_manager"]]
        tags:
            - { name: sonata.admin, manager_type: orm, group: "sonata_user", label: "users", label_catalogue: "SonataUserBundle",label_translator_strategy: "sonata.admin.label.strategy.underscore",icon: "<![CDATA[<i class='fa fa-users'></i>]]>" }

And of course in the class I will add the setter method "setAuthorizationChecker":

use Sonata\AdminBundle\Admin\AbstractAdmin;

class UserAdmin extends AbstractAdmin
{

    protected $entityManager;

    public function setEntityManager(EntityManagerInterface $entityManager)
    {
        $this->entityManager = $entityManager;
    }

...
}