Log analysis with Kinesis – Lambda – ElasticSearch – Kibana pipeline

ELK (ElasticSearch – LogStash – Kibana) is my favourite stack for managing and analysing server logs for years. Recently, when deploying a log management module for a small company on Amazon Web Service environment, I found that AWS Kinesis can help me achieve the same target with much less effort of server management and configuration. The only server the company need to manage is a t2.micro one, which runs a Kibana instance and a Nginx reverse proxy. The other parts of the stack runs only with AWS services. I’ll describe what I used in this post.

An overview

I built a module to monitor nginx log files on several EC2 instances which run a web application behind a load balancers. The diagram below tells you how it works. In short, it includes these components:

  • An AWS ElasticSearch Service instance which stores the log data
  • A Kibana instance allows user analyse and visualise ES data
  • A Kinesis Firehose stream which delivers transformed log data to ElasticSearch as well as back up the raw log data to S3 storage
  • A Lambda function which transforms log lines into JSON structure
  • An S3 bucket store the raw log data
  • Kinesis agents run on EC2 instances which read Nginx access logs and post to the data stream
Kinesis Lamda ElasticSearch Kibana pipeline diagram

Setting up components

Nginx log format and the AWS Kinesis agents

Nginx instances was set up to use this format for their access logs:

log_format  main  '$time_local $remote_addr $remote_user $request_method $request_uri '
                  '$server_addr $server_port $host '
                  '"$http_user_agent" "$http_referer" '
                  '$status $bytes_sent $request_time '
                  '"$http_x_forwarded_for" $http_x_forwarded_port $http_x_forwarded_proto';

The access log files was configured to be rotated daily.

AWS Kinesis agents were installed following this guide. The filePattern value is the location of nginx access log. The deliveryStream is the name of Firehose stream, which would be created later.

ElasticSearch and Kibana

Amazon provides a good solution for those who don’t want to manage a complicated ElasticSearch cluster: AWS ElasticSearch Service. There are several articles on the Internet comparing different deployment approaches for ElasticSearch cluster. While self-hosted instances requires huge workload of maintenance,  dedicated cloud ElasticSearch services are more expensive than AWS ES. After creating a ES domain, we have 2 URLs: one for the ElasticSearch RESTful API, one for Kibana interface.

I planned to store access log events in one index per month. Each index would be named as “access-log-YYYY-MM”. The timestamp part would be formatted automatically by Kinesis. I also created an index template on ElasticSearch to define the data type of fields in the events. The structure of each event was decided by the lambda transformation function, which will be addressed later in this post. The final template is as follow:

{
  "template": "access-log-*",
  "mappings": {
    "log": {
      "properties": {
        "geoip": {
          "dynamic": true,
          "properties": {
            "location": {
              "type": "geo_point"
            }
          }
        },
        "method": {
          "type": "keyword"
        },
        "bytesSent": {
          "type": "integer"
        },
        "originalIP": {
          "type": "ip"
        },
        "forwardedProtocol": {
          "type": "keyword"
        },
        "requestTime": {
          "type": "half_float"
        },
        "port": {
          "type": "keyword"
        },
        "clientIP": {
          "type": "ip"
        },
        "browser": {
          "dynamic": true,
          "properties": {
            "os": {
              "dynamic": true,
              "properties": {
                "family": {
                  "type": "keyword"
                }
              }
            },
            "family": {
              "type": "keyword"
            },
            "device": {
              "dynamic": true,
              "properties": {
                "family": {
                  "type": "keyword"
                }
              }
            }
          }
        },
        "host": {
          "type": "keyword"
        },
        "serverIP": {
          "type": "ip"
        },
        "device": {
          "dynamic": true,
          "properties": {
            "type": {
              "type": "keyword"
            }
          }
        },
        "forwardedPort": {
          "type": "keyword"
        },
        "status": {
          "type": "keyword"
        }
      }
    }
  }
}

 

Kinesis Firehose stream

To deliver logs from Kinesis agent to ES, I created a Kinesis Firehose stream. These are some important notes:

  • The stream name would be used in the agent.jsonfile of the Kinesis agent on each EC2 instances.
  • The index name should match with the prefix in the ES index template. Here, I chose “access-log” for index name.
  • I set the index rotation to monthly (OneMonth) because I just want to keep the logs of the last 30 days. Old indices will be deleted to free up space of the ES domain.
  • I chose “all documents” as the S3 backup mode because I want to archive all raw log lines.
  • To transform a log line to a JSON structure, I enabled “data transformation” feature and chose a lambda function as the transformer.

After the Kinesis Firehose stream was created, I had a data pipeline with the following steps:

  1. Log lines are collected by Kinesis agents in EC2 instances.
  2. Log lines are submitted to the Firehose stream, each line becomes an event with a unique ID and the data set to the original log line.
  3. When the total data of events in the stream becomes larger than the S3 buffer size, the data of current events in the buffered will be written to the S3 bucket.
  4. When the total data of events in the stream becomes larger than the ES buffer size, the configured lambda function will be invoked. All current events in the buffer will be passed to the function handler as a collection of items. Each items has a id and a data field which is base64-encoded from the log line. The results of the lambda function, which is also base64-encoded, will replace the input events in the stream and be forwarded to the next steps.
  5. At this steps, events data are decoded to JSON structure and inserted to the ES index as a batch insertion.
  6. Users use Kibana to access the ES database, do queries, analyses and visualises the data.

Lambda transformation function

This component plays an important role in the pipeline. The transformation function convert the unstructured data (lines of text) into semi-structured data (JSON objects). It adds more data to the event such as geoIP information from the client IP, information about the client OS, devices and browsers, … It cast the strings to numbers for specific fields, too. For this configuration, I needed the support of some third-party libraries of NodeJS. Therefore, I created a deployment package for NodeJS 6, uploaded it to an S3 bucket and used it as the lambda function code. This is the body of my lambda function:

'use  strict';
console.log('Loading function');

let patterns = require('node-grok').loadDefaultSync();
let geoip = require('geoip-lite');
let device = require('device');
let countries = require('countryjs');
let moment = require('moment');

let grokPattern = "%{HTTPDATE:timestamp} %{IP:clientIP} %{NOTSPACE:username} %{WORD:method} %{NOTSPACE:uri} %{IP:serverIP} %{NUMBER:port} %{NOTSPACE:host} %{QUOTEDSTRING:userAgent} %{NOTSPACE:referer} %{NUMBER:status} %{NUMBER:bytesSent} %{NUMBER:requestTime} %{QUOTEDSTRING:forwardedFor} (?<forwardedPort>-|%{NUMBER}) (?<forwardedProtocol>-|%{NOTSPACE})";
let pattern = patterns.createPattern(grokPattern);

/**
 * @see http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/x-forwarded-headers.html#x-forwarded-for
 * @param strVal
 * @return string|null
 */
const getForwardedIp = (strVal) => {
    let ips = strVal.split(",").map((x) => x.trim());
    let originalIP = ips[ips.length - 1];

    if (originalIP === "-") {
        return null;
    } else {
        return originalIP;
    }
};

let transform = (event, context, callback) => {
    let success = 0; // Number of valid entries found
    let failure = 0; // Number of invalid entries found

    /* Process the list of records and transform them */
    const output = event.records.map((record) => {
        const entry = (Buffer.from(record.data, 'base64')).toString('utf8');
        const match = pattern.parseSync(entry);

        let result = {
            message: entry,
            tags: []
        };
        if (match) {
            /* Prepare JSON version from Apache log data */

            result = Object.assign(result, match);
            result.tags.push('grok_parsed_success');

            // trim the double quotes from the user agent, referer and forwarded for
            result.userAgent = result.userAgent.substr(1, result.userAgent.length - 2);
            result.referer = result.referer.substr(1, result.referer.length - 2);
            result.forwardedFor = result.forwardedFor.substr(1, result.forwardedFor.length - 2);

            // convert strings to numbers
            result.bytesSent = +result.bytesSent;
            result.requestTime = +result.requestTime;

            // get the original IP
            let forwardedIp = getForwardedIp(result.forwardedFor);
            if (!forwardedIp) {
                result.originalIP = result.clientIP;
            } else {
                result.originalIP = forwardedIp;
            }

            // geoip lookup
            let resolvedIp = geoip.lookup(result.originalIP);
            result.geoip = resolvedIp;
            if (resolvedIp) {
                result.tags.push('geoip_looked_up_success');
                result.geoip.countryName = countries.name(result.geoip.country);

                // the GeoJSON format has the longitude before the latitude in the array, we must change
                // geoip.ll into geoip.location object
                result.geoip.location = {
                    lat: result.geoip.ll[0],
                    lon: result.geoip.ll[1]
                };
                delete result.geoip.ll;
            } else {
                result.tags.push('geoip_looked_up_fail');
            }

            // useragent parser
            result.device = device(result.userAgent, {parseUserAgent: true});
            result.browser = result.device.parser.useragent;
            delete  result.device.parser;

            // add @timestamp field
            result["@timestamp"] = moment(result.timestamp, 'DD/MMM/YYYY:HH:mm:ss Z', true).toISOString();
            delete result.timestamp;

            const payload = (Buffer.from(JSON.stringify(result), 'utf8')).toString('base64');
            success++;
            return {
                recordId: record.recordId,
                result: 'Ok',
                data: payload,
            };
        } else {
            result.tags.push('grok_parsed_fail');
            /* Failed event, notify the error and leave the record intact */
            failure++;
            return {
                recordId: record.recordId,
                result: 'ProcessingFailed',
                data: record.data,
            };
        }
    });
    console.log(`Processing completed.  Successful records ${success}, Failed records ${failure}.`);
    callback(null, {records: output});
};

exports.handler = transform;

For each event in the input collection, I use the node-grok package to parse the log line like the way I did with the awesome grok library of LogStash. The grok pattern was defined based on the Nginx log format. Some fields which enclosed in double quotes were processed to strip those quotes out.  After that, the original IP address was gotten from the client IP and the content of the X-FORWARDED-FOR HTTP header. That IP was parse by the geoip-lite library with the help of the countryjs library to get the country name from the ISO 3166-1 alpha 2 code. The user-agent string was parsed with the device library to get the information of the client OS, devices as well as the browsers. The @timestamp field of each event was created by parsing the Nginx $time_local time format. At the end, the event data is base64-encoded.

Conclusion

After running the above stack in few days. I see that the events are delayed about 5 minutes before appearing on the Kibana dashboard. Another caveat is that the Kinesis agent does not run on Windows instances. To visualise geographical data on the Kibana tiled map, I need to set up an additional Kibana instance because the Kibana version packaged in AWS ES is modified and does not work with the map layers. I also hide the AWS ES domain from public access and allow users to use the Kibana interface via an Nginx reverse proxy. Compare to my another approach with ElasticSearch, LogStash, Kibana and File beat, the stack mentioned in this post reaches the same level of high-availability with less configuration and maintenance effort. The Kinesis Firehose stream seems to be more sophisticated way to process this type of data.

Reparsing old log files with Logstash 5

Reparse files with logstash 5

In my current research, a set of log files need to be processed by Logstash before sent to ElasticSearch. After a the first run, I realized that the log format was changed once, which made my Logstash configuration fail in processing half of the log lines. I updated the filter and ran Logstash again, hoping that it would reparse the log file with the new configuration. Unfortunately, no line was parsed! This blog post tell about my experience in making Logstash to parse old files without changing files names.

Continue reading “Reparsing old log files with Logstash 5”

Fix the errors of GitHub Metadata and SSL certificate when running Jekyll serve

I decided to play with Jekyll today. It is an amazing static site generator. I ran local server by execute this command:

$> jekyll serve

I could browse my site locally then. When I had updated one of my files, Jekyll stated regenerating the output HTML file and I got this errors on the command line window.

GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.

Error: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

Error: Run jekyll build –trace for more information.

As I am not familiar with Ruby, it took me an hour to investigate and fix this problem. Actually, there are one warning and one error in the message above.

GitHub Metadata warning

GitHub Metadata is a gem required by Jekyll. We need a GitHub personal token to make it work. Follow the excellent guide on GitHub to create a token, copy it to a safe place.

Now, create new environment variable in your system named JEKYLL_GITHUB_TOKEN with the value pasted from the created token. Open a new instance of command line and make sure this command displays correct information (I use Windows machine).

$>echo %JEKYLL_GITHUB_TOKEN%

You can read more at the GitHub Metadata README file.

OpenSSL default certificate error

Please do not do these steps in a production server!

Now, down load the .pem file from https://curl.haxx.se/ca/cacert.pem. Then, create another environment variable called SSL_CERT_FILE pointing to the location of downloaded .pem file.

Open new instance of command line window and run this command, be sure that there is no error now.

$> ruby -rnet/https -e "Net::HTTP.get URI('https://github.com')"

Conclusion

With the steps above, I resolved my problems with Jekyll server command. If you try and find any trouble with them, please let me know by leaving comments at the bottom of the post.

Laravel Active version 3 released!

Laravel Active is rewritten with a absolutely new API and  released with the new major version. This document explains the new API and gives some examples about its usage.

Installation

This version is compatible with the Laravel 5 only. Require it as one of your application dependencies via Composer:

composer require hieu-le/active

Add its service provider to your providers array in the config/app.php

HieuLe\Active\ActiveServiceProvider::class,

If you want to use an alias, register that alias inside the aliases array in the config/app.php

'Active' => HieuLe\Active\Facades\Active::class,

Get the active class based on a condition

Usage:

  • Use the alias: Active::getClassIf($condition, $activeClass = 'active', $inactiveClass = '')
  • Use the application container: app('active')->getClassIf($condition, $activeClass = 'active', $inactiveClass = '')
  • Use the helper function: active_class($condition, $activeClass = 'active', $inactiveClass = '')

Explanation: if the $condition is true, the value of $activeClass is returned, otherwise the value of $inactiveClass is returned. The package comes with several methods to help you create conditions easier. You will get the class as a string as the result of these API.

active_class(true); // 'active'
active_class(false); // ''
active_class(if_uri([$currentUri]), 'selected'); // 'selected'
active_class(if_uri_pattern([$pattern1, $pattern2]), 'active', 'other'); // 'other'

Check the current URI

All of checking methods return boolean result (true or false). You can use the result in the condition of active_class or write your own expression.

Check the whole URI

Usage:

  • Use the alias: Active::checkUri(array $uris)
  • Use the application container: app('active')->checkUri(array $uris)
  • Use the helper function: if_uri(array $uris)

Explanation: you give an array of URI, the package will return true if the current URI is in your array. Remember that an URI does not begin with the slash (/) except the root.

Check the URI with some patterns

Usage:

  • Use the alias: Active::checkUriPattern(array $patterns)
  • Use the application container: app('active')->checkUriPattern(array $patterns)
  • Use the helper function: if_uri_pattern(array $patterns)

Explanation: you give an array of patterns, the package will return true if the current URI matches one of the given pattern. Asterisks may be used in the patterns to indicate wildcards.

Check the query string

Usage:

  • Use the alias: Active::checkQuery($key, $value)
  • Use the application container: app('active')->checkQuery($key, $value)
  • Use the helper function: if_query($key, $value)

Explanation: the package will return true if one of the following condition is true:

  • The current query string contains a parameter named $key with any value and the value of $value is false.
  • The current query string does not contain a parameter named $key and the value of $value is null.
  • The current query string contains a parameter named $key whose value is a string equals to $value.
  • The current query string contains a parameter named $key whose value is an array that contain the $value.
// the current query string is ?x=1&y[0]=a&y[1]=b

if_query('x', null); // true
if_query('x', 1); // true
if_query('x', 2); // false
if_query('y', 'a'); // true
if_query('y', 'c'); // false
if_query('z', null); // false

Check the current route

Check the exact route name

Usage:

  • Use the alias: Active::checkRoute(array $routes)
  • Use the application container: app('active')->checkRoute(array $routes)
  • Use the helper function: if_route(array $routes)

Explanation: you give an array of route names, the package will return true if the name of the current route (which can be null) is in your array.

Check the route name with some patterns

Usage:

  • Use the alias: Active::checkRoutePattern(array $patterns)
  • Use the application container: app('active')->checkRoutePattern(array $patterns)
  • Use the helper function: if_route_pattern(array $patterns)

Explanation: you give an array of patterns, the package will return true if the name of the current route (which can be null) matches one of the given pattern. Asterisks may be used in the patterns to indicate wildcards.

Check the route parameter value

Usage:

  • Use the alias: Active::checkRouteParam($key, $value)
  • Use the application container: app('active')->checkRouteParam($key, $value)
  • Use the helper function: if_route_param($key, $value)

Explanation: the package will return true if one of the following condition is true:

  • The current route contains a parameter named $key whose value is $value.
  • The current route does not contain a parameter named $key and the value of $value is null.

Read more about route parameter in the Laravel documentation.

Get the current values

Get the current action

Usage:

  • Use the alias: Active::getAction()
  • Use the application container: app('active')->getAction()
  • Use the helper function: current_action()

Explanation: if the current route is bound to a class method, the result will be a string like App\Http\Controllers\[email protected]. If the route is bound to a closure, the result will be the Closure string.

Get the current controller class

Usage:

  • Use the alias: Active::getController()
  • Use the application container: app('active')->getController()
  • Use the helper function: current_controller()

Explanation: if the current route is bound to a class method, the result will be the full qualified class name of the controller class, like App\Http\Controllers\YourController. If the route is bound to a closure, the result will be the Closure string.

Get the current controller method

Usage:

  • Use the alias: Active::getMethod()
  • Use the application container: app('active')->getMethod()
  • Use the helper function: current_method()

Explanation: if the current route is bound to a class method, the result will be the name of the controller method. like yourMethod. If the route is bound to a closure, the result will be the empty string.

Example

The example below illustrate the usage of this package in a sidebar with Bootstrap link group:

<div class="list-group">
    <a href="" class="list-group-item {{ active_class(if_route('users.list') && if_query('active', 1)) }}">
        Active users
    </a>
    <a href="#" class="list-group-item {{ active_class(if_route('users.list') && if_query('active', 0)) }}">
        Inactive users
    </a>
    <a href="#" class="list-group-item  {{ active_class(if_action('App\Http\Controllers\[email protected]')) }}">
        Add users
    </a>
</div>

 

 

 

Install nodejs 4 on 64-bit Windows machine

Yesterday, I met a problem when installing Nodejs 4 on my 64-bit Windows 7 machine at the company. The installer exited with an error message telling me that:

There is a problem with this Windows Installer package. A DLL required for this install to complete could not be run. Contact your support personnel or package vendor

I tried with both 32-bit and 64-bit versions of all Nodejs version from 4.0.0 to 4.2.1 but with no luck. I tried the `msiexec` command line to get the log file and posted to Nodejs repository to find any ideas from the community. The content of the log file is:

=== Logging started: 10/19/2015  13:44:59 ===
Action start 13:44:59: INSTALL.
Action start 13:44:59: SetInstallScope.
CustomAction SetInstallScope returned actual error code 1157 (note this may not be 100% accurate if translation happened inside sandbox)
Info 2898.For WixUI_Font_Normal textstyle, the system created a 'Tahoma' font, in 0 character set, of 13 pixels height.
Error 1723. There is a problem with this Windows Installer package. A DLL required for this install to complete could not be run. Contact your support personnel or package vendor.  Action SetInstallScope, entry: SetInstallScope, library: C:\Users\CODEFO~1\AppData\Local\Temp\MSI7073.tmp 
MSI (c) (78:E4) [13:45:04:065]: Product: Node.js -- Error 1723. There is a problem with this Windows Installer package. A DLL required for this install to complete could not be run. Contact your support personnel or package vendor.  Action SetInstallScope, entry: SetInstallScope, library: C:\Users\CODEFO~1\AppData\Local\Temp\MSI7073.tmp 

Action ended 13:45:04: SetInstallScope. Return value 3.
Action start 13:45:04: FatalError.
Info 2898.For WixUI_Font_Bigger textstyle, the system created a 'Tahoma' font, in 0 character set, of 19 pixels height.
Action ended 13:45:05: FatalError. Return value 2.
Action ended 13:45:05: INSTALL. Return value 3.
=== Logging stopped: 10/19/2015  13:45:05 ===
MSI (c) (78:E4) [13:45:05:193]: Windows Installer installed the product. Product Name: Node.js. Product Version: 4.2.1. Product Language: 1033. Manufacturer: Node.js Foundation. Installation success or error status: 1603.

Finnaly benjamingr gave me a suggestion that solved the problem. I cleared my Temp folder ignoring any busy files and reran the x64 installer (the node-v4.2.1-x64.msi file). The new version of Nodejs and NPM installed successfully:

$> node -v
v4.2.1
$> npm -v
2.14.7

My original issue on GitHub here.

Mailtrap vs Mandrill: sending emails in development

After a long time using Gmail SMTP as my main service for sending email in development environment. I switched to Mailtrap and Mandrill for some recent projects. This post will introduce my experiment when working with these two awesome mailing services.

Mailtrap.io

Mailtrap is a fake SMTP server for development teams to test, view and share emails sent from the development and staging environments without spamming real customers.

As its slogan, Mailtrap focuses to non-production environment. It “traps” all email sending services from your application and display these email in a virtual inbox. You can preview all your sent emails without caring about the recipient addresses exist or not. If you use real email addresses (in staging server for example), there will be no real email sent to them, too, which means there is no spam to your customers.

Mailtrap virtual inbox with a spam analysis
Mailtrap virtual inbox with a spam analysis

You can create many inbox  and use different inbox in different enviroments. However, the free plan of Mailtrap support only one, and a maximum of 50 emails inside the inbox. When this inbox is full, you can continue trapping your emails by clear all old ones.

You can see many useful information about each email inside the inbox via detail panel:

  • The address of sender and recipient as well as the sending time
  • The rendered HTML email, HTML source
  • The text email if available
  • The spam analysis to make sure your email is not recognized at spam by other machines
  • The result of email HTML validation in many email clients

Mandrill

Mandrill is a reliable, scalable, and secure delivery API for transactional emails from websites and applications. It’s ideal for sending data-driven transactional emails, including targeted e-commerce and personalized one-to-one messages.

Different from Mailtrap, Mandrill is a service allowing sending real email even in production environment. To prevent your customer from receiving testing email when develop your application, you can switch to test mode of your Mandrill account. In this mode, Mandrill works like Mailtrap: not sending any emails, just show them in its reports with the email content.

Mandrill outbound report in test mode
Mandrill outbound report in test mode

You can see basic information about your email inside the Outbound report of Mandrill:

  • Email sender, recipient and delivering time
  • Email subject
  • Email content: HTML and text (via another popup window)

Comparision

Feature Mailtrap.io Mandrill
Use in production No Yes
Free tier Yes Yes
Trapping emails Yes (default) Yes (After turn test mode on)
Email information
  • Sender
  • Recipient
  • “Delivery” time
  • Subject
  • email HTML
  • email text
  • Sender
  • Recipient
  • “Delivery” time
  • Subject
  • email HTML
  • email text
Extra information
  • HTML validation in many email clients
  • Span analysis report
  • Using email templates in the account
  • Statistic about email opening and clicking
Support SMTP Yes Yes (real mode only)
Support REST api Yes Yes (required when using test mode)
Separate emails in environments By sending email to different virtual inbox By adding tags into STMP headers or REST payload.
Refresh time Instant With a little delay

My favorite one

I cannot tell you which service is better. Each one has its own pros and cons. In my personal project, I use Mandrill for production environment because it awesome delivery feature. In development servers, I use Mailtrap to analyze my emails and preview the result quickly. I hope that after reading my reviews, you can have another inspiration for sending emails in your application.

 

Solve the Akismet problem with CloudFlare SSL

Askimet over SSL

Today, I’ve spent time on diving in the solve the problem when using Akismet anti-spam plugin over the HTTPS with the setup using CloudFlare flexible SSL.

Problems

Akismet is one of most common anti spam plugin used in WordPress. I’ve used this plugin for years and really been satisfied with it. The problem occurs when I put this blog after the CloudFlare CND services and enable its flexible SSL support.

The first thing I’ve tried is install the CloudFlare flexible SSL plugin for WordPress. After that, most of problems was resolved but one. My Firefox still gives me a notification that their are some mixed content on my site, which means some of my resources were loaded from unsecured  protocol. I opened the web console and realized that the Akismet js file is still hosted over non-SSL domain.

[blocked] The page at ‘https://www.hieule.info/’ was loaded over HTTPS, but ran insecure content from ‘http://blog-static.hieule.info/wp-content/plugins/akismet/_inc/form.js’: this content should also be loaded over HTTPS.

I tried to install many HTTPS forcing plugins from WordPress repository but nothing helps.

The reason

The reason why the CloudFlare flexible SSL plugin works for other resources but the Akismet ones is the order WordPress loads its plugins. The CMS sort active plugins by their names before saving into the database and load the plugin using that order. Therefore, the Akismet was loaded before the CloudFlare flexible SSL, and all of Akismet resources was still using the non-SSL domain.

My working solution

The most easy way is install a WordPress plugin to customize the loading order of the CMS active plugins. I can find this plugin and confirmed that it works correctly. However, I do not want to install too much plugin into my WordPress installation. I tried to change the order of active plugin right before they are saved to the database. Fortunately, WordPress support the active_plugins hook called right there. I add these lines of code into my functions.php in my theme:

function pureclean_sort_plugins() {
    $active_plugins = get_option('active_plugins');
    $this_plugin = 'cloudflare-flexible-ssl';
    $this_plugin_key = array_search($this_plugin, $active_plugins);
    if ($this_plugin_key) { // if it's 0 it's the first plugin already, no need to continue
        array_splice($active_plugins, $this_plugin_key, 1);
        array_unshift($active_plugins, $this_plugin);
        update_option('active_plugins', $active_plugins);
    }
}
add_action("activated_plugin", "pureclean_sort_plugins");

Finally, deactivate a plugin and reactive it to make these code work (don’t forget this step). The problem was resolved.

References:

Laravel (v4, v5) global site message system

The idea about global site messages

Messages here are something you want to show to users to tell them about the result of some actions. For example, after pressing the login button, a sentence appears in the new page to tell users that they have been logged in or there are any errors. In my opinion, a practical site message system should has following properties:

  • A message must belongs to one of those types: success (indicates that the previous action is done as expected), info (provides more information), warning (tells user that the action was completed, but there are something wrongs) and error (means we cannot complete the action by some reasons)
  • Each message can be generated when processing a request
  • Messages is visible in the current response or in the next one request.

When working with a project, I have created a package for Laravel (both version 4 and 5) which has all above functions: Laravel Alert (hieu-le/laravel-alert). I usually use Bootstrap alert to display site messages. So that the output of this package will follow Bootstrap alert format by default. Of course, we can change the appearance and more configurations.

Installation

You will need Composer to use this package, it is a package manager which will install dependencies of your project automatically via a json file called composer.json . There is a nice tutorial of using Composer for those who are not familiar with this awesome tool.

Add this dependency to your composer.json  file, please note that you must choose version 1.x if working with Laravel 4 and version 2.x for Laravel 5:

In a Laravel 4 project:

"hieu-le/laravel-alert": "~1.0"

In a Laravel 5 project:

"hieu-le/laravel-alert": "~2.0"

Run composer update  to make Composer install my package and its dependencies. After that, you must add the package Service Provider and register a alias to use the package easier. Open the app/config/app.php  in Laravel 4 (or config/app.php  in Laravel 5). Add the following line to the providers  array:

'HieuLe\Alert\AlertServiceProvider',

And add this line to the aliases  array

'Alert' => 'HieuLe\Alert\Facades\Alert',

Basic usage

If you want to add a message, call one of these method based on the message type:

Alert::success("Well done! You successfully read this important alert message.");
# or
Alert::info("Heads up! This alert needs your attention, but it's not super important.");
# or
Alert::warning("Warning! Better check yourself, you're not looking too good.");
# or
Alert::error("Oh snap! Change a few things up and try submitting again.");

Currently, once added, a message cannot be changed or removed. They will be displayed in group of message types, in the order of they registered. To display these message, call this method in your view template:

Alert::dump();

With those inputs, the default output will be something like this:

    <div class="alert alert-success" role="alert">
      <button type="button" class="close" data-dismiss="alert" aria-hidden="true">×</button>
      <h4><i class="icon fa fa-check"></i> Success!</h4>
      <p>Well done! You successfully read this important alert message.</p>
    </div>
    <div class="alert alert-info" role="alert">
      <button type="button" class="close" data-dismiss="alert" aria-hidden="true">×</button>
      <h4><i class="icon fa fa-info"></i> Info!</h4>
      <p>Heads up! This alert needs your attention, but it's not super important.</p>
    </div>
    <div class="alert alert-warning" role="alert">
      <button type="button" class="close" data-dismiss="alert" aria-hidden="true">×</button>
      <h4><i class="icon fa fa-warning"></i> Warning!</h4>
      <p>Warning! Better check yourself, you're not looking too good.</p>
    </div>
    <div class="alert alert-danger" role="alert">
      <button type="button" class="close" data-dismiss="alert" aria-hidden="true">×</button>
      <h4><i class="icon fa fa-ban"></i> Error!</h4>
      <p>Oh snap! Change a few things up and try submitting again.</p>
    </div>
Example of Laravel Alert message appearance
Example of Laravel Alert message appearance

Change the icons of message types

Laravel Alert comes with a set of icons for each type of message. To display these default icon, the Font Awesome must be installed in your project. You can choose another font, or use images, texts, too. Let start with copying the package config file to another location to edit.  Run one of these command from you project root folder:

Laravel 4:

php artisan config:publish hieu-le/laravel-alert

Laravel 5:

php artisan vendor:publish

A new file will be created at app/config/packages/hieu-le/laravel-alert/config  (or config/alert.php  in Laravel 5 projects). Change the content of each message type inside the icons  array to the content of your desired icons (or images, texts). For example:

return [
    'icons'       => [
        'success' => '<img src="imgs/icons/success.png" />',
        'info'    => '<img src="imgs/icons/info.png" />',
        'warning' => '<img src="imgs/icons/warning.png" />',
        'error'   => '<img src="imgs/icons/error.png" />',
    ],
]

Change the appearance of output

If you do not like the default output of messages, you can use your own view template to display them by replacing the value of view element in the package configuration file (see above section for details). The view will be rendered for each message type that contains at least one message. In your view, there will be two variables passed:

  • $icon : the string of the icon of the message type
  • $messages : the array contains all messages of the message type

Using with Laravel validation errors

For those who have worked with Laravel validation errors output, there are two ways to integrate with Laravel Alert. The first one is “copy” the validation errors and add to the global message system managed by the package manually:

if ($validator->fails())
{
    foreach($validator->messages()->all() as $message)
    {
        Alert::error($message);
    }
}

Another approach is keep your current controller code and update your view template by passing an additional argument into the Alert::dump method like this (remember that Laravel include a special variable called $errors in each view):

echo Alert::dump($errors->all());

Conclusion

With the help of the Laravel Alert package, developers now can manage and display global site messages easier and more semantically. If you need more feature or found a bug, please raise an issue in the Github repository.

Symfony Dom Crawler Component and UTF-8 HTML

I have used the Dom Crawler Component of Symfony to parse some result from Google SERP pages. After the first try, I learned that the result of <span class="lang:php decode:true crayon-inline ">$crawler->html()</span> is not as I expected: all UTF-8 characters was displayed incorrectly.  The problem can be reproduced easily via this script (supposed that all classes has been loaded via Composer autoload or anything else):

$html = <<<'HTML'
<!DOCTYPE html>
<html>
  <head>
    <title>Đây là một chuỗi UTF-8</title>
  </head>
  <body>
  </body> 
</html> 
HTML; 

$crawler = new \Symfony\Component\DomCrawler\Crawler($html);
echo $crawler->html();

The output will be:

<head><title>Đây là một chuỗi UTF-8</title></head>
<body>

</body>

Which is absolutely not what we want.

Quick fix:

Instead of passing the HTML content to the crawler constructor, passing it in the next call via addHtmlContent method

$html = <<<'HTML'
<!DOCTYPE html>
<html>
<head>
  <title>Đây là một chuỗi UTF-8</title>
</head>
<body>

</body>
</html>
HTML;

$crawler = new \Symfony\Component\DomCrawler\Crawler();
$crawler->addHtmlContent($html) # this line is added
echo $crawler->html();

 

And the output will be fine:

<head><title>Đây là một chuỗi UTF-8</title></head>
<body>

</body>

Note: the <!DOCTYPE>  and <html>  markup need adding manually.

 Explanation

This unexpected behavior happens with HTML content that does not have a meta element telling the crawler that the content is UTF-8 encoded. In the constructor, the component check for these declaration to decide whether using UTF-8 or the default ISO-8859-1 charset (defined in HTTP 1.1 spec). The addHtmlContent method will allow us to choose which charset is used.

Laravel 4 life cycle: Request, Route and Router object

Today, I’ll share you about one of my experiment when working with Laravel 4 application: distinguish Request, Route, and Router object of the application instance. All these object play a part in the routing phase, the step that decide which method of which controller will be call on each time the Laravel application run.

The Request object

A Request object is mapped from the raw HTTP request that received from web server. There is no information about “controller” or “method” stored in it. This object is created in two ways in Laravel:

  • Created directly from PHP superglobal when the application is booted by web server. This is most common situation when a user reach your application from an URL.
  • Created manually when runnin a unit test

In this article, I’ll talk about the Request object that created by first method. This object contains information about:

  • All parameters from $_GET
  • All parameters from $_POST
  • All cookie information from $_COOKIE
  • All uploaded file information from $_FILES
  • All server information from $_SERVER

The Request object is the first object that created in the application life cycle among these three. It is created inside the constructor of the Application  class.

 The Route object

A route is a 3-tuple of these information:

  • Which HTTP method is currently handled (GET, POST, PUT or DELETE, …)
  • Which URI is currently served (/foo, /bar/1, or /foo/1/bar, …)
  • Which action will be executed. An action is whether a closure object or a method of some controller, both will return a response in the end (an HTML page or a JSON content, …)

A route may contains a name defined in the routes.php  file (and any files that included by that file). Without this object, Laravel cannot know how to continue serving the current request because it cannot decide which is the correct action to do (the C part in MVC definition). The Route object is the last created object among these three

The Router object

As a result, a question is asked: “How is the route object created?”. Well, the answer is in this section: the Router object. It is created independently with the Request object via the RoutingServiceProvider which is called after the application is created (and after the Request object is created). This object keep the reference to the current Request object and the matched Route object that matched with defined rules.

When created, the Router object:

  • creates a collection of registered routes
  • allows new route registered inside routes.php
  • look up it registered route collection to choose the correct route with the current request

So, we can think this object as a link between the HTTP request and the Laravel controller.

Conclusion

Each of three objects keep some information that may be useful in different situation. In my experience:

  • When you want to known information about the HTTP request (server, cookie, get and post params, … ), use the Request object
  • When you want to known information about the current action (controller, method, closures, …), use the Route object
  • If you does not decide what information will be get, use the Router object which can access both of above. One of my package can be used as an example for this situation.

Happy coding!