Stack Optimization for Magento

Caching is always the First Step towards performance enhancement.

Why use Redis for Magento?

Lets have a look at how the Magento caching (which uses the Zend_Cache library) works.
Each cache entry consists of the following information

  • the cached data
  • a cache key (or ID), that uniquely identifies this entry and is used to retrieve the data from the cache
  • a cache lifetime, after which the cache entry expires
  • zero or more cache tags

On the cache management page most of the cache tags used by Magento are listed. Depending on the modules and extensions installed there could be more or less tags, e.g. CONFIG, LAYOUT_GENERAL_CACHE_TAG, BLOCK_HTML, TRANSLATIONS, FULL_PAGE_CACHE, …3

Magento offers several different options what to use as a cache storage, and each of these storage systems is used by means of a PHP class called “cache backend”.

By default, cache data is stored in files (located in the directory var/cache/).
Another option is DB – which is not recommended as it would cause harm than good.
Memory is ideal option in this scenario,except for one problem: APC and memcached only support storing simple key-value pairs, so the cache tags are lost! This renders the whole caching rather useless, because every time we need to clear only one part of the cache, EVERY cache entry is cleared.

But, do not despair! The Zend Framework contains a solution to this problem. There is a special cache backend called Twolevels. The Twolevels backend uses a fast cache backend (i.e. APC or memcached) for the cache data, and a slow backend (i.e. files or database) for the lifetime and the cache tag information. This way we can have the best of both worlds!

Now that we use Redis – it is capable of storing keys-values as well as cache tags!!! with APC only for caching PHP complied code – And since Magento makes heavy use of cache tags, the effect is quite noticeable, depending on the number of records in the cache. During Migration I had faced issue of Magento pointing to old DB which can be avoided using the following redis cli command

#redis-cli -h <IP> flushdb

Magento makes it easy to configure all this by the way – have a look at the file app/etc/local.xml.additional for further information.

Install DisableLog Magento Extension
Magento, by default for each and every request magento would put an entry on DB (which updates 5 tables). This fills up the database slowly and also gives a slightly lower pageload. And if you’re using external site-statistics-tools like GoogleAnalytics and Piwik away, this logging is pointless. Disable it using this extension.

Mysql DB Caching  [Not Implemented]
query_cache_type=1
query_cache_size=64M   ,   query_cache_limit=2Mb

They have a list of settings there for my.cnf as follows:

key_buffer = 512M
max_allowed_packet = 64M
table_cache = 512
sort_buffer_size = 4m
read_buffer_size = 4m
read_rnd_buffer_size = 2m
myisam_sort_buffer_size = 64m
tmp_table_size = 128m
query_cache_size = 96m
query_cache_type = 1
thread_cache_size = 8
max_connections = 400
wait_timeout = 300

APC for PHP Opcode Cache

Following are the settings in the /etc/php/php.d/apc.ini file

extension=apc.so
apc.shm_size=256M
apc.num_files_hint=10000
apc.user_entries_hint=10000
apc.max_file_size=5M
apc.stat=0

Find the apc.php file at /usr/share/php/apc.php and copy it into your web server root. This file will help you determine how effectively APC is working on your system and whether you need to make adjustments to the amount of RAM you make available to APC.

Apache – Setting MaxClients

Apache – Enable Expires Headers

Browsers use caching extensively, and can save a lot of the elements included in a web site locally so that they can be served from the browser’s cache rather than the web server on the next request. This can help quite a bit in shortening load times. The problem is for the browser to know when a file can be served from the cache, and when not – because the local copy is outdated.  To solve this issue, browsers rely on two HTTP headers, Expires and Cache-Control.

Magento’s default .htaccess file already configures these according to Yahoo’s performance recommendations (more on them below), but does not enable them by default. To enable them, all you need to do is add the following lines to your Apache server configuration (usually found in /etc/apache2/apache.conf):

<IfModule mod_expires.c>
ExpiresActive On
</IfModule>

Remove unnecessary apache modules
Example :

LoadModule proxy_connect_module modules/mod_proxy_connect.so

Apache may not any configuration related to proxy still it is loaded on spawning of each apache process.

Remove unnecessary php modules for performance & security
Example:

#mv /etc/php.d/sqlite3.ini /etc/php.d/sqlite3.disable

Other compiled-in modules can only be removed by reinstallating PHP with a reduced configuration. You can download php source code from php.net and compile it as follows with GD, fastcgi, and MySQL support:

./configure --with-libdir=lib64 --with-gd --with-mysql --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/usr/com --mandir=/usr/share/man --infodir=/usr/share/info --cache-file=../config.cache --with-config-file-path=/etc --with-config-file-scan-dir=/etc/php.d  --enable-fastcgi --enable-force-cgi-redirect

Use mod_deflate in apache configuration

Check if compression is enabled across multiple web browsers : http://nontroppo.org/tools/gziptest/

Use FastCGI to run PHP [Not Implemented]

If you are using Apache as your web server, there are two ways you can sensibly set up PHP. The first way is to use the mod_php module, which is easiest to use and hence the default with many hosting providers. In this case, a PHP interpreter is run with each process of the web server and on stand-by until a script is executed.

If you are wondering why your Apache needs a lot of memory, probably the fact that you are using mod_php is a big part of the answer. On a large site, there may well be hundreds of Apache processes running, and each has its own PHP interpreter. However, only very few of them – often less than one in ten – actually need to run PHP. The rest serve static files or simply wait for new requests. Because PHP uses a lot of memory, it is a good idea to see if you can avoid the overhead generated by having dozens and dozens idle PHP processes running.

The way to avoid the overhead is to use FastCGI instead of mod_php. With FastCGI, a separate internal daemon is run on your web server that is contacted by the web server only when execution of PHP required. Thus you do not need to carry the PHP baggage for all requests.

Setting up FastCGI requires you to make some changes to your server configuration, but the benefits will be large. A good starting point is this article: http://2bits.com/articles/apache-fcgid-acceptable-performance-and-better-resource-utilization.html. For more details, check the Apache web site and the documentation of your distribution.

Alternative: Turn Off Keep-Alive if you have to use mod_php [Not Implemented]

If you cannot, or do not want to, switch to FastCGI, you can still do something to reduce the memory usage per visitor. This is important because each server has only so much memory, and if you can serve a visitor with less memory, you can server more visitors in total and scale further with given resources.

As I pointed out above, the big problem with mod_php is that you need to keep a PHP interpreter running with each Apache process, and that most Apache requests do not involve PHP. By default, Apache enables a feature called HTTP Keep-Alive, which lets visitors re-uses a connection for several requests. This makes the web site faster for the visitor, because images and other static files can be loaded without continuously re-connecting with the web server. But it also means that your web server will have many idle processes, waiting for new requests that may never arrive. And each of these idle processes runs a full PHP interpreter.

To turn off Keep-Alive, search your Apache configuration files for the KeepAlive directive. If the directive is not set, add the following line to your config file

KeepAlive off
and then restart Apache. If it is set, ensure that it is set to “off“. You should start to see lower memory usage immediately.

However there are other few down-sides to mod_php though: you’re forced to use Apache’s less efficient prefork MPM and your Apache processes will get large. Serving static files with 32MB+ Apache processes isn’t very efficient.

What is the difference between PHP CGI and PHP FastCGI?

PHP CGI
When PHP is run as a CGI program the web server spawns a process each time a request is made. This can be less efficient unless your operating system is good at spawning processes quickly. Unlike mod_php, CGI application won’t bloat your Apache, so the processes can remain small for service static files – important for high traffic sites.

PHP FastCGI
When PHP is run as a FastCGI, a number of daemon processes are started and sit waiting for the web server to request them to run PHP code. This avoids the cost of spawning new processes each time in addition to not bloating web server processes. Due to the nature of long-running processes, FastCGI may require some periodic process maintenance. Lighttpd’s spawn-fgi script provides a method to limit the number of requests a single FCGI process is allowed to serve in an attempt to reduce the effect of memory leakage.

OS Optimizations

NoAccessTime
Select the partition which has the documentroot, NFS etc
Edit /etc/fstab file and add “noatime” to the options in the fourth column, and remount everything with the mount -a command.
Now the file access time isn’t updated after every access hence giving us a little bit of performance gain.

NFS I/O – Diagnostic
If Using check its metrics using the

#nfsiostat
op/s        rpc bklog
1.91        0.00
read:             ops/s           kB/s          kB/op        retrans        avg RTT (ms)    avg exe (ms)
0.005       0.116      25.601        0 (0.0%)       3.681       3.730
write:            ops/s           kB/s          kB/op        retrans        avg RTT (ms)    avg exe (ms)
0.004       0.011       2.840        0 (0.0%)      1107.043       6.886

Here avg RTT for write is very high!! hence we can say that the write performance is poor looks like this is due to a network or NFS server latency because “avg RTT” should be much lower.
Where,
“avg RTT”: “the network + server latency of the request”
“avg exe”: “the total time the request spent from init to release”

NFS-PerformanceTune
When mounting an NFS export on an NFS client one can specify NFS mount options that over-ride the defaults. e.g.
#mount -t nfs -o intr,rsize=65536,wsize=65536,noacl,nocto,nodiratime nfsserver:/mnt/export /mnt/nfsclient-mountpoint/
Here’s a list of important NFS mount options:

intr
Use the intr mount option when using the default hard mount option. It allows signals to interrupt file operations, thus allow recovery from what appears to be an NFS hang.

rsize
The typical default read size transferred in a packet is 32768 bytes. Increasing this value may increase the performance depending on the size of the data being read. Recommended values for this parameter are numbers within the power of 2 (4096, 8192, …). Large values may not work with NFS version 2. Note: Setting this size to a value less than the largest supported block size will adversely affect performance.(max value = 65536)

wsize
The typical default write size transferred in a packet is 32768 bytes. Increasing this value may increase the performance depending on the size of the data being written. Recommended values for this parameter are numbers within the power of 2 (4096, 8192, …).Note: Setting this size to a value less than the largest supported block size will adversely affect performance.(max value = 65536)

relatime/noatime
To avoid updating the inode’s access time use noatime. Alternately, use relatime to update the access time only when the access time is earlier than the modify or change time of the inode.

noacl
Disables Access Control List (ACL) processing.

nocto
Suppress the retrieval of new attributes when creating a file.

nodiratime
Setting this value disables the NFS server from updating the directory access time. This is the directory equivalent setting of noatime.

noac
Disable all forms of attribute caching entirely. This extracts a significant performance penalty but it allows two different NFS clients to get reasonable results when both clients are actively writing to a common export on the server.

Current status on NFS
The amount and size of data, that the server and the client uses, for passing data between them is very much important.
Use df -hT to get the NFS Server IP and enter the following to get the status

#grep <IP> /proc/mounts
10.0.0.205:/DoNotDelete/media /DoNotDelete/media nfs4 rw,relatime,vers=4.0,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.0.2.52,local_lock=none,addr=10.0.0.205 0 0

Decreasing the size of read and write in RPC packets, will increase the total number of network IP packet’s that need to be send over the network.

Which means if you have 1 MB of data, dividing it into equal chunks of 32KB will increase the number of chunks, and if you divide it in equal chunks of 64KB the number of chunks will be reduced. Which means you need to send a high number of IP packet’s over the network if you decrease these values, and if you increase these values, you will have to send less number of IP packets over the network.

So our decision on modifying this parameter, must always depend on the network capability. If suppose you have 1 Gigabit port on your NFS server and client, and your network switches connecting these server’s also are capable of 1G ports, then i would suggest to tweak these parameter’s to a higher value.

Maximum Transfer Unit (MTU) of Network Interface
Running command “tracepath <nfs-server>” from NFS client should provide one with current path MTU on the NFS client. MTU of network card should be set equal to the path MTU obtained with tracepath command. This should  help in bringing down packets getting dropped to a lesser value. For reference see ifconfig man pages.
Security Best Practice for the PHP App
http://www.cyberciti.biz/tips/php-security-best-practices-tutorial.html

More into Magento File System

1.Dir : magento/app
This is your Magento application directory. This is the directory where the final class Mage… application file (Mage.php) is stored.

2.Dir : magento/app/code
This is your Magento code directory. This is base directory for the three Magento code pools (core, community, local).

3.Dir : magento/app/etc
The etc folder is where Magento stores system level (as opposed to module level) configuration files. The name etc is borrowed from the *nix family of operating systems, and Magento’s configuration files are all XML based.

File : magento/app/etc/local.xml – is where we mention DB credentials, session storage (using memcache/db) , caching backing (eg.redis) etc are configured.

4.Dir : magento/media
Magento’s media folder is where media files (images, movies, etc.) related to data (products) is stored.
This dir is chosen as the shared directory in a multi-server environment.

5.Dir : magento/skin
Skin folder contains images, CSS and Javascript files used by your themes.This is not the only folder where you’ll find images, CSS or javascript though. This folder is meant for files that are customized per theme.

6.Dir : magento/var
The var folder is another one borrowed from the *nix world. The var stands for Variable files, and is intended to store files which are expected to change during normal system operations.

7.Dir : magento/var/cache
Magento, rather famously, makes heavy use of caching for activities that might bog down the system if they had to be performed every-time a page loads. For example, Layout XML files are merged once, and then the tree is cached so they don’t need to be merged again. The cache folder is one place where Magento will store these cached results.

8.Dir : magento/var/log
Magento’s log folder is where is stores the system and exception logs. These logs can be turned on from the Admin Panel’s section. The apache/web-server user will need write permission on this folder and the files therein.
System -> Configuration -> Developer -> Log Settings

9.Dir : magento/var/session
During installation you have the option of storing user sessions on disk, or in the database. The session folder is where the user sessions are written out to and read from if you choose to store them in the filesystem.

10.Dir : magento/media/upload
There are a number of Admin Panel features which allow you to upload media files (default logos, etc.). The upload folder is where Magento stores these files.

More Steps
1. Increase Ulimit

2. Go to: System->Configuration->CATALOG/Catalog->Frontend

Use Flat Catalog Category: Yes

Use Flat Catalog Product : Yes

Go to: System->Configuration->ADVANCED/Developer: Merge javascript and CSS

 

To change the domain name of a magento site, we need to upate a table called core_config_data .
URL – http://www.magentocommerce.com/wiki/1_-_installation_and_configuration/update_site_url_in_core_config_data

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s