Web Server Issues

1. N/W  I/O

2. Disk I/O

3. CPU

4. Memory

A. Dig the root cause : too many processes may start thrashing,

which may seem like a Disk I/O problem, but actually is a memory

problem

Tools : sar,systat,iostat,top,vmstat,nstat

---------------------------------------

Apache performance testing : TSUNG

Apache usually dies at 5000 concurrent connections.

It usually works in pre-fork/worker model.

Pre-fork : OS kernel does the load balancing.

Worker : Apache master process does the load balancing.

Usually every process spawns many threads.

A process has 4GB address virtual memory

shared by all its threads.

So, if there are too many, for e.g. 500, threads,

one thread may bajaao the band for everyone else.

So,it's better to have 5 processes with 100 threads each.

But in apache, creating many threads has its own problems : 

every thread has its kernel stack, process stack, which eats

up resources.

Lighttpd works on event driven model.

Accepts connections, passes the baton

to fast-cgi.exe, when it returns, lighttpd

returns to requester.

Since it is event driven, it scales better.

Work is on to make lighttpd work better on multiple cores,

so that it can spawn a different process for every core.

-------------------------------

FrontEnd optimization : 

in .htaccess set headers(for e.g. TTL(time to live), time after which the client should fetch the image again) for static images.

caching proxy layer : varnish/squid : saves db query time, php script running time.

a separate caching layer, as opposed to php's in-built caching may help diagnose the issue better.

- proxy because caching runs at :80, server at :8080

you can also look into php opcode caching...

------------------------

look at 31 yahoo tips, yahoo performance blog, mysql performance blog

------------

If there are too many threads running, and you may be using external

libraries(code) from them, it's better to kill a thread after it has served

5000(n) requests. Since, after that it will have many memory leaks,

which will be hard to find.

-------------------

MySql General Info

ibdata/iblog are write ahead logs(queries are written in the logs before executing them on the server)

- used for recovering a crashed server

binlog -> logs of queries after committing

- for slave level replication

Slave level replication is async, i.e. there are 2 threads : 

1. slave reads queries from server and writes them to its relay log
2. another process reads from relay log and executes them

hence it's async

If it were sync, it will hamper performance of master, since

the master will have to wait for the slave to write in db.

------------------------------------

To set up a slave : 

When you take the dump from the master, specify an option, which will

tell the slave, from where to start reading the logs.

You can specify in slave config, about which dbs/tables to include/exclude.

---------------------------

show slave status\G

start slave

----------------------

keep taking periodic solid dumps

Intellij Idea : Using framework as a RSL(Remote shared library) in a Flex project

In Flex Builder 3: 

Project -> Properties -> Flex Build Path -> Library Path -> Framework linkage in Flex Builder.

Flex builder also copies the required files : 

framework_x.y.z.abcd.swz

framework_x.y.z.abcd.swf

In Intellij Idea 9.0.3 : 

Right click on a Module -> Module Settings -> Flex Compiler Settings -> Advanced->

Select Use Framework as Runtime Shared Library (RSL)

Unlike Flex Builder, Idea doesn't copy the required .swz and .swf files.

You have to copy them manually to the location where your main application swf resides.

Copy them from here : 

{location of your flex sdk}/frameworks/rsls/

To verify if RSL is working for you, clear the Flash Player Cache and run your flex application.

You should get a swz or swf file of roughly 525KB in your cache.

On Windows XP, the location of the Flash Player Cache : 

C:\Documents and Settings\{User name}\Application Data\Adobe\Flash Player\AssetCache\

On linux, it's : 

.adobe/Flash_Player/AssetCache

Installing and running hadoop on windows xp in standalone mode

Prereqs : have cygwin installed

1. Download hadoop from http://hadoop.apache.org/common/releases.html#Download

2. unpack the file hadoop-0.20.2.tar.gz.gz in C:/ (using tar xvf filename)

3. Add the following to conf/hadoop-env.sh in the unpacked folder : 

export JAVA_HOME=/cygdrive/c/Java/jdk1.6.0_23

(assuming that C:/Java/jdk1.6.0_23/bin contains javac.exe and other binaries)

4. Download companion source code for Hadoop:The Definitive Guide from https://github.com/tomwhite/hadoop-book/archives/master

5. Unpack the file tomwhite-hadoop-book-32dae01.tar.gz

6. in the unpacked folder : cd ch02/src/main/java

7. mkdir -p build/classes

8. $ javac -verbose -classpath C:\\hadoop-0.20.2\\hadoop-0.20.2-core.jar MaxTemperature*.java -d build/classes
9. export HADOOP_CLASSPATH=build/classes

10. hadoop MaxTemperature ../../../../input/ncdc/sample.txt output

your output is in output folder

Shell: showing lines which are present only in one file

comm -13 <(sort a.txt) <(sort b.txt)
will show only the lines which are present in b.txt

but not in a.txt.

Vim regexp example

I have a file : 

alert

  alert

  //alert

So, line 1 has "alert" without any white spaces.

line 2 has white spaces followed by "alert"

line 3 has white spaces,"//" followed by "alert"

I want to find "alert" which is not commented, i.e. not following "//"

Here is the regexp : 

/\(^\|[^\/\/]\)alert  

which is basically : 

(beginning of line OR absense of "//") followed by "alert"

Shell scripts on Cygwin, xargs in general

Lessons learned today : 

1. Convert the scripts and the files they operate upon to unix, using d2u.

2. For debugging, add #!/bin/bash -x at the top of the script, so that you know

which variable gets expanded to what?

3. For debugging xargs 
(i) find out whether the command being used with xargs expects single operand or multiple operands, for e.g.

find . -name expects a single operand, whereas grep -i can operate on multiple operands.

(ii) if you need to convert multiple lines of input to single line, do : 

xargs --max-args=1

(iii) use -t option with xargs to see what's going on?