Tuesday, January 31, 2017

Redis (Cluster) notes

Benchmark tool
Comes with Redis. Very good with variety of options. 

Server on AWS  (client on my local machine)
PS C:\Users\user> redis-benchmark -t set -n 100000 -h <MY_AWS_FREE_TIER_IP> -p 81
====== SET ======
  100000 requests completed in 40.93 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

0.00% <= 14 milliseconds
0.22% <= 15 milliseconds
10.06% <= 16 milliseconds
24.73% <= 17 milliseconds
30.37% <= 18 milliseconds
32.67% <= 19 milliseconds
43.62% <= 20 milliseconds
63.83% <= 21 milliseconds
71.76% <= 22 milliseconds
74.68% <= 23 milliseconds
79.34% <= 24 milliseconds
89.36% <= 25 milliseconds
95.18% <= 26 milliseconds
97.42% <= 27 milliseconds
98.39% <= 28 milliseconds
98.87% <= 29 milliseconds
99.11% <= 30 milliseconds
99.36% <= 31 milliseconds
99.54% <= 32 milliseconds
99.65% <= 33 milliseconds
99.72% <= 34 milliseconds
99.80% <= 35 milliseconds
99.85% <= 36 milliseconds
99.89% <= 37 milliseconds
99.91% <= 38 milliseconds
99.93% <= 39 milliseconds
99.95% <= 40 milliseconds
99.96% <= 41 milliseconds
99.97% <= 42 milliseconds
99.98% <= 44 milliseconds
99.98% <= 45 milliseconds
100.00% <= 46 milliseconds
100.00% <= 50 milliseconds
100.00% <= 50 milliseconds
2443.38 requests per second

Localhost: (both client and server)
PS C:\Users\user> redis-benchmark -t set -n 100000
====== SET ======
  100000 requests completed in 1.42 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

89.97% <= 1 milliseconds
99.88% <= 2 milliseconds
99.91% <= 3 milliseconds
99.93% <= 4 milliseconds
99.95% <= 8 milliseconds
99.96% <= 9 milliseconds
99.98% <= 10 milliseconds
99.99% <= 11 milliseconds
99.99% <= 12 milliseconds
99.99% <= 13 milliseconds
99.99% <= 14 milliseconds
99.99% <= 15 milliseconds
100.00% <= 16 milliseconds
100.00% <= 17 milliseconds
100.00% <= 18 milliseconds
100.00% <= 19 milliseconds
70521.86 requests per second

With pipelining (QPS is much higher but latency for 90 percentile requests is much higher)
PS C:\Users\user> redis-benchmark -t set -n 100000 -P 100
====== SET ======
  100000 requests completed in 0.21 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

0.00% <= 1 milliseconds
0.60% <= 2 milliseconds
1.00% <= 3 milliseconds
1.10% <= 4 milliseconds
1.30% <= 5 milliseconds
1.40% <= 6 milliseconds
10.70% <= 7 milliseconds
35.80% <= 8 milliseconds
46.70% <= 9 milliseconds
57.00% <= 10 milliseconds
69.70% <= 11 milliseconds
79.10% <= 12 milliseconds
86.60% <= 13 milliseconds
91.70% <= 14 milliseconds
94.00% <= 15 milliseconds
97.00% <= 16 milliseconds
99.30% <= 17 milliseconds
100.00% <= 17 milliseconds
467289.72 requests per second

  1. Redis is single threaded, can fork another thread(process?) for persistence.
  2. If running in cluster mode, one node should have n/2 instances (master + slave) where n = NUM_CORE since one process for serving commands, another for persistence
  3. RDB vs AOF persistence
  4. Slave can be configured to become master if master hasn't been contacted in a while.
  5. Total ~16K Hashslots.
  6. Resharding (redistribution of keys) is always manual, whether adding a node or deleting one. Failover is automatic since slave already has the the same keys as master.
  7. Pipelining will increase throughput but 90-95 percentile latency will be very high. Essentially 100 percentile latency will be lower than non-pipeline version but for other percentiles it will be very high.
  8. Others : Recently Geo commands were added. Though tiles38 is also there for the similar stuff.

Forgot alias for android keystore file

keytool -v -list -keystore file.jks

keytool is in jre/bin

Sunday, January 29, 2017

Using Apache as forward proxy

<VirtualHost *:80>
  ServerName proxy.yourdomain.com
  ProxyRequests On
  SSLProxyEngine On

  ProxyPass        /revoke https://myca.com/revoke
  ProxyPassReverse /revoke https://myca.com/revoke

  <Location />
    Order Deny,Allow
    Allow from all
  </Location>
</VirtualHost>

using apache/nginx as reverse proxy server (map to different ports based on domain name)

Assuming nginx is running on port 82 and you want to serve nginx.domain.com with nginx.


<VirtualHost *:80>
    ServerAdmin me@mydomain.com
    ServerName nginx.domain.com
    ProxyPreserveHost On

    # setup the proxy
    <Proxy *>
        Order allow,deny
        Allow from all
    </Proxy>
    ProxyPass / http://localhost:82/
    ProxyPassReverse / http://localhost:82/
</VirtualHost>

CSRF/XSS summary

If you are using cookies for authentication, someone can embed URLs from your domain in a random webpage to trigger side effects.
For e.g. if you have a GET url for http://domain.com/logout and someone creates <img src="http://domain.com/logout"/> in his webpage and your user visits that page, he would be immediately logged out.
Similarly POST URLs can be embedded in <form> elements.

But if you are not using cookies, for e.g. you might be using JWT and storing the token in localstorage, you are safe.

Best solution: Don't use cookies for authentication

XSS
-----

A bad guy posts a message on a forum. Message contains a js script tag. Whenever anyone visits the forum, that javascript runs and steals that person's cookie.

Solution: Escape any html or user submitted content you publish on your website.

eternal bash history

# Eternal bash history.  # ---------------------  # Undocumented feature which sets the size to "unlimited".  # http://stackoverflow.com/questions/9457233/unlimited-bash-history  export HISTFILESIZE=  export HISTSIZE=  export HISTTIMEFORMAT="[%F %T] "  # Change the file location because certain bash sessions truncate .bash_history file upon close.  # http://superuser.com/questions/575479/bash-history-truncated-to-500-lines-on-each-login  export HISTFILE=~/.bash_eternal_history  # Force prompt to write history after every command.  # http://superuser.com/questions/20900/bash-history-loss  PROMPT_COMMAND="history -a; $PROMPT_COMMAND"

Saturday, January 28, 2017

windows docker

Windows 10 only
enable hyper-v
install docker

Commands:
docker run -it --entrypoint=/bin/bash dharm0us/ubuntu
Ctrl-PQ to quit without killing container

docker images to list images
docker ps to list containers
docker attach <container-id>

docker tag <image-id> dharm0us/ubuntu:latest
docker login
docker push dharm0us/ubuntu:latest

docker commit -m"test commit" <container-id> dharm0us/ubuntu:latest
docker push dharm0us/ubuntu:latest

Share directory
 docker run -v //c/projects:/projects  -it --entrypoint=/bin/bash dharm0us/ubuntu

Run with sudo
 docker run --privileged  -v //c/projects:/projects  -it --entrypoint=/bin/bash dharm0us/ubuntu

Run with port mapped
docker run --privileged -p 80:80 -v //c/projects:/projects  -it --entrypoint=/bin/bash dharm0us/ubuntu

Inspect container config
docker inspect <container_id>

Friday, January 27, 2017

setting up Go/GoLang on Windows

1. Download and install at C:/go
2. It should set GOROOT at C:/go, if it doesn't setx GOROOT C:/go
3. setx GOPATH C:/projects/go
4. cd C:/projects/go
5. mkdir src; mkdir src/proj
6. cd src/proj
7. touch main.go
8. touch constants.go
9. go get ./...
10. cd ..
11. go build proj/
12. ./proj.exe

main.go
package main

import (
       "fmt"       "github.com/ChimeraCoder/anaconda"       "net/url")

func main() {
       fmt.Println("hi")
       anaconda.SetConsumerKey(CONSUMER_KEY)
       anaconda.SetConsumerSecret(CONSUMER_SECRET)
       api := anaconda.NewTwitterApi(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)

       v := url.Values{}
       v.Set("track", "Kohli")
       twitterStream := api.PublicStreamFilter(v)
       for {
              x := <-twitterStream.C
              switch tweet := x.(type) {
              case anaconda.Tweet:
                     fmt.Println(tweet.Latitude())
                     fmt.Println(tweet.Text)
                     fmt.Println("-----------")
              case anaconda.StatusDeletionNotice:
              // pass              default:
                     fmt.Printf("unknown type(%T) : %v \n", x, x)
              }
       }
}
Constants.go
package main

const CONSUMER_KEY = ""const CONSUMER_SECRET = ""const ACCESS_TOKEN = ""const ACCESS_TOKEN_SECRET = ""


Saturday, January 21, 2017

intellij idea setting java language level to 8

Project Settings -> Modules -> Sources -> Language level : 8
Project Settings -> Project -> Project SDK -> 1.8
Project Settings -> Project -> Project Language Level -> 8
File > Settings > Build, Execution, Deployment > Java Compiler >
Project Bytecode version -> 1.8
File > Settings > Build, Execution, Deployment > Java Compiler > Per
module Bytecode version -> 1.8

Friday, January 20, 2017

today's summary

select/poll/epoll used by memcached/redis/nodejs/golang
epoll is latest and linux only.
30% of world's computers are still Windows XP.

nodejs vs golang async
golang - goroutines on top of OS threads, proud to be blocking (goroutine will take care of that), goroutines are cheap
nodejs - single threaded, use callback else it will be blocked

websocket server - Golang is the right compro, perf less than C++ but code much simpler. Nodejs far behind.

In GoLang - you can use utf-8 characters as var names

redis and tiles38 for geospatial data
tiles38 has replication as well and uses Redis RESP protocol

Thursday, January 19, 2017

nginx setup amazon linux

tar -xvzf nginx-1.11.8.tar.gz
cd nginx-1.11.8
./configure --sbin-path=/usr/local/sbin --with-http_ssl_module
make
make install


vim /usr/local/nginx/conf/nginx.conf

Add the following to server section:
   listen       80;
        server_name  domain.com;

        #charset koi8-r;

        #access_log  logs/host.access.log  main;

        location / {
            root  /var/www/html/folder;
            index  index.php index.html index.htm;
        }
        location ~ \.php$ {
                fastcgi_pass   unix:/var/run/php-fpm/php-fpm.sock;
                fastcgi_index  index.php;
                fastcgi_param  SCRIPT_FILENAME  /var/www/html/folder$fastcgi_script_name;
                include        fastcgi_params;
        }




nginx(simple to start) or /usr/local/sbin/nginx
nginx -s stop

sudo yum install php56-fpm
nginx -t to locate config file
service php-fpm start/stop/restart
vim  /etc/php-fpm.d/www.conf -> change user and group
vim /etc/php-fpm.conf

summing memory used on linux

 ps aux --sort rss | awk '{sum+=$5;print $5,sum}'
for RSS
 ps aux --sort rss | awk '{sum+=$4;print $4,sum}'
for VSZ

Blog Archive