Software Troubles and Troubleshooting: January 2017

Tuesday, January 31, 2017

Redis (Cluster) notes

Benchmark tool

Comes with Redis. Very good with variety of options.

Server on AWS (client on my local machine)

PS C:\Users\user> redis-benchmark -t set -n 100000 -h <MY_AWS_FREE_TIER_IP> -p 81

====== SET ======

100000 requests completed in 40.93 seconds

50 parallel clients

3 bytes payload

keep alive: 1

0.00% <= 14 milliseconds

0.22% <= 15 milliseconds

10.06% <= 16 milliseconds

24.73% <= 17 milliseconds

30.37% <= 18 milliseconds

32.67% <= 19 milliseconds

43.62% <= 20 milliseconds

63.83% <= 21 milliseconds

71.76% <= 22 milliseconds

74.68% <= 23 milliseconds

79.34% <= 24 milliseconds

89.36% <= 25 milliseconds

95.18% <= 26 milliseconds

97.42% <= 27 milliseconds

98.39% <= 28 milliseconds

98.87% <= 29 milliseconds

99.11% <= 30 milliseconds

99.36% <= 31 milliseconds

99.54% <= 32 milliseconds

99.65% <= 33 milliseconds

99.72% <= 34 milliseconds

99.80% <= 35 milliseconds

99.85% <= 36 milliseconds

99.89% <= 37 milliseconds

99.91% <= 38 milliseconds

99.93% <= 39 milliseconds

99.95% <= 40 milliseconds

99.96% <= 41 milliseconds

99.97% <= 42 milliseconds

99.98% <= 44 milliseconds

99.98% <= 45 milliseconds

100.00% <= 46 milliseconds

100.00% <= 50 milliseconds

2443.38 requests per second

Localhost: (both client and server)

PS C:\Users\user> redis-benchmark -t set -n 100000

====== SET ======

100000 requests completed in 1.42 seconds

50 parallel clients

3 bytes payload

keep alive: 1

89.97% <= 1 milliseconds

99.88% <= 2 milliseconds

99.91% <= 3 milliseconds

99.93% <= 4 milliseconds

99.95% <= 8 milliseconds

99.96% <= 9 milliseconds

99.98% <= 10 milliseconds

99.99% <= 11 milliseconds

99.99% <= 12 milliseconds

99.99% <= 13 milliseconds

99.99% <= 14 milliseconds

99.99% <= 15 milliseconds

100.00% <= 16 milliseconds

100.00% <= 17 milliseconds

100.00% <= 18 milliseconds

100.00% <= 19 milliseconds

70521.86 requests per second

With pipelining (QPS is much higher but latency for 90 percentile requests is much higher)

PS C:\Users\user> redis-benchmark -t set -n 100000 -P 100

====== SET ======

100000 requests completed in 0.21 seconds

50 parallel clients

3 bytes payload

keep alive: 1

0.00% <= 1 milliseconds

0.60% <= 2 milliseconds

1.00% <= 3 milliseconds

1.10% <= 4 milliseconds

1.30% <= 5 milliseconds

1.40% <= 6 milliseconds

10.70% <= 7 milliseconds

35.80% <= 8 milliseconds

46.70% <= 9 milliseconds

57.00% <= 10 milliseconds

69.70% <= 11 milliseconds

79.10% <= 12 milliseconds

86.60% <= 13 milliseconds

91.70% <= 14 milliseconds

94.00% <= 15 milliseconds

97.00% <= 16 milliseconds

99.30% <= 17 milliseconds

100.00% <= 17 milliseconds

467289.72 requests per second

Redis is single threaded, can fork another thread(process?) for persistence.
If running in cluster mode, one node should have n/2 instances (master + slave) where n = NUM_CORE since one process for serving commands, another for persistence
RDB vs AOF persistence
Slave can be configured to become master if master hasn't been contacted in a while.
Total ~16K Hashslots.
Resharding (redistribution of keys) is always manual, whether adding a node or deleting one. Failover is automatic since slave already has the the same keys as master.
Pipelining will increase throughput but 90-95 percentile latency will be very high. Essentially 100 percentile latency will be lower than non-pipeline version but for other percentiles it will be very high.
Others : Recently Geo commands were added. Though tiles38 is also there for the similar stuff.

Forgot alias for android keystore file

keytool -v -list -keystore file.jks

keytool is in jre/bin

Sunday, January 29, 2017

Using Apache as forward proxy

ServerName proxy.yourdomain.com

ProxyRequests On

SSLProxyEngine On

ProxyPass /revoke https://myca.com/revoke

ProxyPassReverse /revoke https://myca.com/revoke

Order Deny,Allow

Allow from all

</Location>

</VirtualHost>

using apache/nginx as reverse proxy server (map to different ports based on domain name)

Assuming nginx is running on port 82 and you want to serve nginx.domain.com with nginx.

ServerAdmin me@mydomain.com

ServerName nginx.domain.com

ProxyPreserveHost On

# setup the proxy

Order allow,deny

Allow from all

</Proxy>

ProxyPass / http://localhost:82/

ProxyPassReverse / http://localhost:82/

</VirtualHost>

http://serverfault.com/questions/195611/how-do-i-redirect-subdomains-to-a-different-port-on-the-same-server

Now let's reverse the roles, using Nginx as reverse proxy server and Apache as the real server:

https://www.digitalocean.com/community/tutorials/how-to-configure-nginx-as-a-web-server-and-reverse-proxy-for-apache-on-one-ubuntu-14-04-droplet

CSRF/XSS summary

If you are using cookies for authentication, someone can embed URLs from your domain in a random webpage to trigger side effects.

For e.g. if you have a GET url for http://domain.com/logout and someone creates <img src="http://domain.com/logout"/> in his webpage and your user visits that page, he would be immediately logged out.

Similarly POST URLs can be embedded in <form> elements.

But if you are not using cookies, for e.g. you might be using JWT and storing the token in localstorage, you are safe.

Best solution: Don't use cookies for authentication

XSS

-----

A bad guy posts a message on a forum. Message contains a js script tag. Whenever anyone visits the forum, that javascript runs and steals that person's cookie.

Solution: Escape any html or user submitted content you publish on your website.

eternal bash history

# Eternal bash history.  # ---------------------  # Undocumented feature which sets the size to "unlimited".  # http://stackoverflow.com/questions/9457233/unlimited-bash-history  export HISTFILESIZE=  export HISTSIZE=  export HISTTIMEFORMAT="[%F %T] "  # Change the file location because certain bash sessions truncate .bash_history file upon close.  # http://superuser.com/questions/575479/bash-history-truncated-to-500-lines-on-each-login  export HISTFILE=~/.bash_eternal_history  # Force prompt to write history after every command.  # http://superuser.com/questions/20900/bash-history-loss  PROMPT_COMMAND="history -a; $PROMPT_COMMAND"

Saturday, January 28, 2017

windows docker

Windows 10 only

enable hyper-v

install docker

Commands:

docker run -it --entrypoint=/bin/bash dharm0us/ubuntu

Ctrl-PQ to quit without killing container

docker images to list images

docker ps to list containers

docker attach <container-id>

docker tag <image-id> dharm0us/ubuntu:latest

docker login

docker push dharm0us/ubuntu:latest

docker commit -m"test commit" <container-id> dharm0us/ubuntu:latest

docker push dharm0us/ubuntu:latest

Share directory
docker run -v //c/projects:/projects -it --entrypoint=/bin/bash dharm0us/ubuntu

Run with sudo
docker run --privileged -v //c/projects:/projects -it --entrypoint=/bin/bash dharm0us/ubuntu

Run with port mapped
docker run --privileged -p 80:80 -v //c/projects:/projects -it --entrypoint=/bin/bash dharm0us/ubuntu

Inspect container config
docker inspect <container_id>

Friday, January 27, 2017

setting up Go/GoLang on Windows

1. Download and install at C:/go

2. It should set GOROOT at C:/go, if it doesn't setx GOROOT C:/go

3. setx GOPATH C:/projects/go

4. cd C:/projects/go

5. mkdir src; mkdir src/proj

6. cd src/proj

7. touch main.go

8. touch constants.go

9. go get ./...

10. cd ..

11. go build proj/
12. ./proj.exe

main.go

package main

import (
       "fmt"       "github.com/ChimeraCoder/anaconda"       "net/url")

func main() {
       fmt.Println("hi")
       anaconda.SetConsumerKey(CONSUMER_KEY)
       anaconda.SetConsumerSecret(CONSUMER_SECRET)
       api := anaconda.NewTwitterApi(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)

       v := url.Values{}
       v.Set("track", "Kohli")
       twitterStream := api.PublicStreamFilter(v)
       for {
              x := <-twitterStream.C
              switch tweet := x.(type) {
              case anaconda.Tweet:
                     fmt.Println(tweet.Latitude())
                     fmt.Println(tweet.Text)
                     fmt.Println("-----------")
              case anaconda.StatusDeletionNotice:
              // pass              default:
                     fmt.Printf("unknown type(%T) : %v \n", x, x)
              }
       }
}

Constants.go

package main

const CONSUMER_KEY = ""const CONSUMER_SECRET = ""const ACCESS_TOKEN = ""const ACCESS_TOKEN_SECRET = ""

Saturday, January 21, 2017

intellij idea setting java language level to 8

Project Settings -> Modules -> Sources -> Language level : 8
Project Settings -> Project -> Project SDK -> 1.8
Project Settings -> Project -> Project Language Level -> 8
File > Settings > Build, Execution, Deployment > Java Compiler >
Project Bytecode version -> 1.8
File > Settings > Build, Execution, Deployment > Java Compiler > Per
module Bytecode version -> 1.8