Beautiful Soup

You didn't write that awful page. You're just trying to get some data
out of it. Beautiful Soup is here to help. Since 2004, it's been
saving programmers hours or days of work on quick-turnaround screen
scraping projects.

http://www.crummy.com/software/BeautifulSoup/

Installing PostGIS on centos 6.5

sudo rpm -ivh http://yum.postgresql.org/9.4/redhat/rhel-6-x86_64/pgdg-centos94-9.4-1.noarch.rpm

yum install postgresql94 postgresql94-server postgresql94-libs
postgresql94-contrib postgresql94-devel

sudo yum install postgis2_94

yum install pgrouting_94

yum install php-pgsql

service postgresql-9.4 initdb
----------------------------------------------------------------
connecting :

su postgres

psql -p 5432

SELECT setting FROM pg_settings WHERE name = 'config_file';

postgres=# create database shops_new;

CREATE DATABASE db1;
\l - show databases
\d <table>; describe table
\connect db1; connect to db1
\dt show tables


postgres=# \connect db_new;

You are now connected to database "db_new" as user "postgres".

db_new=# create extension postgis;

CREATE EXTENSION

db_new=#

ALTER TABLE your_table ADD COLUMN idBIGSERIAL PRIMARY KEY;


db_new=# ALTER TABLE table1 ADD COLUMN name character varying(128) NOT
NULL DEFAULT '';

db_new=# ALTER TABLE table1 ADD COLUMN address character varying(1024)
NOT NULL DEFAULT '';

ALTER TABLE table1 ADD COLUMN male int;

SELECT AddGeometryColumn ('','shops','location',4326,'POINT',2);


INSERT INTO shops(name,address,location,male,female,free,rating)
VALUES("name1","address1",ST_GeomFromText('POINT(-71.060316
48.432044)', 4326), 1,1,1,1);

http://www.cyberciti.biz/faq/psql-fatal-ident-authentication-failed-for-user/

twitter bulk unfollow

Open "Following" page on Twitter and scroll down to have all following accounts showed in the browser.
Next run the following JavaScript code in your development console:
$('.button-text.unfollow-text').trigger('click');

http://www.quora.com/How-can-you-unfollow-everyone-on-Twitter-at-once

Command for computing max number of requests per second

 cat access_log.2014-09-23.txt | awk '{print $4}' | sort | uniq -c | sort -n

(assuming $4 is the timestamp)

per hour (sorted on hour):
cat access_log.2015-02-08.txt | awk '{print substr($4,14,2)}' | sort | uniq -c | sort -n -k2

List of IPs with number of requests in a particular hour(15) :
 cat access_log.2015-03-09.txt | awk '{if('15' == substr($4,14,2)) print $1}' | sort | uniq -c | sort -n

Notes from Violent Python

1. The European Network and Information Security Agency provides an excellent resource for analyzing network traffic. They provide a live DVD ISO image that contains several network captures and exercises. You can download a copy from http://www.enisa.europa.eu/activities/cert/support/exercise/live-dvd-iso-images. 

2. NMap can be used in python via a library.

3. Pexpect(python) can be used for automating interactive applications - for e.g.   

4. ftplib for brute force ftp user credentials.

5. Metasploit as a penetration testing tool.

6. wigle.net for finding lat/long for a wifi router   

Python simple script to connect to a host/port and get response.

__author__ = 'admin'

import socket

s = socket.socket()

s.settimeout(3)

s.connect(("server.com",21))

ans  = s.recv(1024)

print ans

MySql db restore from dump failing

Solution : increase max_allowed_packet

Details :

Recently, I had a strange problem. While doing source dbdump.sql (dbdump.sql was generated by mysqldump) - initial data was loaded fine but after a while data was not being restored properly due to foreign key checks failing.

The dump file was disabling foreign key checks at the top and enabling again at the bottom  - so it was highly surprising.

To debug this I wrote a php script which executed queries line by line from the dump. I observed that it failed at a query where the data being inserted was huge which made database connection go away. And when it came back the foreign key check was enabled again - which in turn made a lot of things fail.

So I increased max_allowed_packet to 64M in my.ini and restarted mysql server. Solved.