grep exclude file extensions

grep SendBuildStatusUpdate -r --exclude=\*.{ini,cs} .

Advanced features

https://www.coursera.org/learn/competitive-data-science/lecture/mpCps/statistics-and-distance-based-features

Statistics and distance based features: groupby and nearest neighbor methods
Neighbors - for e.g. to predict rental prices, features could be number of schools/hospitals in a radius.
CTR example - ad price, ad position, user_id, page_id - you can use group by on user/page to add new features. Or even the previous history of the user.

Bray curtis metric.
-------------------------------------
Matrix factorizations: documents/words - dimensionality reduction.

mean encoding

A very popular/important video:
https://www.coursera.org/learn/competitive-data-science/lecture/LGYQ2/regularization

Mean encoding regularization
CV loop
LOO - Leave one out - using target variable to generate the new feature makes our encoding biased.

Smoothing.
Noise.
Expanding mean.
------------------
generalizations and extensions of mean encodings: for regression/multiclass.
Many to many relations: for e.g. classification of users based on the apps installed on their phones. Each user can have multiple apps, each app can be installed by many users. Hence, many-to-many relation.

In this case, convert data to long representation. So that, each row will have <user_id, app_id, target> like <uid1, app_id1, target(0 or 1)>. Now you can take mean of targets for every app. But how to map it back to users?

Interactions and numerical features -?

Regression/Classification metrics optimization

https://www.coursera.org/learn/competitive-data-science/lecture/SQ9Uq/regression-metrics-optimization

MAE (L1 metric/L1 loss) - some methods don't support this metric since second derivative is zero.
RMSLE

Classification metrics optimization

https://www.coursera.org/learn/competitive-data-science/lecture/hvDC5/classification-metrics-optimization-i

logloss - very popular(like MSE for regression) - all NNs by default optimize logloss for classification. RFs turn out to be very bad in terms of logloss but they can be made better.

logloss requires models to output posterior probabilities - but what does it mean? logloss is easy to implement.

accuracy -

installing dropbox on centos linux

yum install docker - y
alias dropbox="docker exec -it dropbox dropbox"
alias dropbox-start="docker run -d --restart=always --name=dropbox -v
/home/val/Dropbox:/dbox/Dropbox -v /home/val/.dropbox:/dbox/.dropbox
-e DBOX_UID=1000 -e DBOX_GID=100 janeczku/dropbox"
sudo systemctl start docker
dropbox-start
docker ps (will give you the pid for e.g. a6f6a1d0866f)
docker logs --follow a6f6a1d0866f (now you will see a web link to
connect dropbox to your pc)

metrics and loss function

Sometimes both are same for e.g. MSE. But at times they will be
different since it is hard to define loss function for some metrics.
In XgBoost you can write cusotm loss functions but they should have
smooth derivatives otherwise it will go crazy.

Early stopping - In case we don't know how to write a loss function
for the target metric, there is a simple solution - Early Stopping.
Keep optimizing the loss function but stop when the model starts over
fitting as per the target metric.

area under curve (AUC) simple explanation

Best explanation I have found is here:
https://www.coursera.org/learn/competitive-data-science/lecture/EhJzY/classification-metrics-review

1) Area under curve
For every threshold plot number of TPs on y-axis and FPs on x-axis. If
dataset can be clearly separated by a threshold then the AUC will be
1(max value).

2) Pair ordering
Consider all possible pairs such that one item is TP and another is
FP. AUC is the probability that the FP is ranked higher than TP (so
that threshold puts TPs on the left and FPs on the right).