Sunday, September 6, 2020

free video editing tools

handbrake => burn subtitles into mp4
avidemux => cut video anywhere(not only the ends)
freemake => vob to mp4 converter
Posted by at 2 comments:

Thursday, August 13, 2020

grep extract only matching text

 cat filename | grep --extended-regexp --only-matching --regexp="data-text=\"[a-zA-Z0-9 ]*\""

I had an html where I wanted to extract text like "data-text=abcd"
Posted by at No comments:

Wednesday, August 5, 2020

html video player with zooming

<html><head><meta name="viewport" content="width=device-width"></head><body>
<style>
.video-container {
  position: absolute;
  top: 0;
  bottom: 0;
  width: 100%;
  height: 100%; 
  overflow: hidden;
}

.video-container video {
  /* Make video to at least 100% wide and tall */
  min-width: 100%; 
  min-height: 100%; 
  
  /* Setting width & height to auto prevents the browser from stretching or squishing the video */
  width: auto;
  height: auto;
  
  /* Center the video */
  position: absolute;
  top: 50%;
  left: 50%;
  transform: translate(-50%,-50%);
}


////////////////////////////////////////

body { 
  background: #000;
  color: #fff;
  font-family: 'Oxygen', sans-serif;
}
  
.overlay {
  background: rgba(0,0,0,0.5);
  position: absolute;
  bottom: 0;
  left: 0;
  right: 0;
  width: 95%;
  max-width: 45em;
  margin: auto auto 1em;
  box-sizing: border-box;
  padding: 2em;
  line-height: 1.5;
  text-align: center;
  
  :last-child { margin-bottom: 0; }

  h1 {
    font-size: 14pt;
    font-weight: normal;
    text-shadow: 0 0 .3em #000;
    margin: 0 0 1em;
  }

  p {
    font-size: 11pt;
    text-shadow: 0 0 .3em #000;
    margin: 1em 0;
  }
  a {
    color: #fff;
  }
}

code { font-family: monospace; }
</style>

<div class="video-container">
<video controls="" autoplay="" name="media"><source src="mp4link" type="video/mp4"></video>
</div>
</body></html>
Posted by at No comments:

Monday, July 27, 2020

Ignoring whitespace diff in Git GUI

Edit => Options => Additional Diff Parameters(Both in NRP repository
and Global) = -w --ignore-all-space
Posted by at No comments:

Thursday, April 23, 2020

Create a self signed cert, convert .pem to .cer


Create private key:
openssl genrsa -out privkey.pem 2048
 
See what's inside the private key:
openssl rsa -noout -text -in privkey.pem

Create public key using the private key
openssl req -new -x509 -key privkey.pem -out cacert.pem -days 3650

See what's inside the public key:
openssl x509 -in cacert.pem -text

Convert public key from .pem to .cer
openssl x509 -inform PEM -in cacert.pem -outform DER -out a.cer

Posted by at No comments:

Tuesday, April 21, 2020

System design interviews

Some common points:
1. Logging - async logging - traceId which is passed to every layer to enable better debugging and log collation. Hot/cold storage for logs - ssd(recent logs) vs hdd(older logs)

2. For media(videos) - how to handle long connections - LB will maintain state while backend undergoing change - for real time streaming, it's ok - just resume from current time. For static videos, resume from last known location. Use DSR, where LB is bypassed in the reverse direction and data directly goes to client.

3. Analytics - separate storage for archival. EMR jobs. HDFS. Pig scripts.

4. Caching - Distributed vs. each node having the full replica. In the second case, each one just listens for updates which flow from the master. How much RAM does each node have? Prevents a hop but not scalable. (Aerospike vs Redis distributed). Routing in the first case - client frequently syncs with the master for latest routing table updates so as to directly hit the backend vs. touching master for every call.

5. Queuing - Kafka. Producers/consumers/intermediate storage - each layer should be able to independently scale. How to partition for listeners? How much data to store waiting for consumers to consume. Each consumer should be able to go back and forth in the queue irrespective of how much has been processed.

6. LB - DNS(round robin/geo aware) -> L3(BGMP)->L4(TCP)->L7(http/redis). Passthru vs connection termination. DSR? SSL termination? Http2?

7. Search - Lucene/Autocomplete based on tries/

8. Social media content - Taiji - User clique based DC routing. A user is likely to consume content generated by friends. So cache data for a clique in the same DC.

9. Separate out read/write flows - scale indep'ly. 100:1 read:write ratio.

10. Redundancy - region/zone

11. News feed - push/pull/hybrid

12. DB partitioning - distribute tables? foreign keys? joins?

13. Security,permissions,file sharing.

14. Dropbox - files split in chunks - to better manage retries.  Exponential backoff. Mobile clients do pull based sync to preserve data/battery. Response queue for each client. Request queue global. Inline Dedupe vs post processing dedupe.

15. Yelp/Uber - quad tree

16. Ticketmaster - transaction isolation levels

17. 
Posted by at No comments:

Friday, March 13, 2020

Multi tier load balancing with Linux

Multi-tier load-balancing with Linux


Summary in my words:
 
Multiple L7 load balancers talk to multiple backend servers. Backend servers don't have to worry about path MTU discovery, TCP congestion control algorithms, avoidance of the TIME-WAIT state and various other low-level details. But how do you load balance the L7 LBs?

L3 load balancing can help. All L7 LBs can advertise the same IP to their nearby routers. With a combination of ECMP and BGP protocols, routers can load balance the packets to L7 LBs. But adding/removing an L7 LB in this setup breaks existing connections(which is problematic for long connections like downloads) since routers don't use consistent hashing. Also, if one router becomes unavailable, other routers may forward the packets to a different LB since every router decides individually.

To mitigate this, use L4 LBs between L3 and L7. The purpose of this tier is to ensure that all members take same scheduling decision for an incoming packet based on destination IP and port. Consistent hashing is used to route packets to L7 LBs so removal/addition of an L7 LB isn't problematic. Additionally L7 LBs return response directly to L3 routers(DSR) to improve perf.

So, routers use ECMP + BGP to route packets to L4 LBs(all of which advertise the same IP). L4 LBs use consistent hashing scheme to talk to L7 LBs. And L7 LBs return the response directly to L3.

Actual Snippets from the article:

A combination of:
 
  1. ECMP routing
  2. Stateless L4 load-balancing
  3. Stateful L7 load-balancing

L7 (Last Tier): 

Working in the highest layers of the OSI model, it can also offer additional services, like TLS-termination, HTTP routing, header rewriting, rate-limiting of unauthenticated users, and so on. Being stateful, it can leverage complex load-balancing algorithm. Being the first point of contact with backend servers, it should ease maintenances and minimize impact during daily changes.

It also terminates client TCP connections. This introduces some loose coupling between the load-balancing components and the backend servers with the following benefits:
  1. connections to servers can be kept open for lower resource use and latency,
  2. requests can be retried transparently in case of failure,
  3. clients can use a different IP protocol than servers,
  4. and servers do not have to care about path MTU discovery, TCP congestion control algorithms, avoidance of the TIME-WAIT state and various other low-level details.

As is, this solution is not complete. We have just moved the availability and scalability problem somewhere else. How do we load-balance the requests between the load-balancers?

First tier: ECMP routing (L3)
 
On most modern routed IP networks, redundant paths exist between clients and servers. For each packet, routers have to choose a path. When the cost associated to each path is equal, incoming flows are load-balanced among the available destinations. This characteristic can be used to balance connections among available load-balancers.

If we assume you already have BGP-enabled routers available, ExaBGP is a flexible solution to let the load-balancers advertise their availability.

Unfortunately, this solution is not resilient when an expected or unexpected change happens. Notably, when adding or removing a load-balancer, the number of available routes for a destination changes. The hashing algorithm used by routers is not consistent and flows are reshuffled among the available load-balancers, breaking existing connections.

Moreover, each router may choose its own routes. When a router becomes unavailable, the second one may route the same flows differently which again breaks existing connections.

If you think this is not an acceptable outcome, notably if you need to handle long connections like file downloads, video streaming or websocket connections, you need an additional tier. Keep reading!

Second tier: L4 load-balancing

The second tier is the glue between the stateless world of IP routers and the stateful land of L7 load-balancing. It is implemented with L4 load-balancing. The terminology can be a bit confusing here: this tier routes IP datagrams (no TCP termination) but the scheduler uses both destination IP and port to choose an available L7 load-balancer. The purpose of this tier is to ensure all members take the same scheduling decision for an incoming packet. There are two options:

  1. stateful L4 load-balancing with state synchronization across the members,
  2. or stateless L4 load-balancing with consistent hashing.

The first option increases complexity and limits scalability. We won’t use it. The second option is less resilient during some changes but can be enhanced with a hybrid approach using a local state.

We use IPVS, a performant L4 load-balancer running inside the Linux kernel, with Keepalived, a frontend to IPVS with a set of healthcheckers to kick out an unhealthy component. IPVS is configured to use the Maglev scheduler, a consistent hashing algorithm from Google. Among its family, this is a great algorithm because it spreads connections fairly, minimizes disruptions during changes and is quite fast at building its lookup table. Finally, to improve performance, we let the last tier—the L7 load-balancers—sends back answers directly to the clients without involving the second tier—the L4 load-balancers. This is referred to as direct server return (DSR) or direct routing (DR).
   
With such a setup, we expect packets from a flow to be able to move freely between the components of the first two tiers while sticking to the same L7 load-balancer.

Tier 0: DNS load-balancing

Optionally, you can add DNS load-balancing to the mix. This is useful either if your setup is spanned across multiple datacenters, or multiple cloud regions, or if you want to break a large load-balancing cluster into smaller ones. It is not intended to replace the first tier as it doesn’t share the same characteristics: load-balancing is unfair (it is not flow-based) and recovery from a failure is slow.



Posted by at No comments:
Subscribe to: Posts (Atom)

Blog Archive