<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.3.3">Jekyll</generator><link href="https://deepmusings.net/feed.xml" rel="self" type="application/atom+xml" /><link href="https://deepmusings.net/" rel="alternate" type="text/html" /><updated>2024-04-15T21:04:19-07:00</updated><id>https://deepmusings.net/feed.xml</id><title type="html">Deep Musings</title><subtitle>Personal blog for random personal thoughts by Bhavdeep Sethi</subtitle><author><name>Bhavdeep Sethi</name></author><entry><title type="html">Apache Spark - Can it be lupus?</title><link href="https://deepmusings.net/2015/08/16/apache-spark-can-it-be-lupus.html" rel="alternate" type="text/html" title="Apache Spark - Can it be lupus?" /><published>2015-08-16T00:50:31-07:00</published><updated>2015-08-16T00:50:31-07:00</updated><id>https://deepmusings.net/2015/08/16/apache-spark-can-it-be-lupus</id><content type="html" xml:base="https://deepmusings.net/2015/08/16/apache-spark-can-it-be-lupus.html"><![CDATA[<p>
Hmm...so you're trying to diagnose problems in your Spark Job and can't seem to make progress? I've been there.
Working with Spark is similar to working with a Machine Learning algorithm with respect to the number of (hyper) parameters.
Tuning them properly can be really hard initially. But, if you understand the semantics behind each of them, use it properly, you'll see it's true potential. I can't stress on how much of an improvement it is over the standard Map Reduce paradigm.</p>

<p><br />
I’ve played with Spark a lot over the last month. Here are some of the things I’ve learned that may help you cure your ailing Spark Job.
<br />
<!--more--></p>

<p>Start by reading these:</p>

<ul>
  <li><a href="http://spark.apache.org/docs/latest/tuning.html" target="_blank">Tuning Spark</a> (You may want to change to the Spark Version currently deployed)</li>
  <li><a href="http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-1/" target="_blank">Tune your Spark Jobs Part 1</a></li>
  <li><a href="http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/" target="_blank">Tune your Spark Jobs Part 2</a></li>
</ul>

<p><br /></p>
<hr />

<h3>Not all Spark programs are created equal</h3>

<p>Spark has a lot of parameters that you can play with. Each can have quite an impact on the running time of your spark job. The four parameters that are probably the most critical are, <code> --num-executors, --driver-memory, --executor-memory and --executor-cores</code>. Just increasing the num-executors does not mean your job will run faster. When you persist/cache any RDD, that is shared by all tasks running in an executor. So in most cases, increasing the parallelism (executor-cores) will have more effect as they can access the same cache. However, make sure you change executor-memory accordingly. The executor memory is shared among all the tasks. So if you have 16g and you can run 3 tasks in parallel, then each task will get around 5G. If you notice that only a fraction of your RDD is getting cached, you need to increase the amount of partitions it can access. Increasing the number of executors and reducing the executor-cores might help with this.</p>

<p><br /></p>
<hr />

<h3> Things to be aware of when you use variables.</h3>
<p>If you’re passing values via command line arguments to your Spark jobs, you need to be extra careful. The values that you pass are only accessible to the driver class. If you’re using Flags with default values, then the executors will receive the default values and not the ones you passed! For mandatory flags, it will just pass null. To avoid such cases, you can broadcast these flags and access them using the broadcast variables.
<br /></p>
<hr />

<h3>Kryo. Always.</h3>
<p>
Always always use Kryo. You can check the difference between Kryo and Java Serialization <a href="https://github.com/eishay/jvm-serializers/wiki" target="_blank">here</a>. When caching, it will use less far less space and will still be much faster. Seriously, don't use java serialization. </p>
<p><br /></p>
<hr />

<h3>StackOverflow problems? Cut the DAG.</h3>
<p>If you have long DAG of RDD objects, it needs to be serialized as part of the task creation. Serializing this leads to stack overflow error. It's generally common with algorithms where you perform iterations on the same RDD (eg. ALS). To fix these errors, you need to enable check-pointing (sparkContext.setCheckpointDir) and do a rdd.count() to cut the DAG.
</p>
<p><br /></p>
<hr />

<h3>Java heap space/OOM issues. </h3>
<p>
There are many variations of these errors. Typical ones are "Out of Java heap Space", "OOM errors". But, there are ones that are more subtle like "Failed without being ACK'd", "Futures timed out after [n seconds]".  These generally happen when the threads are busy with GC and couldn't send the heartbeat message to the driver.  A good way to debug these are by going through the GC logs. <br />
<pre> spark.executor.extraJavaOptions=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps </pre>
<br />
You can tweak your configuration based on the behavior seen by the gc logs. If there are lot of Full GC cycles happening, then you tuning config parameters like "spark.storage.memoryFraction" may help. For the timeout errors, changing the following defaults: <pre>spark.akka.frameSize=120</pre><pre>spark.rpc.askTimeout=120</pre></p>
<p><br /></p>
<hr />

<h3>Everyday I'm shuffling.</h3>
<p>When doing operations like joins, intersection, cartesian, etc. consider partitioning your RDD using a Hash Partitioner. In particular, if you’re joining a large RDD with a relatively smaller one, consider partitioning the larger RDD at the outset. This will ensure that only the elements of the smaller RDD are shuffled. &lt;/p&gt;
<br /></p>
<hr />

<h3>Broadcast it! </h3>
<p>Broadcasting data can make your jobs really really fast. It obviously helps knowing what data size you're dealing with. Assume you're taking a cartesian between two RDDS, one of size 300k, other 100k. The resultant RDD would be 90 x 10<sup>9</sup>. That's 90 billion pairs. The network shuffle required here will most likely kill your Spark job. However, if you broadcast the 100K RDD, creating the cartesian takes less than 5 minutes as it involves a simple flatMap operation over each element in it's own partition. </p>
<p><br /></p>
<hr />

<h3>Prefer Integers over Strings</h3>
<p>Prefer using Integers over Strings and large objects since it affects serialization/shuffle times. If you need to use a large object simply as a key, or in cases where your algorithm demands integers, we can use zipWithIndex to create a object -&gt; integer (with the reverse mapping) RDD and cache it. If it's small enough, just create a BiMap and broadcast it. </p>
<p><br /></p>
<hr />

<h3>Be Careful with Enums!</h3>
<p>When you use transformations like reduceByKey,countByKey,etc. the internal implementation of Spark uses hashCode to do the calculations. Now,
Java Enums do not have consistent hash code across different VMs. Now, since the enum keys will be distributed across different executors, even same keys may have different hash keys! This will lead to unexpected results. If you collect this result, the keys will be sent to the driver, and the now the keys which had different hash codes will get a consistent hash and will override each other. A simple way to avoid this is using the name/toString method for the keys.</p>
<p>
Spark is really really powerful. Use it properly, and you can solve most of your problems.
Remember, it's never lupus!
</p>
<p><img src="/assets/images/its-not-lupus.jpg" /></p>]]></content><author><name>Bhavdeep Sethi</name></author><category term="engineering" /><category term="big data" /><summary type="html"><![CDATA[Hmm...so you're trying to diagnose problems in your Spark Job and can't seem to make progress? I've been there. Working with Spark is similar to working with a Machine Learning algorithm with respect to the number of (hyper) parameters. Tuning them properly can be really hard initially. But, if you understand the semantics behind each of them, use it properly, you'll see it's true potential. I can't stress on how much of an improvement it is over the standard Map Reduce paradigm.]]></summary></entry><entry><title type="html">Monolithic Repo vs Micro Repos</title><link href="https://deepmusings.net/2015/06/06/single-repo-vs-multiple-repos.html" rel="alternate" type="text/html" title="Monolithic Repo vs Micro Repos" /><published>2015-06-06T22:21:51-07:00</published><updated>2015-06-06T22:21:51-07:00</updated><id>https://deepmusings.net/2015/06/06/single-repo-vs-multiple-repos</id><content type="html" xml:base="https://deepmusings.net/2015/06/06/single-repo-vs-multiple-repos.html"><![CDATA[<p>Recently, I have been working with a single repo which had the entire codebase of the company. It was basically a hierarchial maven project with each service being a sub-project/module. Prior to this, I’ve always worked in an environment where each service had it’s own repository. I guess there are pros and cons of each method.</p>

<p>Some of the points in favor of the single codebase approach I’ve heard are:</p>

<!--more -->

<ul>
  <li>You don’t have to worry about versioning. You end up having a single version for your product. This helps communications with QA, clients and managers. Since everyone has to pull in the same changes, everyone has access to the same thrift sources as well.</li>
  <li>Increased collaboration. Since, you have access to the entire codebase, you can look/learn/contribute to the code written by other teams as well.</li>
  <li>Reduces code duplication. Code discovery is much easier and reduces the amount of code duplicated across teams.</li>
</ul>

<p>However, there are problems with this approach as well:</p>

<ul>
  <li>Every time someone modifies code in a separate module, you have to recompile/generate sources to fix the compilation errors. Not to mention compiling takes a lot of time as well.</li>
  <li>I’ve seen Intellij giving weird path errors even though everything is fine. A synchronize with re-importing the pom fixes this but, it’s still annoying.</li>
  <li>Unnecessary dependency. I’ve seen people continue using a particular framework because it’s available and the entire code base uses it. A lot of times people tend to force their solutions to the existing framework. With Micro repos, each service can use the framework best suited for the job.</li>
</ul>

<p>I personally prefer working with separate repos. It’s a lot cleaner and for me, it symbolizes true service oriented architecture.Considering most of the time I’ll be developing on my local machine, I need that environment to be super fast. Having a good build tool probably makes the distinction between single vs micro repos even more obscure.</p>

<p>Yet, the monolithic repo seems to be the pre-dominant approach. I believe Google and Facebook do this. But the build environments in these companies are quite matured as well. So, what about the startups out there? Is there a preference?</p>]]></content><author><name>Bhavdeep Sethi</name></author><category term="engineering" /><category term="git" /><summary type="html"><![CDATA[Recently, I have been working with a single repo which had the entire codebase of the company. It was basically a hierarchial maven project with each service being a sub-project/module. Prior to this, I’ve always worked in an environment where each service had it’s own repository. I guess there are pros and cons of each method.]]></summary></entry><entry><title type="html">Deep Learning</title><link href="https://deepmusings.net/2015/05/23/deep-learning.html" rel="alternate" type="text/html" title="Deep Learning" /><published>2015-05-23T09:25:57-07:00</published><updated>2015-05-23T09:25:57-07:00</updated><id>https://deepmusings.net/2015/05/23/deep-learning</id><content type="html" xml:base="https://deepmusings.net/2015/05/23/deep-learning.html"><![CDATA[<p>
As part of the final project for <a href="http://www.cs.columbia.edu/~jebara/4772/" target="_blank">COMS W4772: Advance Machine Learning</a>, a course I took at Columbia University during Spring 2015, my friend Rhea and I were exploring various adaptive learning techniques.  The idea was to study and compare various adaptive learning methods on different datasets and evaluate the pros and cons of each.
</p>

<p>
We kept the Stochastic Gradient Descent (SGD) as the baseline and compared it with Momentum, Nestorov's Accelerated Gradient (NAG), AdaGrad, and AdaDelta on the Digits (digit image data for OCR task), 20newsgroup (document collection) and Labeled Faces  in the Wild (LFW) (face image data) datasets.
</p>
<!--more-->
<p>You can read the final report <a href="/assets/AML_Project_Report.pdf" target="_blank">here</a>!
</p>

<p>Also, as part of another research project, I was studying the effects of the different hyper-parameters on learning in a deep neural network. Using synthetic data, I was evaluating the role of learning rate, batch size, activation functions, hidden units, parameter initialization, learning methods in the working of a neural network. I then ran experiments on 20newsgroup data set to compare the final results as compared to SVM and Naive Bayes.</p>

<p>A TL;DR version is as follows:</p>

<ul>
  <li>Too high learning rate causes divergence; too low causes to learn very slowly. Try various values to see which gives you faster convergence. A better approach is to use adaptive learning (explained later).</li>
  <li>Too high batch size (behaves like standard gradient descent) is computationally slow as it processes many training examples for each update. Too low batch size (behaves like online learning) learns much faster but affected by noise. A moderate value (10%-30% of data size) works well.</li>
  <li>Use Rectified Linear Units (ReLUs) or tanh functions.</li>
  <li>Increasing the hidden units let’s you learn complex functions. Increasing too much may cause over-fitting. Choose a value using a validation set.</li>
  <li>Simplest approach is to initialize randomly using a Gaussian distribution with mean 0 and standard deviation 1.</li>
  <li>Use adaptive learning techniques like AdaGrad, AdaDelta, etc. to let learning rate vary per dimension. This is very important for deep neural nets since the magnitude of gradients differ a lot between different layers. Eliminates the need of a global learning rate.
<br /></li>
</ul>

<p>The longer version can be found <a href="/assets/DeepLearning_Project.pdf" target="_blank">here!</a></p>]]></content><author><name>Bhavdeep Sethi</name></author><category term="engineering" /><category term="machine learning" /><category term="columbia" /><summary type="html"><![CDATA[As part of the final project for COMS W4772: Advance Machine Learning, a course I took at Columbia University during Spring 2015, my friend Rhea and I were exploring various adaptive learning techniques. The idea was to study and compare various adaptive learning methods on different datasets and evaluate the pros and cons of each.]]></summary></entry><entry><title type="html">My Internship Experience</title><link href="https://deepmusings.net/2015/05/10/my-internship-experience.html" rel="alternate" type="text/html" title="My Internship Experience" /><published>2015-05-10T17:05:12-07:00</published><updated>2015-05-10T17:05:12-07:00</updated><id>https://deepmusings.net/2015/05/10/my-internship-experience</id><content type="html" xml:base="https://deepmusings.net/2015/05/10/my-internship-experience.html"><![CDATA[<p>
The first career fair I ever attended in Columbia was somewhere around October 17, 2015. It was an <i>interesting</i> experience to say the least. There was so much energy/excitement/ambitions...it was like an adrenaline rush... and of course, how can we forget the free swag. ;)
</p>
<p>The first few interview calls I got because of that event were from Snapchat and Dataminr. While I’ve been conducting interviews from the past three years, I don’t have much experience sitting on the other end of the table. And this showed on my first interview with Snapchat. <br /></p>

<pre>
Snapchat guy: Hey, do you use Snapchat?
Me: -looking at my Windows phone- Umm...not really.
Snapchat guy: So, what do you want to do here?
Me: Um...that's a good question.
</pre>

<!-- more -->

<p>I was supposed to code using coderpad, but I wasn’t aware that it compiles the program on saving it. Ended up writing code like you do on a piece of paper. By the end of it, code had compilation errors as well. Needless to say, I tanked the Snapchat interview. I realized jumping into the interview process without preparing wasn’t such a good idea. <br /></p>

<p>The next interview scheduled was with Dataminr. Learning from my last experience, I had started preparing with CTCI/leetcode and felt better prepared this time. The interview went pretty well, as far as I was concerned, but I didn’t hear anything from them for a month. I reached out to the HR for an update and they said they didn’t want to move ahead. Hmm…alright then. Least they could have done was informed me. But these things happen quite often from what I’ve read online. <br /></p>

<p>I had also applied to a couple of companies via the college portal. Got a call from the GoldmanSachs Strats group because of that. After the initial screening, they invited me for the SuperDay. The interviews went pretty well and got an offer from them after two weeks. Honestly speaking, I wasn’t really keen on it. I really wanted to work in a startup so I stopped giving interviews for the other companies. <br /></p>

<p>I participated in the FirstMark Elite(Venture Capitalist) internship program and was invited to attend their event. At the event, I interviewed with Symphony Commerce, Conductor and Live Gamer. I had an amazing time talking to each of these companies. Eventually, I decided to move ahead with Symphony Commerce. <br /></p>

<p>I had two phone interviews with Symphony Commerce. The first one was with Stanley Chan, Head of Core &amp; Infrastructure. The next interview was with Art Rivilis, Head of Engineerying at Symphony. I had a really good technical discussion with both of them and the projects they mentioned seemed really exciting. I got an offer from them and had I not accepted my next offer, I would definitely be going here. It looks like a really fun bunch of guys doing really kick-ass work. Stan was really helpful every time I spoke to him. I hope I can work with him in the future. <br /></p>

<p>Sometime before the FirstMark Elite event, Shannon Harrington from TellApart had reached out to me. I hadn’t heard about the company before so I checked out their company blogs. I realized their work was pretty similar to my last project (User Insights) at Flipkart. Got on a call with Shannon and I was blown away. She is by far the most amazing HR person I’ve worked with. I was discussing lambda architecture with her on my first call. Hell, I’ve not been able to do that with some of the most brilliant engineers I’ve worked with. I knew I had found the startup I wanted to join. <br /></p>

<p>TellApart took a total of five interviews. Yes, for an internship! They mentioned they had pretty high standards and were very serious about it. Hmm…Challenge accepted. ;)</p>

<p>After two phone interviews, they flew me to their Burlingame office for three more onsite interviews. Each round had two interviewers. The first one was taken by Mike Chang and Matt Forbes and it consisted of reviewing piece of code they had shared with me few days back. The scope of the problem soon changed and we were discussing how the code will adapt to it.
The second was with the team I was supposed to join. Jesh Bratman and Sean Xie deep dived into my User Insights project adding new scenarios along the way. The last one with Wade (VP of Engg) and Kevin (Engineering Manager) was more around culture fit. <br /></p>

<p>Got a spot offer from them and it was too hard to say no. I turned down an offer from Amazon and interview calls from Pinterest and Facebook for this. TellApart was my top choice and I was just glad it all worked out. <br /></p>

<p>A few days back Twitter acquired TellApart. I was expecting this to happen. But I was expecting Facebook to do it. Also, my timeline was like a year off. Either way, so now I’ll be interning with the TellApart team at Twitter. Not exactly a startup, but should be a good experience nonetheless :) I’ll write a post about how that goes!</p>]]></content><author><name>Bhavdeep Sethi</name></author><category term="personal" /><summary type="html"><![CDATA[The first career fair I ever attended in Columbia was somewhere around October 17, 2015. It was an interesting experience to say the least. There was so much energy/excitement/ambitions...it was like an adrenaline rush... and of course, how can we forget the free swag. ;) The first few interview calls I got because of that event were from Snapchat and Dataminr. While I’ve been conducting interviews from the past three years, I don’t have much experience sitting on the other end of the table. And this showed on my first interview with Snapchat.]]></summary></entry><entry><title type="html">Interesting Papers from Cloud &amp;amp; Big Data</title><link href="https://deepmusings.net/2015/05/03/interesting-papers-from-cloud-and-big-data.html" rel="alternate" type="text/html" title="Interesting Papers from Cloud &amp;amp; Big Data" /><published>2015-05-03T08:13:07-07:00</published><updated>2015-05-03T08:13:07-07:00</updated><id>https://deepmusings.net/2015/05/03/interesting-papers-from-cloud-and-big-data</id><content type="html" xml:base="https://deepmusings.net/2015/05/03/interesting-papers-from-cloud-and-big-data.html"><![CDATA[<p>
One of courses that I took in Columbia, with a really good course <i><b>content</b></i>, was Cloud and Big Data.
I took the course in Fall '14 and TA'ed it in Spring '15.
Every week, two papers were released that the students were supposed to read and submit a short summary about.
</p>
<p>
These are some really good papers and if you're interested in Distributed Systems, you'll really love reading them.

I'm sharing the list here (in no particular order) and hope you enjoy reading them!

</p>
<!--more-->
<p>
<h3> Cloud &amp; Big Data Reading Paper List </h3>
</p>

<p>
<ul>
<li>
<b>The Google File System</b><br />
Ghemawat, Sanjay;Gobioff, Howard; and Leung, Shun-Tak. ACM SIGOPS Operating Systems Review, 37(5) . 29-43.<br />
<a href="http://static.googleusercontent.com/media/research.google.com/en/us/archive/gfs-sosp2003.pdf">GFS: Google File System</a><br />
</li>
<br /><br />
<li>
<b>CloudCmp: Comparing Public Cloud Providers</b><br />
Ang Li; Xiaowei Yang; Duke University, {angl, xwy}@cs.duke.edu; Srikanth Kandula; Ming Zhang; Microsoft Research {srikanth, mzh}@microsoft.com<br />
<a href="https://www.cs.duke.edu/~angl/papers/imc10-cloudcmp.pdf">CloudCmp: Comparing Public Cloud Providers</a><br />
</li>
<br /><br />
<li>
<b>Xen and the Art of Virtualization</b><br />
Barham, Paul;Dragovic, Boris;Fraser, Keir;Hand, Steven;Harris, Tim;Ho, Alex;Neugebauer, Rolf;Pratt, Ian; and Warfield, Andrew. ACM SIGOPS Operating Systems Review, 37(5) . 164-177.<br />
<a href="http://www.cl.cam.ac.uk/research/srg/netos/papers/2003-xensosp.pdf">Xen and the Art of Virtualization</a><br />
</li>
<br /><br />
<li>
<b>Bigtable: A Distributed Storage System for Structured Data</b><br />
Chang, Fay;Dean, Jeffrey;Ghemawat, Sanjay;Hsieh, Wilson C;Wallach, Deborah A;Burrows, Mike;Chandra, Tushar;Fikes, Andrew; and Gruber, Robert E. ACM Transactions on Computer Systems (TOCS), 26(2) 2008.<br />
<a href="http://static.googleusercontent.com/media/research.google.com/en/us/archive/bigtable-osdi06.pdf">Bigtable: A Distributed Storage System for Structured Data</a><br />
</li>
<br /><br />
<li>
<b>Live Migration of Virtual Machines</b><br />
Clark, Christopher;Fraser, Keir;Hand, Steven;Hansen, Jacob Gorm;Jul, Eric;Limpach, Christian;Pratt, Ian; and Warfield, Andrew. Proceedings of the 2nd conference on Symposium on Networked Systems Design &amp; Implementation-Volume 2. 273-286.<br />
<a href="https://www.usenix.org/legacy/events/nsdi05/tech/full_papers/clark/clark.pdf?q=live-migration-of-virtual-machines">Live Migration of Virtual Machines</a><br />
</li>
<br /><br />
<li>
<b>Cassandra—A Decentralized Structured Storage System</b><br />
Lakshman, Avinash and Malik, Prashant. Operating systems review, 44(2) 2010. 35.<br />
<a href="https://www.cs.cornell.edu/projects/ladis2009/papers/lakshman-ladis2009.pdf">Cassandra: A Decentralized Structured Storage System</a><br />
</li>
<br /><br />
<li>
<b>Dynamo: Amazon's Highly Available Key-Value Store</b><br />
DeCandia, Giuseppe;Hastorun, Deniz;Jampani, Madan;Kakulapati, Gunavardhan;Lakshman, Avinash;Pilchin, Alex;Sivasubramanian, Swaminathan;Vosshall, Peter; and Vogels, Werner. ACM SIGOPS Operating Systems Review, 41(6) . 205-220.<br />
<a href="http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf">Dynamo: Amazon's Highly Available Key-Value Store</a><br />
</li>
<br /><br />
<li>
<b>Serving Large-scale Batch Computed Data with Project Voldemort</b><br />
Roshan Sumbaly; Jay Kreps; Lei Gao; Alex Feinberg; Chinmay Soman; Sam Shah;<br />
LinkedIn, Usenix<br />
<a href="https://www.usenix.org/legacy/events/fast/tech/full_papers/Sumbaly.pdf">Serving Large-scale Batch Computed Data with Project Voldemort</a><br />
</li>
<br /><br />
<li>
<b>PNUTS: Yahoo!’s Hosted Data Serving Platform</b><br />
Brian F. Cooper, Raghu Ramakrishnan, Utkarsh Srivastava, Adam Silberstein, Philip Bohannon, HansArno Jacobsen, Nick Puz,
Daniel Weaver and Ramana Yerneni <br />
Yahoo! Research.<br />
<a href="http://www.mpi-sws.org/~druschel/courses/ds/papers/cooper-pnuts.pdf">PNUTS: Yahoo!’s Hosted Data Serving Platform</a><br />
</li>
<br /><br />
<li>
<b>Hive: A Warehousing Solution Over a Map-Reduce Framework</b><br />
Thusoo, Ashish;Sarma, Joydeep Sen;
Jain, Namit;Shao, Zheng;Chakka, Prasad;Anthony, Suresh;Liu, Hao;Wyckoff, Pete; and
Murthy, Raghotham. Proceedings of the VLDB Endowment, 2(2) 2009. 1626-1629. <br />
<a href="http://www.vldb.org/pvldb/2/vldb09-938.pdf">Hive: A Warehousing Solution Over a Map-Reduce Framework</a><br />
</li>
<br /><br />
<li>
<b>MapReduce: Simplified Data Processing on Large Clusters</b><br />
Dean, Jeffrey and Ghemawat, Sanjay. Communications of the ACM, 51(1) 2008. 107-113. <br />
<a href="http://static.googleusercontent.com/media/research.google.com/en/us/archive/mapreduce-osdi04.pdf">MapReduce: Simplified Data Processing on Large Clusters</a><br />
</li>
<br /><br />
<li>
<b>An Analysis of Facebook Photo Caching</b><br />
Qi Huang, Ken Birman, Robbert van Renesse (Cornell University), Wyatt Lloyd (Princeton University), Sanjeev Kumar, Harry C. Li (Facebook Inc.)<br />
<a href="http://www.cs.cornell.edu/~qhuang/papers/sosp_fbanalysis.pdf">An Analysis of Facebook Photo Caching</a><br />
</li>
<br /><br />
<li>
<b>Scaling Memcache at Facebook</b><br />
Rajesh Nishtala, Hans Fugal, Steven Grimm, Marc Kwiatkowski, Herman Lee, Harry C. Li, Ryan McElroy, Mike Paleczny, Daniel Peek, Paul Saab, David Stafford, Tony Tung, and Venkateshwaran Venkataramani, Facebook Inc. NSDI 2013<br />
<a href="https://cs.uwaterloo.ca/~brecht/courses/854-Emerging-2014/readings/key-value/fb-memcached-nsdi-2013.pdf">Scaling Memcache at Facebook</a><br />
</li>
<br /><br />
<li>
<b>Hey, You, Get Off of My Cloud: Exploring Information Leakage in Third-Party Compute Clouds</b><br />
Thomas Ristenpart, Eran Tromer, Hovav Shacham, Stefan Savage<br />
<a href="https://cseweb.ucsd.edu/~hovav/dist/cloudsec.pdf">Hey, You, Get Off of My Cloud: Exploring Information Leakage in Third-Party Compute Clouds</a><br />
</li>
<br /><br />
<li>
<b>Apache Kafka: a Distributed Messaging System for Log Processing</b><br />
Jay Kreps, Neha Narkhede, Jun Rao<br />
<a href="http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf">Apache Kafka: a Distributed Messaging System for Log Processing</a><br />
</li>
<br /><br />
<li>
<b>Spark: Cluster Computing with Working Sets</b><br />
Matei Zaharia, Mosharaf Chowdhury, Michael J. Franklin, Scott Shenker, Ion Stoica, University of California, Berkeley<br />
<a href="http://www.cs.berkeley.edu/~matei/papers/2010/hotcloud_spark.pdf">Spark: Cluster Computing with Working Sets</a><br />
</li>
<br /><br />
<li>
<b>Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing</b><br />
Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J. Franklin, Scott Shenker, Ion Stoica
University of California, Berkeley<br />
<a href="https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf">RDD: A Fault-Tolerant Abstraction for In-Memory Cluster Computing</a><br />
</li>
<br /><br />
<li>
<b>Discretized Streams: Fault-Tolerant Streaming Computation at Scale</b><br />
Matei Zaharia, Tathagata Das, Haoyuan Li, Timothy Hunter, Scott Shenker, Ion Stoica, University of California, Berkeley<br />
<a href="https://www.cs.berkeley.edu/~matei/papers/2013/sosp_spark_streaming.pdf">Discretized Streams: Fault-Tolerant Streaming Computation at Scale</a><br />
</li>
<br /><br />
<li>
<b>Shark: SQL and Rich Analytics at Scale</b><br />
Reynold S. Xin, Josh Rosen, Matei Zaharia, Michael J. Franklin, Scott Shenker, Ion Stoica<br />
<a href="https://www.cs.berkeley.edu/~matei/papers/2013/sigmod_shark.pdf">Shark: SQL and Rich Analytics at Scale</a><br />
</li>
</ul>
<br /><br />

</p>]]></content><author><name>Bhavdeep Sethi</name></author><category term="engineering" /><category term="distributed systems" /><category term="bigdata" /><category term="nosql" /><category term="cloud" /><summary type="html"><![CDATA[One of courses that I took in Columbia, with a really good course content, was Cloud and Big Data. I took the course in Fall '14 and TA'ed it in Spring '15. Every week, two papers were released that the students were supposed to read and submit a short summary about. These are some really good papers and if you're interested in Distributed Systems, you'll really love reading them.]]></summary></entry><entry><title type="html">My Personal Iron Blogger</title><link href="https://deepmusings.net/2015/04/24/my-personal-iron-blogger.html" rel="alternate" type="text/html" title="My Personal Iron Blogger" /><published>2015-04-24T13:28:52-07:00</published><updated>2015-04-24T13:28:52-07:00</updated><id>https://deepmusings.net/2015/04/24/my-personal-iron-blogger</id><content type="html" xml:base="https://deepmusings.net/2015/04/24/my-personal-iron-blogger.html"><![CDATA[<p>
I noticed one of my <a href="http://www.sheki.in/">friends</a> had joined <a href="http://iron-blogger-sf.com/the-rules/">Iron blogger</a>. The idea is that you write a blog post every week. If you miss a week, you add $5 to a common pool. Even though he has been slacking for the past few weeks, I still think it's a pretty cool way to keep you motivated. I tried looking for the NY chapter, couldn't find it and then got busy with assignments. </p>

<p>
Few days back, during one of the random tweet session with <a href="https://twitter.com/sushibar_92">Surashree</a> (let's stalk her together) , she and I decided that we're gonna prod each other to come up with a new post every week. So, now I have my own personal Iron Blogger. Heh. How about that! I think this is better than the original group approach because we can actually harass each other if we don't follow this. The downside is you can expect lots of spam from of us henceforth. ;)
</p>
<p>
Hmm...still need to come up with some sort of penalty for missing a post though.
</p>

<p>
<i>P.S. Surashree, does this post count? :p</i>
</p>]]></content><author><name>Bhavdeep Sethi</name></author><category term="personal" /><summary type="html"><![CDATA[I noticed one of my friends had joined Iron blogger. The idea is that you write a blog post every week. If you miss a week, you add $5 to a common pool. Even though he has been slacking for the past few weeks, I still think it's a pretty cool way to keep you motivated. I tried looking for the NY chapter, couldn't find it and then got busy with assignments.]]></summary></entry><entry><title type="html">Tryst With Destiny</title><link href="https://deepmusings.net/2015/03/16/tryst-with-destiny.html" rel="alternate" type="text/html" title="Tryst With Destiny" /><published>2015-03-16T20:51:48-07:00</published><updated>2015-03-16T20:51:48-07:00</updated><id>https://deepmusings.net/2015/03/16/tryst-with-destiny</id><content type="html" xml:base="https://deepmusings.net/2015/03/16/tryst-with-destiny.html"><![CDATA[<p>Do you believe in destiny? Do you believe that things that are meant to happen, will happen, regardless of what you do? Does fate decide the course of your life Or are you responsible for your future and there is nothing called as luck? Personally, I'm not sure what to think. I do know that the last four years have completely changed my life. And thinking about it, all of this came down to a single decision, sometime in April 2011. </p>

<!--more-->

<p>I come from a middle class family. Right from childhood, I’ve always dreamt of reaching a point where money is not an issue.
When I was a kid, I used to love visiting my cousin because he always had the latest gaming consoles and toys. I used to envy him so much. I always used to get a cheaper substitute of what I actually wanted for my birthday. As children, me and my sisters never got any pocket money either. If we wanted something, we had to ask our mom for it. And then, she decided if we get the money or not. I remember collecting whatever coins I could lay my hands on so that I could join my friends when they went out to eat. This might sound like sacrilege to the younger folks out there, but it wasn’t that bad. Deep inside I’m glad all of this happened actually. All these small things ensured that I’ve always valued money.</p>

<p>I’ve seen my parents bargain for every single rupee. Many a times walking away from buying something because of few rupees. And countless times killing their wishes to save money for what we wanted.</p>

<p>I was pretty much living my parents dream. I didn’t know what I wanted. I didn’t know what an IIT was. And neither did my parents. My parents knew whatever advice the successful uncles and aunts gave. And with that, I joined a mediocre under-grad university picking up “Information Technology” as my major, since my uncle told my parents it had a lot of demand.</p>

<p>I wish I could get the four years of my under-grad life back. They were such a complete waste of time. I did not learn anything and had not done anything of value there. While I did particularly well in the technical subjects because of inherent interest, I had no idea what I was actually learning. The knowledge was completely theoretical. Most of my batch-mates were interested in doing an MBA post their undergrad and so they never bothered in the technical subjects. Engineering was just a means to an end. The ones who did not want to do an MBA, wanted to do a Masters from US. Since I was not interested in MBA, I assumed that Masters must be my thing.</p>

<p>Somehow, my parents were not convinced. I kept postponing giving the GRE exam, since my parents kept dodging the topic entirely. The registration fees was 160$ which was hell lot of money. Others had finished giving the exams and had moved to the next phase of the application process. I kinda had to threaten my parents that I won’t be giving my under-grad mid-terms (they were in a week) till I gave the exam. They reluctantly agreed. I immediately booked the date giving the exam after two days, since I felt prepared. I got a 1510 on my GRE (333 by current standard), which was pretty awesome. Eventually, I realized that I couldn’t afford to do a Masters that time. Financially, it was not a viable option for my family. I accepted it and prioritized getting a job instead.</p>

<p>I remember celebrating with my friends when we got a job offer from Infosys. It was paying 3.5 Lakh INR per annum and we thought we had struck gold. It was a really big deal for us. I was getting calls from my relatives congratulating me about this feat.
I found out that a friend of a friend was buying a house that time. It was in a very old building in a relative inexpensive area. The 1 BHK he purchased cost a whooping 60 lakhs INR. I realized that with my current salary, it would take me 20 years to own even a very small house. Considering if I use all of my salary (after taxes) for 20 years! It dawned upon me that there was no way I could afford a house in Mumbai. I wondered how will I make it in life. I decided the day I was able to earn 1 lakh INR per month, I could call myself successful.</p>

<p>For a variety of reasons (that’s a blog post for another time ;) ), after working in Infosys for a short time, I decided to leave Infosys for good. The decision was quite sudden actually and I caught my family and friends off-guard with it. I had just got a call from a friend saying they needed someone. And that’s all I needed. I simply quit. I didn’t even wait to get an offer from the other side. I don’t know why. I just knew I had to leave that place. I was ready to work for free as long as I was working on something meaningful. I joined my friends startup(MIME360) and decided I would apply for a Masters that year itself. I’d take up an education loan if that’s what it took. Like any startup, I was working pretty much on the entire stack. Since I had not done anything during my under-grad, MIME360 was the place I actually started to code! Every day was a new learning experience. And it was fun! At MIME360, I got a chance to be mentored by Rahul Chari and Sameer Nigam, the co-founders, two people who I look up to the most in my professional life. I’d leave everything in a flash and be with them if they ask me to join them for a new venture.</p>

<p>Soon, MIME360 got acquired by Flipkart (biggest marketplace in India), and my life changed instantly. My salary tripled in a day. Suddenly, I was surrounded by folks from IIT, BITS, IIIT, and other top universities in India. I had to work my ass off during the first year to keep up with them. It was like somebody had turned on a fire hose of information pointing straight at me. I tried to soak in as much as I could without getting blown away. The growth I saw as an engineer in that one year tremendously gratifies me. In time, I picked up learning new things on my own and started sharing back. The same brilliant IIT, BITS students were now coming to me for advice. I loved the work I was doing at Flipkart. And I was getting paid shit load of money to do it as well.</p>

<p>Remember the definition of success I had? Out of nowhere, I had reached my target. And so, the definition of success changed again. During the three years at Flipkart, my definition of success kept evolving with my growth. By the end of it, I could buy that house in 2 years. And still have money to spare. I had reached a point that I never had to be concerned about buying something I wanted.</p>

<p>And yet, there were times I felt guilty about it. Many times I interviewed people much much older than me and it made me realize that my decisions will be affecting their careers. I saw people working for tens of years and still earning one fifth of what I was earning. And I was not even 25. This world was not a fair place. I decided to leave Flipkart for a new challenge. After hearing the experience of my colleagues over the years, I yearned going back to studies and relishing a proper engineering education. And now, I could sponsor the education myself. What’s the worst that could happen? Even if a Masters didn’t work out, I could always come back and recover the cost in less than a year. I had already vested my Flipkart ESOPs as well so I knew my family had something to fall back on.</p>

<p>Thanks to the work I had done at Flipart, I got into Columbia University even though I had a mediocre under-grad. At Columbia, I got a Teaching Assistant job, which led to a scholarship and eventually an amazing internship, all because of my work at Flipkart. Unlike most international students at Columbia, I had immense clarity on what I want to do. The network of friends I made through Flipkart have been invaluable for this. I don’t think I can even come close to quantify the impact the three years working at Flipkart had on me.</p>

<p>And all of this wouldn’t have happened if I hadn’t quit Infosys that day. I didn’t know anyone who had quit so early. The HR folks did their best to scare me about leaving. And yet I did. I just knew one thing. It felt right. I took a risk and it paid off. No guts, no glory, right? Was I lucky? Hell yes. No matter how hard you work, there is always Murphy’s law to screw things up, so I like to believe in luck.</p>

<p>I always wonder where would I be now if I hadn’t decided to quit? Or what if my friend hadn’t called? What would life be like? Would I still have found a way to be at this level? Would I have found such awesome people to guide me? There’s no way to answer this. All I can do is be thankful that it happened and I followed my heart. I’ll let Robert Frost summarize things for me.</p>

<blockquote>
  <p>I shall be telling this with a sigh <br />
Somewhere ages and ages hence: <br />
Two roads diverged in a wood, and I— <br />
I took the one less traveled by, <br />
And that has made all the difference <br /></p>
</blockquote>]]></content><author><name>Bhavdeep Sethi</name></author><category term="personal" /><category term="musings" /><summary type="html"><![CDATA[Do you believe in destiny? Do you believe that things that are meant to happen, will happen, regardless of what you do? Does fate decide the course of your life Or are you responsible for your future and there is nothing called as luck? Personally, I'm not sure what to think. I do know that the last four years have completely changed my life. And thinking about it, all of this came down to a single decision, sometime in April 2011.]]></summary></entry><entry><title type="html">Using Multiple Heat Layers with Google Maps</title><link href="https://deepmusings.net/2015/01/09/using-multiple-heat-layers-with-google-maps.html" rel="alternate" type="text/html" title="Using Multiple Heat Layers with Google Maps" /><published>2015-01-09T18:03:17-08:00</published><updated>2015-01-09T18:03:17-08:00</updated><id>https://deepmusings.net/2015/01/09/using-multiple-heat-layers-with-google-maps</id><content type="html" xml:base="https://deepmusings.net/2015/01/09/using-multiple-heat-layers-with-google-maps.html"><![CDATA[<p>For one of the assignments in Cloud &amp; Big Data course at Columbia University, we had to display a Heat Map Layer showing positive and negative tweets from Twitter.  </p>
<p> Now, using Heatmaps with Google Maps is pretty trivial. Google Maps has pretty decent documentation along with working examples. </p>

<p>But how do we do it when we want to show multiple heat maps working independently on the same Google Map?</p>

<!--more-->

<p> Let's take a closer look at the standard Heatmap example first. </p>
<p>Go through the example given <a href="https://developers.google.com/maps/documentation/javascript/examples/layer-heatmap" title="Heatmaps examples">here</a>.</p>

<p>As we see, the following piece of code is responsible for initializing the heat map. It also has a function called changeGradient() which is of interest to us. </p>

<pre>
<code class="javascript">

function initialize() {
  var mapOptions = {
    zoom: 13,
    center: new google.maps.LatLng(37.774546, -122.433523),
    mapTypeId: google.maps.MapTypeId.SATELLITE
  };

  map = new google.maps.Map(document.getElementById('map-canvas'),
      mapOptions);

  var pointArray = new google.maps.MVCArray(taxiData);

  heatmap = new google.maps.visualization.HeatmapLayer({
    data: pointArray
  });

  heatmap.setMap(map);
}


function changeGradient() {
  var gradient = [
    'rgba(0, 255, 255, 0)',
    'rgba(0, 255, 255, 1)',
    'rgba(0, 191, 255, 1)',
    'rgba(0, 127, 255, 1)',
    'rgba(0, 63, 255, 1)',
    'rgba(0, 0, 255, 1)',
    'rgba(0, 0, 223, 1)',
    'rgba(0, 0, 191, 1)',
    'rgba(0, 0, 159, 1)',
    'rgba(0, 0, 127, 1)',
    'rgba(63, 0, 91, 1)',
    'rgba(127, 0, 63, 1)',
    'rgba(191, 0, 31, 1)',
    'rgba(255, 0, 0, 1)'
  ]
  heatmap.set('gradient', heatmap.get('gradient') ? null : gradient);
}
</code>
</pre>

<p>The changeGradient function changes the default gradient values to anything you specify. If you notice, the color is changing from light blue rgba(0, 255, 255, 0) for low density points to Sharp Red rgba(255, 0, 0, 1) as the density increases.
</p>

<p>What you can simply do is, initiate two different heatmaps, say heatmapPositive (light blue to dark blue) and heatmapNegative (yellow to red) in case you want to show two different heat maps for the positive and negative sentiment tweets. </p>

<p>If you want sample code for this, you can find it <a href="https://github.com/BhavdeepSethi/cloudBigData/blob/master/twitterMapSource/twitMap.js" title="Github Link">here</a>.  If you want to see how this looks, you can check out <a href="https://www.youtube.com/watch?v=9Qv7F_43dOk">this video</a> which shows the default state and then with the dual heat maps. </p>

<p> If you want to do much more, there is this neat library called <a href="http://www.patrick-wied.at/static/heatmapjs/" title="heatmap.js">heatmap.js</a>. It has bunch of features like adding legends and tool tips, customized maps, etc.
</p>

<p> Hope you found this useful. If you're aware of any better libraries out there, then Let me know!</p>]]></content><author><name>Bhavdeep Sethi</name></author><category term="engineering" /><category term="javascript" /><summary type="html"><![CDATA[For one of the assignments in Cloud &amp; Big Data course at Columbia University, we had to display a Heat Map Layer showing positive and negative tweets from Twitter. Now, using Heatmaps with Google Maps is pretty trivial. Google Maps has pretty decent documentation along with working examples.]]></summary></entry><entry><title type="html">Moving from WordPress to Github</title><link href="https://deepmusings.net/2014/12/11/moving-from-wordpress-to-github.html" rel="alternate" type="text/html" title="Moving from WordPress to Github" /><published>2014-12-11T18:03:59-08:00</published><updated>2014-12-11T18:03:59-08:00</updated><id>https://deepmusings.net/2014/12/11/moving-from-wordpress-to-github</id><content type="html" xml:base="https://deepmusings.net/2014/12/11/moving-from-wordpress-to-github.html"><![CDATA[<p>
So...the look and feel of the blog changed. For the good, I hope! Basically my GoDaddy web hosting expired and they deleted all the content that was hosted on it. Uh...yeah, I know.:/

<p><em>Luckily</em>, I had enabled the SuperCache Plugin on WordPress. Along with the Backup to DropBox Plugin, I was able to retrieve some of the posts. I'm getting into the writing mode again and not to mention that a lot has happened in the past six months...so I decided to revive the blog. The fact that I had to get involved with WordPress and GoDaddy again had already kept me away from this task for so long. <br />

But not anymore!

<!--more-->

I randomly googled "<a href="http://goo.gl/02SPK4">blogging platform for coders</a>", went to the HN link from the result and read about Jekyll+Github. And I can't believe I hadn't heard about this for so long! I was aware that you could create simple pages for your projects but when I saw what I could actually do (with User and Project pages both), I regretted not looking around for more options earlier. <br />

With Octopress into the picture, you do not even have to bother with the blog structure/theme. It took me about half a day to migrate the entire blog. Or whatever was left of it.

<p><h3>Why GitHub User Pages are way better than WordPress for your blog?</h3>
<ul>
<li>The power of Github! You can add collaborators, markdowns, issue tracking, versioning...do I need to say more?
<li> You can write articles using your everyday IDE! Intellij/Sublime Text FTW!
<li>You own the code. You decide how the code is going to be rendered. In WordPress(WP), you ended up writing articles and having no control of what is going on behind the scene. It took me three days to get a simple jQuery functionality working with WP.
<li> You can run your blog locally. Making changes is so much faster. I can work on the blog even when I don't have an internet connection.
<li> I prefer working on the command line. Period. Navigating along the blog like you would ssh'ing into a web server is so much faster than being forced to use a horrible GUI. And yes, MySQL CLI &gt;&gt;&gt; MySQL Workbench.
<li>With your blog being hosted on Github, you treat it as an everyday project! Publishing a new post is no longer a hassle!
&lt;/ul&gt;
<br />
I could go on and on...but you get the picture. I am so glad I moved to this new platform. It feels like home here. :P

<p> For others, who are considering this move, I strongly encourage it! The Octopress <a href="http://octopress.org/docs/setup/">Get Started guide</a> is all you need to get rolling! You can look at <a href="https://github.com/imathis/octopress/wiki/3rd-Party-Octopress-Themes">this Github repo</a> to go through some of the user made themes. <a href="https://help.github.com/articles/setting-up-a-custom-domain-with-github-pages/">This Github Help</a> page will give you steps for setting up a custom domain with your github page. That's it! Seriously. You're done! Hope to see you on this side.
</p></li></li></li></li></li></li></ul></p></p></p>]]></content><author><name>Bhavdeep Sethi</name></author><category term="personal" /><summary type="html"><![CDATA[So...the look and feel of the blog changed. For the good, I hope! Basically my GoDaddy web hosting expired and they deleted all the content that was hosted on it. Uh...yeah, I know.:/]]></summary></entry><entry><title type="html">Let’s just live for today!</title><link href="https://deepmusings.net/2014/04/25/lets-just-live-for-today.html" rel="alternate" type="text/html" title="Let’s just live for today!" /><published>2014-04-25T15:42:32-07:00</published><updated>2014-04-25T15:42:32-07:00</updated><id>https://deepmusings.net/2014/04/25/lets-just-live-for-today</id><content type="html" xml:base="https://deepmusings.net/2014/04/25/lets-just-live-for-today.html"><![CDATA[<p>I'm don't know the actual source of this poem so can't give proper credit for this. If you know where this verse is from, please reach out to me so I can link the source.  </p>

<p><br /></p>
<blockquote>
  <p>Before the world validates our identity based on race &amp; creed,<br />
Before they tell us where we belong &amp; which air we ought to breathe,<br />
For now, for once, let’s just live for today!
Before life drags us apart and tries molding our fate,<br />
Before fanaticism conquers love &amp; our credibility is put on stake,<br />
For now, for once, let’s just live for today!
Before the night looses its charm &amp; the moon hides &amp; fades,<br />
Before the first gleam of sunshine makes them discover our shade,<br />
For now, for once, let’s just live for today!&lt;/p&gt;
Before social apprehensions make us kneel &amp; break,<br />
Before we’re made to choose between the obvious &amp; man-made,<br />
For now, for once, let’s just live for today!<br /></p>
</blockquote>]]></content><author><name>Bhavdeep Sethi</name></author><category term="personal" /><category term="musings" /><category term="poem" /><summary type="html"><![CDATA[I'm don't know the actual source of this poem so can't give proper credit for this. If you know where this verse is from, please reach out to me so I can link the source.]]></summary></entry></feed>