We need to talk, says one service to another

‘But how?’ asks the other service!
Ever wondered how we communicate? One would not believe how complex and multi-step process it is. It involves some very complex terms like perception, encoding, medium and decoding. Let us take a look at a diagram explaining this:




So what is the relation with microservices? Communication between two services is not much different. It follows through a process very similar to this, in fact it can be explained with the exact same steps! Consider two services a ShippingService written in Java requesting the preferred address of a user identified by Id, from a UserAddressService over REST a call:



Okay, so human communication can be used as an analogy to understand inter-service communication in a microservices stack. So what? So basically, it tells us that the scenarios of failure can also be same and we can understand microservice communication by relating to human communication.

Why

Before we deep dive, we should quickly consider why it matters at all in microservices. It is basically the difference between inter-process and intra-process communication. Consider this, intra-process, meaning communication between components of a single process, is like talking to one-self. You understand yourself, well (usually!), at least there is a whole less ‘miscommunication’ when talking to oneself. This is the kind that happens in monoliths, and why it was not so much of an issue until we started working on microservices, which have inter-process communication. To relate to this consider talking to people instead, known or unknown (API, authentication etc), people of different race, origin, nationality (perception), at a distance or near you (response time), on a phone vs in person (medium/network, protocol), direct vs indirect or at a different time (whoa! yeah, i.e. sync vs async, like leaving a note), speaking unknown language (encoding-decoding, JSON/XML/binary like gRPC), and of different background or upbringing (perception again) and in different mental states (well, stateless vs stateful services), telling them a secret or casual gossip (secure vs insecure). Now you know why working in teams is so difficult, and also microservices!

Types

Let us break down all the phases and see the types of potential problems. This is not some standard classification, but I have found this method effective when understanding the issues with analogies.
  1. Encoding: Failure to encode the object correctly by missing attributes or encoding with different name. For example, frameworks in java allow for choosing different attribute names in JSON/xml format, these are specified as strings and there is no real validation on these other than tests. Also how do we ensure if both services are using the same message structure? This can in part be ensured by sharing a common object model library. But then this goes against the principle of knowing models by views; not every services needs to know what all the attributes of a Customer object are! You might as well use queues for communication then.
  2. Sending the message: These include the whole set of issues which can occur due to not sending the message on correct protocol, address, port or path, the whole address related issues. Integration testing can not always help here, as some of these depend on the production infrastructure as well.
  3. Decoding: Being process is reverse of encoding and comes with all the same issues.
  4. Perception: Now these are some of the serious issues which you tend to miss even in unit and integration tests and can be caught only in later phases, when working against live services. If you encounter these, you can most certainly assume that apart from inter-service, you also have some team communication issues.
  5. Feedback: One of the most important step in the communication is the feedback, the acknowledgement of receipt of the message. There can be plenty of causes for a service to not respond, including all of the issues discussed above, and adding potential networking and issues related to health of the service.
  6. Cascading Failures: A much serious situation which is merely an outcome of failure to identify the issues in service communication and safeguarding against them.
Now that we are acquainted with the terms, I want to tell you a couple of stories; really scary, real stories.

Case study 1

I used to work with a Value Added Services (VAS) company, you know the services you never want yet your carriers charge you for, this was one of those services. The company was one of the few trying to build a useful service you may want to pay for, trying to play fair and by the rules, even at a revenue loss. VAS industry suffers from considerable frequency of fraud and so it was said that the biggest crime is charging your users twice. Such incidents would be immediately flagged as fraud by the carriers and the service would be banned.

We had a call flow where a “Billing Service” makes a call to a “Carrier Integration Service” for charging which in tern makes a call to an external-internal Notification Service (a shared service hosted by a different unit) to send notification to user, after firing the charging request to the carrier. It worked fine for years, until one day this external-internal service suddenly slowed down, timing out all the calls from Carrier Integration Service, which in tern caused a time out on Billing Service, which treated it as a failure and ‘retried’ the billing call. The issue got flagged in minutes, but hundreds of users were charged multiple times that day before the team could respond.



There were a whole lot of things that went wrong here. There were feedback issues, which likely occurred due to network issue, giving rise to perception issues and finally causing a cascade. The worst thing though, was the cascade.

Case study 2

Another such story, not so grave, was when the Billing Service sent a request for partial charging, but the Carrier Integration Service denied honouring it. Both services used enums to identify the keyword used, both had integration tests and all worked fine. It was only after the services were tested against working instances of one another, in a pre-prod environment on a crunch day that it was identified. The issue was stupidly simple, both services used different value for the constant! Again a whole lot of things that went wrong here, most important was the perception issue caused due to the miscommunication between the teams working on these two services.

The Solutions

Discussing every step identified to fix the case studies we saw will be a story in itself, we shall discuss the first solutions applied to the most glaring causes in both cases. For the cascade, we made it a point that every service will implement Hystrix for every single inter-service communication call. Hystrix is a circuit breaker, meaning it wraps a chunk of logic (say a method) and on exception it can flag, throttle, block and bypass the method in question. The idea is to wrap the calls to external services, aka dependencies, and when something goes wrong give them time to recover by bypassing calls and safeguard the sender service from cascading the issue. We had hystrix is some services, but some teams had argued that it is an unnecessary complication for internal services and services that have been working fine. Well, that was before the incident. Everyone just jumped on it as the first change once we recovered from the impact. In my view, circuit breaker is a tool microservices should never be built without. We also ensured that the Carrier Integration Service builds a temporary cache of all the users it processed and validate against it before it fired any call.

For the perception issue, we need to ensure that post encode-decode the receiver understands the same thing as the sender meant. We had hosted stubs against which we tested the services in automation testing phase. These stubs were dumb services implementing the same API as the service they stubbed. These were developed by whichever team needed them, essentially Billing Service team would never develop the stub for the Billing Service; which caused the discrepancy in the stub and actual service behaviour. This had to change, the team developing Billing Service was to be responsible for developing the stubs for Billing Service and team for Carrier Integration Service for it’s service stubs. This way, the stubs always perceived the same as the actual service did.

Now how do we address intra-team communication problems, anyone?

Google Tag Manager: Tag Priorities Vs Tag Sequencing


As most GTM (Google Tag Manager) users will agree, this is a much discussed yet confusing topic! The documentation on these topics is very concise and to be honest precise in describing what these two options do and what to expect, yet some of the side effects of these two options, combined with asynchronous nature of JavaScript are left out to be inferred by the users. And this is where much of the confusion seems to come from. Even the many blogs out there on this very topic barely touch this context.
Hence we are going to discuss this very thing today.

Tag Priority

Tag priority as described by the documentation is a number associated with a tag which identifies the order of firing the tag. Firing, not completion. Secondly, firing is itself an asynchronous process. If you consider all simple HTML tags, firing them meaning adding them to the HTML of the page, which is what the GTM script is responsible for. What tag priority means is that all the html elements will begin to be added in order identified by the Tag Priority but the GTM script will not wait for the elements added before to load and execute before adding next. (also it cannot know if it is finished, more on that later) And hence this does not govern the load order of the tags.

Tag Sequencing

Tag sequencing as described is a setting that governs which tags will fire before and after a particular tag. One can imagine this as a setup-run-cleanup processes, established with GTM tags (which is also apparent in the documentation). Think of it like a unit test; there is a setup which will fire before the tag (@Before in JUnit, or beforeEach in mocha), then the tag itself (@Test in JUnit or it("", ()=>{}) in mocha) and then the after-tag/clean-up tag (@After in JUnit or afterEach in mocha). If you were trying to make sure that a given tag fires before another, you should be happy sequencing exists.

Read the document carefully again and you will see it does not speak about completion yet again! Much like priority, sequencing cannot guarantee the completion of setup tag before firing the middle (test) tag! It will only ensure that setup tag is ‘fired completely’ before moving on to fire the middle / clean up tag.

In the Tag Priority documentation, it correctly states: “Tags will still be fired asynchronously (tags will fire whether or not the previous tag has finished.)” and “Tag Sequencing allows you to specify exactly which tags fire before and after a given tag.”. But says nothing of this sort in the documentation of Tag Sequencing.

By nature of JavaScript, it is difficult to know when execution of a particular snippet completes without explicit notification from the snippet; it can be in the form of an event being fired or a callback being triggered (Promise will come under this too); but as GTM does not ask for either, it cannot really know if your tag is ‘completed processing’.

We can see this by doing a simple experiment. In a GTM container, let us create 3 custom HTML tags as:

1. SetupTag:
<script type="text/javascript" src="https://code.jquery.com/jquery-1.12.4.min.js"></script>
<script type="text/javascript">
  sayOnDoc("'jQuery' object defined on page: " + !!window.jQuery);
</script>

<script type="text/javascript">
  jQuery(document).ready(function() {
    sayOnDoc("'jQuery.ready' fired.");
  });
</script>
<script type="text/javascript">
  document.addEventListener('DOMContentLoaded', function() {
    sayOnDoc("'DOMContentLoaded' fired");
  });
</script>
<script type="text/javascript">
  var gtmName = 'google_tag_manager';
  var insideGTMAndDomReady = window[gtmName]
            && window[gtmName].dataLayer
            && window[gtmName].dataLayer.gtmDom;
  if (insideGTMAndDomReady) {
    sayOnDoc("'GTM ready' done.");
  }
</script>

<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjs/3.16.5/math.min.js"></script>
<script type="text/javascript">
  sayOnDoc("'math' object defined on page: " + !!window.math);
</script>

<script type="text/javascript" async src="https://cdnjs.cloudflare.com/ajax/libs/d3/4.11.0/d3.min.js"></script>
<script type="text/javascript">
  sayOnDoc("'d3' object defined on page: " + !!window.d3);
</script>

<script type="text/javascript" async src="https://cdnjs.cloudflare.com/ajax/libs/angular.js/1.6.5/angular.min.js"></script>
<script type="text/javascript" async=true src="https://cdnjs.cloudflare.com/ajax/libs/ag-grid/14.0.0/ag-grid.min.js"></script>
<script type="text/javascript" async src="https://cdnjs.cloudflare.com/ajax/libs/underscore.js/1.8.3/underscore-min.js"></script>
<script type="text/javascript">
  sayOnDoc("'_' object defined on page: " + !!window._);
</script>
<script type="text/javascript" async src="https://cdnjs.cloudflare.com/ajax/libs/backbone.js/1.3.3/backbone-min.js"></script>

<script type="text/javascript">
  var someScr = document.createElement('script');
  someScr.onload = function(){
    sayOnDoc("Moment js added to page");
  };
  someScr.src = "https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.19.1/moment.min.js";
  document.head.appendChild(someScr);
</script>

<script type="text/javascript">
  setTimeout(function() {
    sayOnDoc("My timer timed out..");
  }, 500);
</script>

2. MiddleTag:
<script type="text/javascript">
  sayOnDoc("Middle tag fired");
</script>

3. CleanUpTag:
<script type="text/javascript">
  sayOnDoc("CleanUp tag fired");
</script>
And create an html file or jsFiddle like this.
Now we simulate both scenarios:

Priority Test

Setup: Have all the three tags triggered on ‘All Pages’. Set priority of SetupTag set to 20, MiddleTag to 10 and leave CleanUpTag empty or zero.


Observation: The Setup tag is the first to fire. Mid way during the execution MiddleTag and CleanUpTag fire, while the Setup tag is still to complete.

Sequence Test

Setup: Have only MiddleTag triggered on ‘All Pages’. Set CleanUpTag as the clean up tag and SetUpTag as the setup tag for the MiddleTag.


Observation: The SetupTag starts to fire, all the script additions from the setup tag are done first, then the MiddleTag fires and finally the CleanUpTag, almost as if the next tag waits for the previous to complete. Yet the asynchronous sections of the SetupTag fire way after the CleanUpTag!

Additional Observations to Note

Note that in both cases the ‘async’ scripts cannot be guaranteed to be executed in order. Also that the event ‘DOMContentLoaded’ is completely skipped, this is because the event happens before GTM starts firing at all. Also interesting to note that although jQuery is loaded before, ‘jQuery.ready’ triggers after the ‘GTM ready’ is written to document, meaning that although jQuery is loaded on the page, there is a delay before the ‘ready’ event is fired; only that the delay is not long enough for text to show up after Middle/CleanUp tags. In both cases the completely asynchronous snippets: adding a script to page and a timeout delay happen way after the CleanUp tag.

There is a whole lot to discuss on the script loading and execution in browser, and asynchronous nature of JavaScript itself, which we have not and cannot cover in this post. There are a whole lot of different things that can happen depending on how we write the code in the tags and the browser you load the tags on. But at least we know that we cannot rely on GTM blindly to sequence the tags, especially if there are any asynchronous components in them.

As a side note, there are a whole lot of different and interesting scenarios that arise when we combine the priorities and sequencing with Tag firing options.

Software development hygiene: Why do we brush our teeth?

Yes, why do ‘you’ brush your teeth?
Is it guaranteed that if we brush our teeth twice a day, floss once a day, gargle with an antiseptic, we will never have toothache or bad breath? And if we did not brush teeth say, for a week, would we be guaranteed to have toothache? For a few months, may be yes, we might, might just have to get some treatment done for a few teeth. So the question, why do we brush our teeth, daily?

And how did we start brushing the teeth? Were we born with a brush in one hand, toothpaste in other and with an utter, inexplicable desire to brush teeth every morning after waking up from sleep and before going back to bed? Assuming that no one would remember how they themselves were born, all parents at least will agree with me, that this is certainly not the case. So the question, how did we start brushing our teeth daily?

And now the question you might have in your mind: “What’s the point?”
Recently, a person on our team raised this question(s): Why do we have unit tests. I have been writing good code, good enough that QAs do not find any critical issues, nor has anything ever severely broken in production because of my changes, why should I write tests? If I could think of all scenarios to unit tests, why do we have dedicated QAs on our team? Why should I pass my code through a static code quality analysis tool? All these processes are slowing us down. I have worked without all these processes in the past and that has worked quite well, why do I need this overhead of processes?

I agree, I hate processes.
Yet we need to appreciate the importance of processes and acknowledge where they are required. Come to think of it, why does a process exist? Can we not work without processes and the overheads thereof? Short answer: No, we cannot. Long answer: We can, given that everyone on the team understands the core reasoning for the existence of the process being bypassed and takes the responsibility of upholding the goal of this process without strict adherence to the process itself.

Well, how did I start brushing my teeth daily? My mother would tell me: till I was a couple years old, she used to brush my teeth. When I became three, she taught me how to do it and would ask me to show how clean my teeth were. She would ask me: “Are they shining when you look in the mirror?”, I would go and check and say “Yes”. When I became four, she would just remind me to brush, and I think at five I had finally started brushing my teeth daily, without having to show her how clean they were. I do not believe your story would be very different than this. It took years of practice and perseverance of our parents to eventually get us to brush our teeth daily so that finally we could get rid of the ‘overhead of process’.

Yes, many processes can be chucked as long as the goal is achieved; but are we, as a team, responsible enough to make sure they are achieved every single time? Let us say we are, but are we ready to carry the burden of remembering every single code smell, every single potential bug and be mindful of it while writing code? Is that even humanly possible? If the answer to that question is yes, sure go ahead and chuck the quality analysis tools, unit tests, pull requests and code reviews; we don’t need them. But if the answer is no, wait till it becomes yes!
We can certainly bypass processes and get an apparent speed-up, but chucking a process before we are ready is sure to give us pain in the tooth (and in a few more places)!

Experience: Introducing JMockit To The Team

Like many codebases out there, our codebase at work had a backlog on unit & integration tests and it was high time we covered it up. So one fine day, it was decided that we shall no longer accept code without tests. Then the question of 'how do we write tests' came up. As one of the architects on the team I introduced the methodology of unit/integration testing and a mocking library (JMockit) to aid in cases where testing could be difficult without one, conducted trainings and hands-on sessions for everyone on the team, set up a peer review process and we were ready. That is all there is to the history of the situation we are in. Well, we are talking about a large team here, some eighty people working on a large codebase as a whole, although divided in multiple microservices.

Today, almost every test we have has a mocked class, and an expectation set on some dependency. Many tests verify how many times a particular internal/private method or a method on a dependency was invoked, some have VOs mocked, or have assertions on return values inside verifications block. A few have gone to the lengths of mocking constructors of certain objects because there was no way to inject them into tested class. We are not even considering the private and static method mocks here.

When I look at the tests that we have today, and look back on the last few months, I wonder if I made mistake while introducing the mocking tool. In the trainings we had, we discussed the purpose of a mocking tool, issues due to overuse, indicators of overuse; heck there was even slide dedicated to this in the developer induction we have here. Architects were involved in many code reviews and tried avoiding these pitfalls, but clearly it was not enough.

Tests are supposed to improve the design of the code. Since the highly tangled, coupled classes, classes breaking SRP (Single Responsibility Principle) are difficult to test, we tend to fix them. As the size of the class increases, the functionality it has grows making it difficult to test, so we split it. As the dependencies being created makes it difficult to test, we change the class to allow injecting them. We end up splitting large methods, redesigning methods with side effects, removing unneeded code, decoupling from the libraries all for making the code testable and effectively get cleaner, maintainable, verifiable code. All of this, only if we wrote tests correctly.

If we started modelling our tests to match our code, we not only lose all the benefits, but the tests start becoming coupled to our code. That brings down the speed of refactoring or new development because every time we change code, we need to change the tests to match the implementation. That brings down our overall productivity. And finally blaming it on tests, we would stop writing them altogether. Back to square one! Mocking tools have a purpose, but if we mock everything, we get our tests tightly coupled to the implementation, adding to the problem. Mocking simple data-structures and VOs is not meaningful, we never test their methods separately, they are not supposed to be tested. (And yes, VOs are data structures, let us not get into that here.) Mocking external libraries is risky, because we then verify their actual behaviour only at runtime, which is exactly what we are trying to avoid by writing tests.

JMockit is a powerful tool, a little too powerful, and harmful if we are not careful. Despite the misuse that we have done, it could do all those things is itself marvelous. I am not convinced to blame it on JMockit, it is as Uncle Ben said to Peter Parker: ‘With great power comes great responsibility’. What we did, is our fault, we should have been more responsible with it. I wish I could go back and change the way we used it, or overused it, but our life is not a Git repository.

Luckily though, we have identified these issues and their severity before they start heavily weighing us down. What we need to do first is avoid more such tightly coupled tests from getting in. The course of action now seems like along with training the team on how to be more responsible with tests (which by the way needs to be a continuous process), we need to identify reviewers and train them on spotting such instances. Revising the review process to have reviews through identified reviewers and not just peer reviews.

Please Give It A REST!

A regular stressful day in the life of a software developer. I was communicating a module we needed to quickly put together. The team was not exactly new. We had a backend guy, and a front-end guy. Interface was designed and agreed upon, we needed to make it quick, we just defined the resources and said that we need the ‘standard CRUD operations done via REST’ on them. And we got down to work.

When everyone reverted that they were done, we sat down to quickly integrate and test, but the front-end and the backend refused to talk, we got the 404, not found. Well, it should have been simple, we had very clear instructions; the resource names and that they were REST.

Or so I thought. REST, the standard, arguably most popular of the web service protocols out there should have made it very easy for the backend and frontend to talk to each-other. But no. Turns out that there are (still) huge misconceptions around REST; there are so many among us who believe that abominations like ‘getCustomer’ and ‘createCustomer’ (yeah, you guessed it write, the HTTP method was ‘GET’ for this one too) are ‘resources’ and are valid REST.

Oh please, give it a rest.

Not the first time have I encountered this, and it would not be the last either. I thought we were over this. But no, I had fallen for it, assuming that common knowledge is common. It is not.
Well, these ‘divergents’ among us, are not totally wrong in assuming what they write is REST, it could probably be referred to as a form of REST. They are on HTTP at least, someone could certainly fit the level-0 in Richardson Maturity Model to it. But this type of API modelling does not really help integrations, has no standard understanding nor predictable way of implementation and yet we tend to stick with it.

The level-2 is where we need to be with REST, as a basic understanding and expectation from APIs: with Verbs and Resources. It conveys a very clear message as what to expect
from the interface. While Level-3 or HATEOAS with a hypermedia client, or if you are so security conscious making the URLs opaque, would be a dream implementation but for a team struggling with ‘resources’ it feels far-fetched. So level-2 it is.

I have seen that there seem to be a whole lot of people with this kind of confusion around, entire applications built with just GET calls for everything, even for inter-service and frontend-backend communication. I wonder at times what could have caused this confusion, and popularity of Spring’s for REST implementation comes to mind time and again. It is also an observation I have made over time, those who think ‘resources’ also tend to be Jersey (JAX-RS) users at some point in time and the other class seems to be comprised largely of Spring users for their REST implementation.

Probably it makes sense, Jersey as a framework leaves little room for the idea of method-wise resource names, you tend to define the resource at the top of the class (or resource!) and HTTP methods are just marked below. Although you have the option of marking each of your method differently with an additional @Path annotation, the trend is not seen as much and people are forced to think in terms of resources and verbs. Whereas the Spring’s implementation of REST is basically the Spring MVS’s controllers and request handling used to simulate how a REST service will work. Although Spring 4 introduced the @RestController it did little to enforce the resource behaviour as did @GetMapping, @PostMapping and the siblings defined in Spring boot; the method implementations are still standard (or implied) @RequestMapping from Spring MVC and coming from there, people may tend to think a need to override the handling and define the java method name in the path, for some reason. Well, I would not know for sure but this seems like a logical explanation for the trend observed.

Now how do we convey to everyone that they do not need to define different paths and let the HTTP verbs do their job is a challenge. This post is just an attempt.

Participating In A 24 Hour Hackathon

Just returned from a 24 hour hackathon, sleepy, red-eyed, tired, exhausted and yet writing this post. You know why? Because I skipped it the last time, and the time before, thinking I will do it the next day and that sleep was more important, but never did it. Not going to make the same mistake again. So here I am.

For those who are unaware of what a hackathon is, it is an event where dreamy eyed people enter and leave with red eyes, it is a single night sprint where people come together to build something that they believe will make them a billionaire or like Silicon Valley series mentions time and again, ‘will make the world a better place’! Well, jokes aside, a hackathon is an event / competition where teams / individuals build software / hardware in a single sprint of 24 hours. Hackathons have a theme, ranging from generic things like improving community to specific themes like solving a specific delivery problem the company faces. Some hackathons are closed, conducted only for the members of an organization, some are open to all. Some hackathon focus on profitability of an idea and implementation, teams winning sponsorships from investors, and some focus only on ideas and imagination of the participants. All in all, a hackathon is a developer’s Disneyland!

If you feel like you have ideas, but no time to develop them, try them out or run them by other people? Hackathon is the place for you to build your dream concept into reality. You are an amazing developer, you can punch in code and get things working in no time, hackathon is the place for you to show your skills. You like to interact with people, share ideas and learn how people feel about them, hackathon is the place for you to validate your product. You have a concept, but are looking for skilled brains to develop it, hackathon is the place for you to spot skills and recruit them. You are a nerd, introvert, who loves to code (the stereotypical software developer), walk right in, there are many like you in there. You are a night-owl who believes that sleep is for ‘cats’, you will fit right into a hackathon. Imagine a place which provides electricity, wifi (I have your attention now, don’t I?), food, seating space and all else you need, and leaves you undisturbed for 24 hours with the freedom to build your dream into reality, now that is a hackathon. (Put like this, it sounds better than Disneyland!)

Now let us say you wish to go to a hackathon and ‘make the world a better place’, you need to have a plan, like everything else. There are things that you should and should not do. There are things that you should and should not carry. I have over time built a list of items I carry to a hackathon, and like other ideas validated it against others during the hackathon. So what we have here is a list of items people in general carry, not every item will apply to you though. It is like a camping list, but for geeks.

Preparing for the Hackathon


  • Prepare for your idea: Think, elaborate it, plan it. This is also a test of your agility, all your skills in iterative, agile, development are going to be tested. Know that a 24 hour hackathon is 3 working days time on your hand, it is a lot, plan how to utilize it best.
  • Choose the right team: Long hackathons tend to be team games. Choose your team wisely. You should have compatible yet slightly overlapping, targeted skills, and equal passion. You probably do not need a marketing executive or the so called ‘product owners’ on your team unless the idea is from their domain. It is not a conference, you do not get a booth there. And passion, yes, you do not want your team members jumping into sleeping bags at the chime of 10 only to get up at 6 next morning.
  • Identify what open source projects can help you, know how to use them. Play around with them. May be send a pull request for missing features. But know what is going to help you get it done faster and plan for it.
  • Set up your machines. You do not want to be downloading a database server or creating cloud service provider accounts at the hackathon.
  • Set up productivity tools suitable for your idea. For example, get accustomed to using a clipboard manager; write and keep handy scripts to automate simple tasks like starting db, launching db shell, clearing tables, generators for various frameworks etc.

Packing for the Hackathon


  • Laptop: Of course! Humans are yet to build a computer one can use to develop software on in thin air, unless that is what your idea is for the hackathon.
  • Laptop charger: You will be surprised how many people forget this. Although your machine has juice for one work day, it is not enough for a hackathon. If you think of it, it is actually 3 work days there. And you don’t want to be making ‘connections’ by asking people for charging cables.
  • Phone & charger: You are going to interact with a whole lot of people, do carry your phone to note down numbers, not everyone brings the business cards, it is not a conference. Hackathon is for thinkers and doers not talkers, yet the excessive use of phone screens drains them, so do carry your chargers. Some hackathon venues do provide charging stations, you can check to confirm if there is one.
  • Peripheral devices: If you prefer to use an external mouse, drawing pad, a VR headset or whatever you need, carry them. Pack the devices you need to build your idea, like a raspberry-pi, a hovercraft kit or whatever. Keep pen-drives handy. You can check or request if the organizers provide an external screen, it will be too huge to carry anyway. It is all developers and geeks there, I would not blame you if you do not trust the security of the wifi there. In that case, carry your own portable hotspot.
  • Identity proof: You have registered online, but the organizers need to know who you are before you can enter.
  • Toiletries: Do I need to explain? Just don’t stink, you do not want to be remembered for that.
  • Medicines: If you are on medicines, do not forget to carry them. If you get acidity by staying up late, carry antacids. You get headaches, carry a mild pain killer. Have allergies? Carry an antiallergic. Afterall it is a competition, you want to be your best throughout.

Wearing for the Hackathon


  • Wear something casual and comfortable: If suits are your thing so be it, but remember you are going to be scratching your head over a lot of things in the next few sleepless hours, be comfortable. You do not want to be scratching other parts of your body due to uncomfortable clothes. You do not need to look pretty/handsome, make-ups and hair-sprays are not required, your personal comfort is.
  • If you are representing a startup, wear them on! Hackathons are good for creating awareness and hype.
  • It is okay to wear your lucky accessories, but limit it to that, avoid the temptation to wear superman under your clothes.

Hacking at the Hackathon


  • Not literally. Do not hack others’ devices, you can get banned from the premises or worse.
  • Divide your hours like they were days. 4 hours represent your half day of work in there. Have regular discussions.
  • Divide your tasks and identify interfaces where your tasks meet.
  • Use version control systems extensively, If it is a hardware product, keep taking photos from all angles. At hackathon, I prefer to change the way I commit; usually I commit often but push after cleaning. At a hackathon I commit and push extremely frequently, on every unit completion. It would not be an exaggeration to say that make the version control system your undo log. This also help prepare for an unfortunate event if your machine decides to take a nap while you are banging the keys.
  • Ban headphones on the team. Unless there is interaction, there is no team.
  • Plan your long breaks, like lunch, dinners and snacks to be in sync with your discussions.
  • Take frequent breaks apart from the discussions, individual or team breaks, but get up and walk around. I drink a lot of water, that forces me to take frequent breaks and helps avoid health issues as well. It is also a good idea in your day to day work.
  • Do not sit it through, walk around, jump around, interact, stay active and awake.
  • If you can, take naps in between. Just make sure you allow your partners to pour a water bottle on your head to wake you up, just in case. You do not want to miss all the fun by sleeping through.
  • There tend to be side events every few hours in long hackathons, participate in them, get to know people.
  • Meet people. You will find a whole lot of them are working on something amazing. You might end up meeting your next employer, co-founder, your living idol, or even your soulmate, if you are looking for that. You never know. All those coming to the hackathon tend to be there for their passion.
  • Have fun. Win or lose, unless you have fun at the event, it is pointless. Have what fun means to you. No one forces you to take part in any of the events or to talk to people, if you just wish to code, so be it. But enjoy the 24 hours, you do not get platforms like this every day.

After the Hackathon


  • Wrap up everything, pack the items and belongings, do not forget your chargers and peripherals.
  • Have one final meeting, plan out what you would like to do with the idea and the code / product built so far.
  • Divide the tasks for the future and decide timelines. If you leave here without a plan, and if you have not won the prize, it is highly likely that the idea will never be pursued further.
  • Know that what you are feeling, the mild body ache, sleepy red eyes, it is similar to a jet-lag and treat it as such, by sleeping only at your usual timing. Avoid taking an untimely nap in the day, set your routine back as soon as possible.
  • Write a blog. ;)


If there is something that should be added to the list to make it more usable, please suggest. See you at a hackathon some day.


Docker As Application Registry

Docker As Application Registry

Docker is great and solves a lot of problems with deployments. It taught VMs to share the resources, like how VMs taught hardware to share resources! Along with production, I have found that docker can work great as an application registry in a local development environment.
By applications I mean software that you install on your OS and launch them with shortcuts and they continue to live and retain state till you uninstall them; not exactly what containers are designed for but can work as. Something like snap or flatpak but with docker and for servers as well, not just for UI apps.

One advantage this has over using standard installers (like apt-get) is you are at complete liberty to start and stop the background processes, like mysql. If you installed mysql this way, you do not need to go and disable the autostart for it, it just does not matter! Similarly for your SonarQube server, you do not need to install it as a daemon, neither do you need to remember where you downloaded it to be able to restart it. Another advantage is most of such applications do have official docker images, it is the intended way to use them now!

One disadvantage of this method is that you always need to address them with their IP, you will not be able to bind them on host network then. But in my view, it is always better to have a dedicated IP, it emulates production scenario better and does not clutter your local machine ports.
So for apps what you need is containers to live long, be able to identify them with name, start and stop them easily and have a dedicated, static address to be able to reach to them. Most of these things are easy, except for a static IP. But once you create a network, you are set. That is it, it is that easy! Create a virtual network, and start your dockers with a name and static IP in that network. Simple!

To create a network:
docker network create -d bridge --subnet="172.30.0.0/24" --gateway="172.30.0.100" --ip-range="172.30.0.0/24" permanet

Now any app you need, just specify this network and a static IP of your choice; like this:
docker run --name mysql-server --network="permanet" --ip="172.30.0.1" -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql/mysql-server:5.7

More complex containers can be created like this:
docker run -it --name gocd-server --network="permanet" --ip="172.30.0.5" -v /yourhome/docker-volumes/gocd/godata:/godata -v /yourhome/docker-volumes/gocd/home:/home/go gocd/gocd-server

I keep a dedicated directory in my home for docker volumes, so I can back it up and use them as is when I change machines or OS. I also have a script where I add all the containers I need, so it is just a matter of copying the volume directory and running the script to create identical setups. Then even your .desktop files work as is!

Here is an interesting setup script for Jenkins (gist), it externalises all data directories from Jenkins, including the plugins and users and mounts your local m2 repository inside Jenkins so as to avoid downloading the libs again :
docker run -it --name jenkins --network="permanet" --ip="172.30.0.6" \
  -v /yourhome/.m2:/var/jenkins_home/.m2 \
  -v /yourhome/docker-volumes/jenkins/workspace:/var/jenkins_home/workspace \
  -v /yourhome/docker-volumes/jenkins/jobs:/var/jenkins_home/jobs \
  -v /yourhome/docker-volumes/jenkins/plugins:/var/jenkins_home/plugins \
  -v /yourhome/docker-volumes/jenkins/users:/var/jenkins_home/users \
jenkins:2.32.3-alpine

You can create as many such networks as you wish for logical separation of groups of such apps; in my case this is the third network (172.30 for that reason), since first two were taken up by some compose scripts.

A list of few such containers I use: mysql (different versions), SonarQube, hystrix-dashboard, zipkin, swagger-ui, a redis-cluster for local use, gocd-server, Jenkins, portainer, postgres, pgadmin etc. I even have a couple Windows software running on wine in such containers, we shall talk about it some day.

We need to talk, says one service to another

‘But how?’ asks the other service! Ever wondered how we communicate? One would not believe how complex and multi-step process it is. It in...