Cassandra : Compaction, compaction and compaction

Hi Folks,

The word "compaction" is heavily used in the Cassandra world. You can see it everywhere while reading documentation, blog post, mailing list and so on.

Sometimes the use of compaction in combination of another word can be a bit misleading. Let's try to take a step back and to sump up a bit all the different concepts where the word compaction is used:

[Compaction]

Let's say the default one when you talk about Cassandra compaction. Cassandra has a write path which is very efficient. The concept is like "let's put as fast as possible the data into a file and in memory". In a second step, the data will be flushed on the disk as is. Ok but then, to enable Cassandra to also have an efficient read path, especially when you have to read a file from the disk, you need to tidy up and rearrange all the files. This process is called the compaction. A bit more technically : SSTTable are immutable and compaction is the action of generating new SSTable by merging and purging the old ones (duplicate, deleted data with expired ttl and tombstone).

Minor [Compaction]

We are still talking about the default compaction. The minor compaction is not really something which is minor.... The minor compaction is the compaction handled automatically by Cassandra as a background process and according to the chosen compaction strategy (see below).

Major [Compaction]

We are still talking about the default compaction mechanism in Cassandra. Contrary to the minor compaction, the major compaction is triggered by a manual action on a node (using nodetool compact). The major compaction can behave differently depending on the compaction strategy (see below). The other main difference between major and minor compaction and that explain the naming difference is that a major compaction has a bigger impact in terms of I/O.

[Compaction] Strategy

As already said, Compaction will merge into SSTables into new ones. Depending on you use case, Cassandra propose different strategies to compact the data new SSTables.  There is a default strategy applied if you do not specify it but you can choose the one you want per table. There are 3 main compaction strategies.

Size Tiered [Compaction] Strategy

This is the default compaction strategy. It fits for the write heavy and general workload. More documentation here:
http://cassandra.apache.org/doc/latest/operating/compaction.html#size-tiered-compaction-strategy

Leveled [Compaction] Strategy

This compaction strategy fits perfectly for read heavy workloads. This strategy involves a bit more I/O than the Size Tiered Compaction. It can be a good idea to combine that with SSDs. More documentation here:
http://cassandra.apache.org/doc/latest/operating/compaction.html#leveled-compaction-strategy

Time Window [Compaction] Strategy

This compaction strategy fits perfectly for time series. Basically the data is compacted regarding the timeMore documentation here:
http://cassandra.apache.org/doc/latest/operating/compaction.html#timewindowcompactionstrategy-operational-concerns

A good blog blog post which deep dive into it:
https://thelastpickle.com/blog/2016/12/08/TWCS-part1.html

Validation [Compaction]

This one is a fake friend from my point of view. It's considered as a compaction but is not really about compaction. The validation compaction is the process of building Merkle tree on nodes during a repair. It's anyway called validation compaction because this action is anyway controlled by the  

compaction_throughput_mb_per_sec  parameter. 

Anti[Compaction]

The anti compaction occurs during incremental repair. The goal is to split into two SSTables the repaired data from the unrepaired data. The 2 sets of data can no longer be compacted together and it's why it's called anticompaction.

Conclusion

Hopefully you get now a better vision of what means compaction regarding the context it's used into the Cassandra world.

Remote working in IT - 7 years later

Hi Everyone,

This post is a follow up of 2 previous articles related to the home working : 

• Yes, I code in underwear @home

• Home working in IT - 2 years later

Why again a post about that?

Because it's still a hot topic folks and I still have a lot of questions about it. The questions are now a bit different because most of the people who know me also know that I'm working from home for a long time. 

In addition to that, you may have notice that there's a small change in the post titles between the previous post and this one. There was evolutions on my way of working remotely.

So let's try to give some answers and try to explain the changes since the previous post.

Home working vs Remote working?

I totally agree with what Hubert said: 

😤 Reminder: remote working !== working alone from home
😍 I’ve been working in a coworking space for 3 years
👍 10 min walk from home, people with different jobs, lots of sharing and friendship
🤩 Noone talks about this, but I love this way of working

— Hubert SABLONNIÈRE 😎 (@hsablonniere) July 7, 2019

For me, the "remote working" is the concept of working remotely from the company you're working for and where a part of your coworkers are not working in the same office. I would describe the "Home working" as one of the possible type of remote working where you work remotely from your home. 

Along this 7 years of remote working, I had the chance to experiment a bunch of different possibilities.

1. Pure home working

I already described this in one of previous posts about the topics. Initially, I started to work remotely from my home and recreated an office in my house. (here)

2. Colleague home working

A good friend of mine, also freelancer, Thomas to not name him, was also working for the same customer in the same context. Our houses are close from each other. It's something like a less than 5 minutes car commute to go to his house. What happened is that, over the time and more and more, we were working together at his house or at my house. It was most of the time not decided upfront but more like : you start to work from your home, you have your first coffee and then a message was popping up on my screen:  

"Hello mate, your house or mine today?"

The difference with an isolated home working is that, to be able to work in the same room, we were working in the living room. The good thing with that is that we were still very flexible and independant which means that to work together, we needed to : 

• be willing to work together. Just a "no, not possible today" was enough without the need of any justification.
• have one of the houses free for work which means without one of the family around the house.

We ended up with few cool setup : 

The drawback of this way of working is related to the lack of isolation between the work and the family. Work hours are unfortunately longer than school hours and we ended up each afternoon with the kids from one or the other family in the living room and that's totally ok when it's from time to time. 

In contrast, when it begins to be the default situation and that you do that for more than one year, I started to feel an embarrassment when the family of my colleague was back home (not due to them!). Usually it ended up by relocating back home to finish the day of work.

3. Coworking space working

To solve some of the drawbacks of the "intensive" colleague home working, I started to search for alternatives. I started to search for a coworking space where i could go one day or two per week in addition to my usual home working. And you know what? I found out that a new coworking space called La maison du coworking (literally "the house of coworking") just opened few months ago and 10 minutes away from home. They were offering from shared desks on an open space to isolated offices per month. I contacted them but they were no yet offering a one day shared desk. They told me : no problem, let's create it!

That's how my colleague and I ended up moving away from the colleague home working to the coworking space working!

The really cool thing is that the open space was really quiet and everyone was working different thing and for different companies. So you do not fall back into the potential issues of open spaces like being disturbed every minutes by people coming to see you. The other benefit is that you can access common facilities of the coworking space : fiber internet connection, shared printers, kitchen, afterwork, meeting room...

For example, here is a meetup we organized in of the meeting room of "La maison du coworking" for a Cassandra Lan party :

It would have been hard to do it in my living room ;-)

4. Office remote working

While working at the coworking place on the open space, I had the opportunities to get a private 4 desks office in the building next to the open space building for a very reasonable price that my customer was willing to pay... There was no reason to say no...

This office was proposed "as a service" by the coworking space which include all the facilities of the open space plus the cleaning once a week

The really cool thing was to get access to : 

• a private place but with all the coworking facilities
• close to my home with a less than 10 minutes commute by car, motorcycle, bike, ...

Compared to the open space, We had access to the same level of facilities but with a private whiteboard. We could also use the office as a meeting room when we needed to host the local sprint planning of the team. 

You can also prepare the talk you'll give at Devoxx france and put all the mess you want in the office because it's yours!

5. Which one to choose?

As a good consultant answer, i could say : "The one that fits better for you"

For me the key thing is that i can still have a choice, everyday i can adapt to the setup that better fits my mood of the day.

Conclusion : The commute switch

As a conclusion, I would say that having an office really close to my house is really a bonus because it enforces a short commute which enables the "brain switch" : When I go to the office in the morning, the 10 minutes commute allows me to prepare my brain for the work switch. Same goes for when I'm going back home in the evening, my brain processes the day and unplugs from the work data. This is really important and you don't get this when you work from home, the switch is instantaneous.

I can still remember my wife saying : "hey, you're still at work?" even if i was on the living room with the kids because my body did the work/family switch but not my brain. If you work from home, I would advise you to take a bit of time to simulate this commute time before you switch into family mode. You can do this by reading the news, surfing on internet to find your next bike (because everyone is always looking for the next bike ;-))...

Enjoy! Your feedback is warmly welcomed in the comment section!

Une Cassandra/Spark Lan party à Devoxx France

Il y aura cette année (2016) à Devoxx France un hands-on particulier :

Une Cassandra / Spark Lan Party

Cet évènement aura lieu le vendredi 22 Avril (le dernier jour de Devoxx France) 

de 11:00 à 13:40 en salle Neuilly 231-232M Lab

http://cfp.devoxx.fr/2016/talk/YVP-4476/Cassandra%2FSpark%20Lan%20Party

Avec Thomas, nous avons été sélectionné pour organiser cet évènement et Amira et Gautier se joignent à l'équipe pour nous filer un coup de main. Merci à eux!

J'ai eu la chance de pouvoir participer à l'organisation de la Cassandra Lan party à Devoxx France en 2012, lors de la première édition. Nous avions réussi à créer un cluster Cassandra de 36 noeuds avec les machines des participants. Si vous voulez en savoir plus, le récit de cette précédente édition se trouve ici : http://www.unchticafe.fr/2012/04/histoire-dune-cassandra-lan-party.html

C'est quoi une Cassandra / Spark Lan party?

Cette fois-ci nous reprenons le concept initial de la Cassandra Lan party : 

- Créer un réseau avec les machines des participants 

- Créer un cluster Cassandra

- Manipuler des données sur le cluster

- Jouer avec le cluster

et en y rajoutant :

- Créer un cluster Spark

- Executer des jobs Spark sur le cluster

- Exécuter des jobs Spark sur le cluster qui jouent avec le cluster Cassandra

Si vous venez à Devoxx France et souhaitez découvrir Cassandra et Spark par la pratique, cet évènement est fait pour vous. 

Pourquoi y participer?

- Parce que vous ne connaissez pas Cassandra

- Parce que vous ne connaissez pas Spark

- Parce que vous ne connaissez pas Cassandra et Spark

- Parce que vous connaissez Cassandra
- Parce que vous connaissez Spark
- Parce que vous connaissez Cassandra et Spark

- Parce qu'une Lan party c'est fun.
- Parce qu'il y a un record du monde de 36 noeuds à battre
- Parce que vous voulez faire partie de l'histoire

Ok je m'emballe.

De quoi avez-vous besoin?

- votre ordinateur portable
- un Jdk 1.8 d'installé si vous voulez gagner du temps
- l'adaptateur ethernet de votre ordinateur (comme pour les Mac Book Pro récents par exemple...). Nous n'utiliserons pas de wifi mais un réseau filaire qui sera créé pour l'occasion.
- un cable RJ45 si vous en avez un
- votre bonne humeur

Pour le reste, on s'occupe de tout! Les valises sont prêtes.

A Vendredi!

Amira/Gautier/Thomas/Jérémy

Devoxx France Night Run

Il y aura donc bien un "Night Run" à Devoxx France : 

http://cfp.devoxx.fr/2016/talk/HZX-2594/Devoxx_France_Night_Run_

English version below

C'est quoi un "Devoxx France Night Run"?

C'est simplement de se regrouper pour aller faire un footing ensemble après une bonne grosse journée de conférence. On peut être geek et aimer la course à pied!

Quand?

Ca sera Jeudi soir. Départ à 20h en tenue de running.

Où?

Le point de rendez-vous pour le départ du footing se fera à l'accueil de Devoxx France au Palais des congrés (à côté du vestiaire et de l'endroit où vous avez retiré votre badge).

Quelle distance?

Une boucle d'approximativement 5km sera prévue à un rythme cool. L'idée, c'est de s'aérer la tête, pas de faire une compétition. Les plus motivés pourront continuer le footing après la première boucle.

Comment?

Les dernières conférences de la journée se terminent à 19h25. Vous aurez donc le temps de vous changer avant le départ (dans les toilettes si vous le souhaitez ou à votre hôtel si vous avez le temps).

Les organisateurs m'ont confirmé que les vestiaires de Devoxx France sont ouverts jusqu'à 22h30. Vous pourrez y laisser vos affaires pendant le footing et les récupérer à la fin de celui-ci.

Inscription?

un doodle a été créé : http://doodle.com/poll/dfhxtmprfzisrbff . L'inscription n'est pas obligatoire mais permettra d'avoir une idée du nombre de participants à attendre avant le départ.

Social?

Un club Strava a été créé pour l'occasion. N'hésitez pas à rejoindre le club et à partager la trace du footing : 

https://www.strava.com/clubs/devoxxfrancenightrun2016

English version

What is a"Devoxx France Night Run"?

It's simply a run together after a big conference day. Being geek does not stop to make a bit of sport.

When?

It will be on Thursday evening. Start at 8:00PM in running attire.

Where?

The meeting spot will be at the Devoxx France reception at the Palais des congrés (close to the cloakroom).

Which distance?

A loop of 5km at a cool pace. The idea is to get a bit of fresh air and not to run for a competition. The most motivated are free to continue after the first loop.

How?

The last conferences of the day end at 7:25PM. You will have time to change your clothes before the start (in the rest room if you want or at your hotel is you have enough time).

The conference organisers confirmed that the cloakroom is opened till 10:30PM. You can leave your things there during the run.

Registration?

Here is the doodle link : http://doodle.com/poll/dfhxtmprfzisrbff . The registration is not mandatory but it will give an idea of the number of attendees. 

Social?

A Strava club has been created for the event. Feel free to join the club and to share your run afterwards : 

https://www.strava.com/clubs/devoxxfrancenightrun2016

A bientôt!

A remote retrospective Story

a bit of context...

I'm working as a remote ScrumMaster since approximatively 2 years with the same team. This team is a full remote team even if most of the times we are doing our sprint planning together IRL. I went with my product owner to setup a new team in Vietnam six months ago. This new team is a bit different because, compared to "mine", all team members of the new team are working together on the same office.

It was quiet challenging because it was planned to make both teams working on the same project and same code base.

Now, 6 months and a product release together later, we decided to make a retrospective with the two teams. The goal was to find out what was working and what was not regarding the collaboration with the two teams.

So I was responsible to setup this retrospective.

the setup

I wanted to port my usual local retrospective format based on sticky note with a whiteboard using good/bad/questions/suggestions areas.

My needs : 

- a good communication tool.

- a good virtual white board.

We were used to use Google Hangout so we sticked with it as our communication channel. The only problem is the 10 slots limitation that i will have to deal with.

After some googling, i found that google drawing could be my virtual white board. I got some inspiration from here : http://www.iliokb.com/2013/02/facilitating-retrospectives-with-remote.html

So we did end up with this kind of setup and a drawing is better than a long speech :

Due to the quality of the internet connection of the green team and to the google hangout limitation, we decided to use the green team scrumMaster as a proxy of the other teams member and to use only one google hangout slot for the green team.

The plan

Here is what I did :

- the week before : 

• fix the date, not so easy to find a slot which everybody.

- few days before : 

• had setup of the google drawing white board, made some try.
• talked to the product owner to explain how it will work.
• talked to the other scrumMaster to explain how it will work.

- the day before :

• sent an email to all participants to remind the goal and scope of the retrospective and to ask them to start to think about that.
• talked again with the other ScrumMaster. As he acted as team's proxy, we discussed together to try to prepare that as better as we can.

- the D day :

• verified that everybody was able to join the shared google drawing.
• just dot it!

Here is the result :

It was fun to hear when everybody was filling the white board "hey, who has stoled my sticky?" ;-)

the retro of the retro

Bad :

• Hangout Limitation (max 10 slots)
• The scrumMaster of the green team was the proxy of the whole teams. It was ok because he was already used to do that for others meeting we already had. Anyway It would have been better to have the whole green team at the same "level"

good :

• It worked well! Using Google drawing as a virtual shared white board was really successful : simple and efficient.
• It's now decided to do that one time per release.

Notes : 

• the virtual retrospective was no longer than the local retrospective i was used to do locally.
• There was more preparation from me compared to a local retrospective.
• I had to allocate more time than initially planned to allow everybody to fill the google drawing.