Pourquoi Spark est mieux que Hadoop MapReduce
Dans cet article, je vais vous montrer l’importance du Spark par rapport à Hadoop MapReduce. En effet, je vais expliquer d’abord l’importance du traitement distribué et parallèle.
Importance du traitement distribué et parallèle
Nous vivons à l’ère des données !
The International Data Corporation (IDC) a estimé que le volume mondial de données sera multiplié encore par 3,7 entre 2020 et 2025, puis par 3,5 tous les cinq ans jusqu’en 2035, pour atteindre la somme vertigineuse de 2 142 zettaoctets. Alors que 1 zettaoctet correspond à 1021 octets, ou mille exaoctets. Bien que les capacités de stockage des disques durs aient considérablement augmenté au fil des ans, la vitesse à laquelle les données peuvent être lues à partir de disques n’a pas été maintenue. Plus de 20 ans plus tard, des disques d’un téraoctet sont la norme, mais la vitesse d’accès étant d’environ 100 Mo/s, il faut plus de deux heures et demie pour lire toutes les données d’un disque (l’écriture est encore plus lente). La façon évidente de réduire ce temps consiste à lire à partir de plusieurs disques à la fois. Supposons que vous disposiez de 100 disques, chacun contenant un centième des données. En travaillant en parallèle, on peut lire les données en moins de deux minutes. Ainsi, une plate-forme fiable et évolutive pour le stockage et l’analyse de façon distribuée est importante.
Il existe actuellement plusieurs framework de traitement distribués et parallèles. Parmi ces Framework, nous avons Apache Spark et Hadoop MapReduce.
Les étapes d’exécution d’un Job MapReduce
- Lorsqu’un job MapReduce démarre, la première étape consiste à lire les données à partir d’HDFS et à exécuter les tâches de map ensuite;
- La sortie des mappeurs est ensuite stockée sur le disque local mais pas sur HDFS pour éviter la réplication;
- Ensuite, la phase de shuffling commence par la lecture de la sortie des mappeurs à partir du disque et, une fois terminée, elle stocke le résultat sur le disque local;
- Enfin, l’étape reduce commence par la lecture de la sortie de l’étape de shuffling et, une fois celle-ci terminée, la sortie de la phase de réduction est enfin stockée dans HDFS.
Cela représente six accès disque pour effectuer un seul job.
Les étapes d’exécution d’un Job Spark
Tout comme une application MapReduce a besoin de mappeurs et de réducteurs, Spark a deux types de processus: les transformations et actions. Ainsi, lorsque vous écrivez un job de Spark, ce job consiste en un certain nombre de transformations et de quelques actions.
- Lorsque le travail de Spark commence, il crée un DAG (graphe acyclique dirigé) du job (étapes qu’il est censé exécuter dans le cadre du job);
- Il examine ensuite le DAG et se souvient des étapes sans aller sur le disque pour effectuer les transformations;
- Ensuite, lorsqu’il rencontre une action, le job Spark va sur le disque, effectue la première transformation, conserve le résultat de la transformation en mémoire, effectue la seconde transformation, conserve le résultat en mémoire, etc. Il le fait jusqu’à ce que toutes les étapes soient terminées .
La seule fois où il retourne sur le disque est pour écrire la sortie du travail. Nous avons ainsi seulement deux accès au disque, ce qui accélère le processus de Spark.
Analyse comparative
- Lors du processus d’exécution dun Job, Spark fait deux accès au disque alors que Hadoop MapReduce fait six, ce qui montre que le processus du Spark est plus accéléré;
- Spark peut fonctionner avec n’importe type de base de données alors que Hadoop MapReduce a besoin nécessairement d’une base de données HDFS pour fonctionner;
- Spark permet de faire le traitement en temps réel et en batch tandis que MapReduce fait seulement le traitement en batch;
- Enfin Spark est 100 fois plus rapide que Hadoop MapReduce.
Ce que tu penses tu le deviens, ce que tu ressens tu l’attires, ce que tu imagines tu le crées
You may also like
Top 8 des Conseils des Experts Power BI
51 Comments
At this time I am going to do my breakfast, afterward
having my breakfast coming yet again to read further news.
Magnificent site. A lot of useful information here.
I’m sending it to some pals ans additionally sharing in delicious.
And obviously, thanks in your effort! https://www.woori88.com
Hey! Quick question that’s totally off topic.
Do you know how to make your site mobile friendly? My web site looks weird when browsing from my iphone4.
I’m trying to find a theme or plugin that might be able to correct this problem.
If you have any recommendations, please share.
With thanks!
What’s up i am kavin, its my first occasioon to commenting anywhere, when i read this article i
thought i could also make comment due to this
good piece of writing.
Grra poker poker play free poker for money online
Good way of explaining, and nice paragraph to get facts on the
topic of my resentation topic, which i am going too convey in school.
webpage bet match betting
Generally I don’t read post on blogs, but I wish to
say that this write-up very pressured me tto check out
and do it! Your writing style has been amazed
me. Thanks, quite nice article.
видео покер онлайн казино web page best online
casino usa
Oh my goodness! Impressive article dude! Many thanks, However I am encountering difficulties with
your RSS. I don’t know why I am unable to subscribe to it. Is there anybod having identical RSS problems?
Anyone that knows the solution can you kindly respond?
Thanx!!
покер онлайн poker for real money poker
for money online
I was wonddering if you ever thought of changing the page layout of your
blog? Its very well written; I love what youve gott to say.
But maybe you could a little more in thhe way off content so people could connect
with it better. Youve got an awful lot of text for only having 1 or two pictures.
Maybe you could space it out better?
dating foreign women (ecole-saintmaudet-cloharscarnoet.ac-rennes.fr)
meet single russian women
Very nice article. I absolutely love this site. Thanks!
онлайн-казино на реальные деньги сша
site online casino video poker
Remarkabloe issues here. I’m very happy to peer your post.
Thank you so much and I’m having a look ahead to contact you.
Will you kindly drop me a e-mail?
ставки на президента site ставки
Attractive portion of content. I simply stumbled upon your site and in accession capital to claim that
I geet in fact loved account your weblog posts.
Anyway I will be subscribinjg in your feeds or evn I success yoou get admission to consistently fast.
Test online casino site casino games online for money
Every weekend i used to visit this site, for the reason that i want enjoyment,
as this this site conations truly nice funny stuff too.
Nice post. I was checking constantly thgis blog aand I am impressed!
Very helpful info specifically thee last part :
) I care for such informstion a lot. I was looking for this particular information for a vey long time.
Thank youu and est of luck.
Here is my web page:
I’m amazed, I must say. Rarely do I come across a
blog that’s both educative and engaging, and without a doubt,
you’ve hit the nail on the head. The problem is something not enough folks are speaking intelligently
about. I am very happy I stumbled across this during my hunt for something
regarding this.
Right here is the perfect ebpage for anyone whho wishes to find out about this topic.
You understand so much its almost touh to argue with you (not that I actually
will need to…HaHa). You certainly put a brand new spin on a subject thnat has been discussed for many years.
Wonderful stuff, just great!
web page
Thankfulness to my father who stated to me
about this webpage, this website is genuinely awesome.
Can I just saay what a reslief too finnd somebody that actually understands what
they’re talking about over the internet.You definitely realize how to brig a problem to light and make
it important. More and mpre people should read
this and understand this side of thhe story.
I was surprised you are not more popular given tgat you certainly possess the gift.
игра в онлайн-казино web site настоящее онлайн
казино на реальные деньги
I am really loving the theme/design of your weblog. Do you ever run into any browser compatibility problems?
A handful of my blog audience have complained about my site not operating correctly in Explorer but looks great in Safari.
Do you have any ideas to help fix this issue?
Please let me know if you’re looking for a author for
your blog. You have some really good articles and I
believe I would be a good asset. If you ever want to take some of the load off,
I’d really like to write some content for your blog in exchange for a link back to mine.
Please shoot me an e-mail if interested. Cheers!
Good day! This is my first visit to your blog! We are a collection of volunteers and starting a new project in a community
in the same niche. Your blog provided us beneficial information to work on. You
have done a marvellous job!
Whenn someobe writes an piece of writing he/she retains the plan of a user
in his/her mind that how a user can be aware of it. So that’s why this post is amazing.
Thanks!
самый большой бонус онлайн казино web site лучшие онлайн
казино
Exccellent blog! Do you have any tips and hints
for aspiring writers? I’m hoping to start my own blog soon but I’m a
little lost on everything. Would you propise sarting withh
a free platform like WordPress or go for a paid option? Thrre
aare so many choices out there that I’m completely confused ..
Any recommendations? Cheers!
Real casino games online site slot machines for money
Your mose of explaining the whole thing in tthis paragraph is
actually pleasant, alll be capable of effortlessly know
it, Thanks a lot.
Online casino suisse web page
игровой клуб PM
naturally like your website however you need to check thhe spelling onn quite a few of your
posts. Manyy of them are rife with speling issues and I in finding it very troublesome to inform the
reality on the other hand I wiol surely come again again.
Casino games online web site onlione casino slots
Hello there! Quick question that’s entirely off topic.
Do you know how to make your site mobile friendly? My
website looks weird when browsing from my iphone4.
I’m trying to find a theme or plugin that might be able
to correct this problem. If you have any recommendations,
please share. Cheers!
Fastidious replies in return of this question with firm arguments and
telling everything about that.
Does your blog have a contact page? I’m having trouble locating it but, I’d like to
send you an email. I’ve got some recommendations for your blog you might be interested
in hearing. Either way, great website and I look forward to seeing it expand over time.
Thanks for sharing your thoughts on Streaming Jav Online Free – JavMama.
Regards
Hello there! Quick question that’s completely off topic.
Do you know how to make your site mobile friendly?
My blog looks weird when viewing from my apple iphone.
I’m trying to find a template or plugin that might be able to
resolve this issue. If you have any recommendations, please share.
Many thanks!
I love your blog site. extremely good colors & style.
Did you design this website yourself or did
you work with someone to do it for you? Plz respond as I’m looking to create my own blog and
wish to know where u got this from. many thanks. http://pesciantica.altervista.org/video-quarta-parte.php
Hello, after reading this remarkable post i am
too happy to share my know-how here with friends.
Superb, what a weblog itt is! This blog gives valuable data
tto us, keep it up.
Also visit my blog post …
I have been browsing on-line greater than three hours nowadays, but I by no means found any interesting article like yours.
It’s lovely worth enough for me. Personally, if all webmasters
and bloggers made just right content material as you probably
did, the net will be much more useful than ever before.
Greetings from Idaho! I’m bored to death at work so I decided to check out your
blog on my iphone during lunch break. I love the info you provide here and can’t wait to take a look when I
get home. I’m shocked at how fast your blog loaded on my cell phone ..
I’m not even using WIFI, just 3G .. Anyhow, superb site!
I couldn’t resist commenting. Exceptionally well written!
Good day very nice site!! Man .. Beautiful .. Superb ..
I will bookmark your site and take the feeds also?
I’m glad to search out a lot of helpful info here within the
post, we want work out extra strategies in this regard, thank you for sharing.
. . . . .
I could not refrain from commenting. Perfectly written!
Good article! We are linkihg to this great post on our
website. Keep up tthe good writing.
homepage sports betting
Great work! This is the kind of information that should be shared across the
internet. Shame on the seek engines for no longer positioning
this submit upper! Come on over and talk over with my site .
Thanks =)
Hello everybody, here every one is sharing these kinds of
experience, therefore it’s nice to read this blog, and I used to visit this web site all the time.
Thank you for sharing in this short article
I can discover a great deal as well as can additionally be a
reference
I enjoy to find your web site and can sign up with to comment
I believe is very useful to be able to read your writing, and on this celebration will I utilize for my
reference resource
Thanks so a lot for sharing, I wish you remain to compose spirit next topic. http://kimathankablog.blogspot.com
Hi, i think that i noticed you visited my weblog thus i came to return the prefer?.I am trying to to find things to improve my website!I guess its adequate to use a
few of your ideas!!
Hi to every one, it’s genuinely a pleasant for me to
pay a visit this site, it contains valuable Information.
Wonderful work! That is the kind of info that
should be shared around the internet. Disgrace on the seek engines
for not positioning this post higher! Come on over and seek
advice from my website . Thanks =)
Please let me know if you’re looking for a author for your site.
You have some really great posts and I think I would be a good
asset. If you ever want to take some of the load off,
I’d absolutely love to write some material for your blog in exchange for a link back to mine.
Please send me an email if interested. Many thanks!
Hi there friends, its wonderful article regarding cultureand fully defined, keep it up all
the time.
Fantastic web-site you have right here, i do concur
on some items however, but not all.
Unquestionably consider that which you said. Your favourite
reason seemed to be on the internet the simplest factor to consider of.
I say to you, I definitely get annoyed even as other people think about
concerns that they plainly do not know about. You controlled to hit the nail upon the highest as smartly as defined out the whole thing without having side effect , other folks could take a signal.
Will probably be back to get more. Thank you
I have been surfing online more than 3 hours today, yet I
never found any interesting article like yours.
It is pretty worth enough for me. In my view, if all site owners
and bloggers made good content as you did, the web
will be much more useful than ever before.
An impressive share! I’ve just forwarded this onto a colleague who has been doing
a little homework on this. And he actually ordered me
dinner simply because I stumbled upon it for him…
lol. So let me reword this…. Thank YOU for the meal!! But yeah,
thanks for spending time to talk about this issue
here on your internet site.
Normally I don’t read post on blogs, however I wish to say that this write-up very pressured me to take a look at and
do so! Your writing taste has been surprised me. Thank you, quite nice article.