Engineering a big data ingestion pipeline is complicated – if you don’t have the right tools. All data, be it big, little, dark, structured, or unstructured, must be ingested, cleansed, and transformed before insights can be gleaned, a base tenet of the  analytics pr...

In the beginning of ETL…. When I started my IT career over 15 years ago I was nothing more than a “Fresh-out” with a college degree and an interest in computers and programming. At that time, I knew the theories behind the Software Development Life Cycle (SDLC) and had put it to some practice in a classroom setting but, I was still left questioning how it relates to the big, bad corporate world. And by the w...

Bereits heute verbindet das Internet der Dinge die digitale mit der physischen Sphäre und schließt die Lücke zwischen Informationssystemen und der Außenwelt. Noch vor kurzem gab es eine Kluft zwischen diesen beiden Bereichen, da Daten aus der physischen Welt sich nicht mit IT-Systemen integrieren ließen. In der Regel wurden extern erfasste Werte erst im Nachhinein eingebunden – etwa im Rahmen einer Stapelverarbeitung oder durch manuelle Eingabe. Dieser Umstand machte eine Verwendung in Ec...

  As part of a POC of Talend v6.1 Big Data capabilities, I was asked by one of our long-time customers, a major e-commerce company, to present a solution for aggregating huge files of clickstream data on Hadoop. The input data was a giant clickstream file (larger than 100GB, or even terabytes) from a website. Our goal was to aggregate the file and add a session ID column. We needed to create session records for all u...

  In my previous post about data quality in the Big Data era, we've seen some of the challenges raised by the recently ...

  With the world becoming more connected and data savvy with every passing year, there’s a rising need for businesses to efficiently manage the trillions of bytes of data that they capture, and gain insights from them. Talend helps businesses do exactly this while boosting developer productivity and reducing time-to-value for ETL data warehouse projects. Talend for Big Data is seeing rapid growth, emerging as a must-have tool for quickly and effectively clea...

In my previous blog “Beyond ‘The Data Vault’” I examined various data storage options and a practical architecture/design for an Enterprise Data Vault Warehouse.  As you may have realized by now I am quite smitten with this innovative data modeling methodology and recommend to anyone who is developing a ‘Data Lake’ or Data Warehouse on Big Data platforms consider th...

This blog is the first in a series of posts explaining the overarching goal and purpose of the Apache Beam project. In the future blogs, we will explain how to use Apache Beam to implement data processing jobs.  When you have an existing big data platform, the continuous evolution of that platform is important. If you’re currently using Apache Hadoop MapReduce jobs to process your data, you may want to migrate to Apache Spark in ord...

Bereits heute verbindet das Internet der Dinge die digitale mit der physischen Sphäre und schließt die Lücke zwischen Informationssystemen und der Außenwelt. Noch vor kurzem gab es eine Kluft zwischen diesen beiden Bereichen, da Daten aus der physischen Welt sich nicht mit IT-Systemen integrieren ließen. In der Regel wurden extern erfasste Werte erst im Nachhinein eingebunden – etwa im Rahmen einer Stapelverarbeitung oder durch manuelle Eingabe. Dieser Umstand...

Blog authored by Nitin Kudikala, Customer Success Architect at Talend As companies try to become data driven enterprises, new distributed data architectures such as Hadoop create new opportunities to derive more information and intelligence from their Data assets. As these technologies evolve, securing the data stored in these data stores has become critical. There is a clear need for comprehensive security as Hadoop holds sensitive inf...