Distributed SQL engines must process data across multiple servers. In this talk, Vladimir will tell, using Apache Flink and Presto as an example, how distributed SQL engines are arranged, and what approaches they use to increase query performance.
Almost any company operating data finds it necessary to store and process data in different systems depending on the tasks. In such a world, a request arises for a service that can quickly and efficiently transfer data between these worlds. To solve this problem, Yandex has developed Data Transfer, a cross-system data replication service and Andrey plans to talk about it.
Imagine that a company needs to build a powerful analytical platform. ManyChat created such configuration, choosing the latest tools for maximum convenience and minimizing the cost of ownership. Nikolay plans to describe the selection process at each step of building the platform, possible risks, and the final experience.
Dmitry will tell how to organize the access and work with data for different specialists — engineers, analysts, data scientists. He will also tell how approaches to the allocation of computing resources and access organization have evolved, how changed the tool set and modeling approaches, how the approaches to the output of results into industrial operation developed.
This talk focuses on techniques employed in hybrid storage systems to reduce cloud footprint and improve efficiencies.
In this talk Sabir will walk you through physical data layout optimizations available with Delta Lake. It will discuss factors that make a query execute fast.
As part of this report, we will study the Data Modeling Methodology, step by step consider the basic principles of creating an effective data model. Let's get acquainted with typical cases and common mistakes, learn the rules that will help you get the most out of your DBMS, and avoid common problems.
As part of this report, we will study the Data Modeling Methodology, step by step consider the basic principles of creating an effective data model. Let's get acquainted with typical cases and common mistakes, learn the rules that will help you get the most out of your DBMS, and avoid common problems.
Find out what is coming down the pipe for Apache Airflow in version 2.3 and beyond.
In this talk, Andy will discuss the challenges in using ML to optimize DBMS knobs and the solutions we developed to address them. My presentation will be in the context of the OtterTune database tuning service. Andy will also highlight the insights learned from real-world installations of OtterTune for MySQL, Postgres, and Oracle.
Is it possible to set up Spark so it never touches hard drives and hence be memory-fast? That's the question that Jacek is going to answer during the talk. You'll know a bit about the internals of Apache Spark and what parts are or could be memory-only and what challenges it poses.
Let's talk about how, before making any changes to the pipeline in a production environment, you need to assess the potential impact on the system. You will find out that sometimes the pipeline is so complex and entangled in dependencies that it is almost impossible to predict the ending without experimenting.
We'll talk about Trino. You'll learn about work with the data from primary sources, combining and enriching them, subsecond requests. We'll also talk about hidden opportunities, new functionalities, what we have in a project, or his forks.
In this talk, Ton will discuss how to get faster and more secure access to data for testing purposes, by generating private data that (a) emulates the state of a dataset/database and (b) increases testing coverage. There are several tools available on OSS, but usually, the devil is in the detail.
This talk will look at the NiFi ETL tool — its pros and cons, tools and methods for monitoring, and the development process for a large number of teams.
In this talk, we'll discuss the topics of Data Lake open architecture, Apache Parquet, and Apache Arrow data formats. Why do we need Apache Iceberg and Deltalake table formats, and how the Nessie project will help build SQL Lakehouse on Data Lake.
BigData MTS has grown and matured, but some of the problems that it received while developing ML still remain. And, as it turned out, they are not alone in their fight against them.
In this session, we will go in deep, with practical examples, on how to map external data with Vertica, which are the Vertica options to push down the queries to external data repositories and the technologies behind it. Differences between Vertica and some other solutions will also be explained.
The data engineer's role is very important and critical. What skills should he have, how well should he know the code, algorithms, and data science? Dmitry was able to identify 2 types of data engineers and will tell about them during this session.
In this talk, Kirill will tell how MTS was able to launch an AI-service of computer vision on EDGE devices in 500 offices of companies. What pitfalls the team faced and how they were able to keep the entire fleet of devices up to date, process, and verify data from all offices.
The participants in the discussion will try to raise various tricky questions in the spirit of "how convenient is it to store raw data NOT in HDFS" and "is it possible to simply transfer everyone to the SQL engine". And also "is it possible to call the daemon with the words Data Mesh, Delta Lake, Anchor" and "how to make Kappa architecture in real life and what is it all about".
Analysis of self-service BI application in terms of data model building.
In this talk, Ekaterina wants to talk about why Citymobil chose Exasol as the DBMS for the warehouse, and Data Vault as the data model.
The talk will focus exclusively on Apache Spark & cost-based engineering will be the theme.
There are several options for how to insert data to ClickHouse correctly, and even more how to do it incorrectly. We'll talk about how to add data to ClickHouse, what pitfalls we can face, and how to avoid them.
Ivan's talk will be about the work on creating a DataCrafter data catalog based on MongoDB, based on large heterogeneous public data of complex formats from unmanaged sources.
During this talk, we will discuss what data engineer's life consists of and how do we help them with Big Data Tools.
Learn how lakeFS simplifies the management of a Data Lake by enabling git-like operations over files in object storage. See how common processes like experimentation, reproducing data and ensuring data quality are simplified with workflows centered around branching, committing, and the merging of data.
In this talk, Evgeny and Nikolay would like to tell how dreams of architectural beauty shutter against reality.
The data storage appeared in Avito more than 7 years ago. During this time, the business has grown several times, and the infrastructure has become more complex. Evgeny will tell how the product approach to platform development helps to solve dozens of analytical problems every day without the multiple growths of the DWH team.
The brief talk about how to implement a SQL layer for any storage using Apache Calcite.
In this talk, Dmitry will tell how to write in Spark functionally using Scala at maximum speed.
How to build an ML pipeline in a computer vision startup.
You'll be introduced to Exasol, the world's fastest analytical database. You will discover how Exasol can simplify your life and make having a data warehouse fun again.
Airflow SaaS implementation in K8s private cloud and experience of migration from Airflow 1.x to Airflow 2.x SaaS.
Erasure coding in Hadoop 3: a story about how the pursuit of the smart economy can turn out to be (almost) a disaster, and how to avoid it. Based on real data petabytes and a sea of tears.