Apache Tez -Overview

  1. tez.am.resource.memory.mb = Tez AppMaster memory
    Set tez.am.resource.memory.mb to be the same as yarn.scheduler.minimum-allocation-mb the YARN minimum container size if out-of-memory (OOM) errors occur under a realistic workload with that setting, start bumping up the number as a multiple of yarn.scheduler.minimum-allocation-mb, but do not exceed the value of yarn.scheduler.maximum-allocation-mb.
Ideally "tez.am.resource.memory.mb=4096mb" should be sufficient for most of the workload jobs.
Ideally “hive.tez.container.size=4096mb” but you can increase this value if you Tez jobs are failing with Container OOM. Try increasing it to 8 or 12 gb accordingly
<property> <name>tez.lib.uris</name> <value>/hdp/apps/${hdp.version}/tez/tez.tar.gz</value> <description>Comma-delimited list of the location of the Tez libraries which will be localized for DAGs. Specifying a single .tar.gz or .tgz assumes that a compressed version of the tez libs is being used. This is uncompressed into a tezlibs directory when running containers, and tezlibs/;tezlibs/lib/ are added to the classpath (after . and .*). If multiple files are specified — files are localized as regular files, contents of directories are localized as regular files (non-recursive). </description> </property>
tez.am.container.reuse.enabled=true
tez.am.container.idle.release-timeout-min.millis
(The minimum amount of time to hold on to a container that is idle. Only active when reuse is enabled)

--

--

--

Support Engineer @DataRobot

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

BigQuery Explained: Querying your Data

How to bypass HWID-Protected Application

20,000,000 $KYVE Token Incentivized Testnet || KYVE on Korellia Network || Korellia Testnet.

Why Practice Pseudocode?

Airtel’s Fault Tolerant, Highly Scalable, Real Time Scheduler

EC2 [RAM, Disk Metric] How To Set Up Custom Metrics With CloudWatch

Let’s Learn Python | Setup and Basics

Decoding the Company Values in a Senior Developer Interview: Customer Focus

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Tamil Selvan K

Tamil Selvan K

Support Engineer @DataRobot

More from Medium

Kafka-Integration-with-Apache-Spark

Quick guide to Apache Pig

Introduction to GeoSpatial streaming with Apache Spark and Apache Sedona

Deep dive into Dask Distributed: Part 2

Typical Dask distributed configuration