Apache Tez -Overview

  1. tez.am.resource.memory.mb = Tez AppMaster memory
    Set tez.am.resource.memory.mb to be the same as yarn.scheduler.minimum-allocation-mb the YARN minimum container size if out-of-memory (OOM) errors occur under a realistic workload with that setting, start bumping up the number as a multiple of yarn.scheduler.minimum-allocation-mb, but do not exceed the value of yarn.scheduler.maximum-allocation-mb.
Ideally "tez.am.resource.memory.mb=4096mb" should be sufficient for most of the workload jobs.
Ideally “hive.tez.container.size=4096mb” but you can increase this value if you Tez jobs are failing with Container OOM. Try increasing it to 8 or 12 gb accordingly
<property> <name>tez.lib.uris</name> <value>/hdp/apps/${hdp.version}/tez/tez.tar.gz</value> <description>Comma-delimited list of the location of the Tez libraries which will be localized for DAGs. Specifying a single .tar.gz or .tgz assumes that a compressed version of the tez libs is being used. This is uncompressed into a tezlibs directory when running containers, and tezlibs/;tezlibs/lib/ are added to the classpath (after . and .*). If multiple files are specified — files are localized as regular files, contents of directories are localized as regular files (non-recursive). </description> </property>
tez.am.container.reuse.enabled=true
tez.am.container.idle.release-timeout-min.millis
(The minimum amount of time to hold on to a container that is idle. Only active when reuse is enabled)

--

--

--

Support Engineer @DataRobot

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

This is the way of SugarCRM — a CRMKonvo

Tabular Editor version 3 coming soon!

Transportation Problem: Balancing

Android Studio Fundamentals 1.3: Menambah dan Mengedit Elemen TextView

Rotating in World Space- Game Dev Series 141

Capacitive Sensing with Piezo

Smaller Batch Sizes: It's All Relative

Flutter Offline Storage Using HIVE database

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Tamil Selvan K

Tamil Selvan K

Support Engineer @DataRobot

More from Medium

Apache Airflow

Apache Hadoop’s Core: HDFS and MapReduce — Brief Summary

How can Apache Spark help your big data?

Apache Spark RDD