Hive Datanucleus ConnectionPool

(No. of Hiveserver2 Instance x 2 x datanucleus.connectionPool.maxPoolSize) + (No. of HMS Instance x 2 x datanucleus.connectionPool.maxPoolSize) = total connections to db
METASTORE_CONNECTION_POOLING_TYPE('datanucleus.connectionPoolingType', 'BONECP',
'Specify connection pool library for datanucleus'),
METASTORE_CONNECTION_POOLING_MAX_CONNECTIONS('datanucleus.connectionPool.maxPoolSize', 10,
'Specify the maximum number of connections in the connection pool. Note: The configured size will be used by\n' +
' 2 connection pools (TxnHandler and ObjectStore). When configuring the max connection pool size, it is \n' +
'recommended to take into account the number of metastore instances and the number of HiveServer2 instances \n' +
'configured with embedded metastore. To get optimal performance, set config to meet the following condition\n'+
'(2 * pool_size * metastore_instances + 2 * pool_size * HS2_instances_with_embedded_metastore) = \n' +
'(2 * physical_core_count + hard_disk_count).'),
(No. of Hiveserver2 Instance x 2 x datanucleus.connectionPool.maxPoolSize) + (No. of HMS Instance x 2 x datanucleus.connectionPool.maxPoolSize) = total connections to db
The current "max_connections" from your configuration file.
$ cat postgresql.conf | grep -i "max_connections"
max_connections = 100 # (change requires restart)
mysql> show variables like 'max_connections';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 151 |
+-----------------+-------+
mysql> SHOW FULL PROCESSLIST
  • The available RAM — The system will need to have enough RAM to handle the additional workload.
  • The thread library quality of the platform — This will vary based
    on the platform. For example, Windows can be limited by the Posix
    compatibility layer it uses (though the limit no longer applies to
    MySQL v5.5 and up). However, there remains memory usage concerns
    depending on the architecture (x86 vs. x64) and how much memory can
    be consumed per application process.
  • The required response time — Increasing the number could increase
    the amount of time to respond to request. This should be tested to
    ensure it meets your needs before going into production.
  • The amount of RAM used per connection — Again, RAM is important,
    so you will need to know if the RAM used per connection will overload the system or not.
  • The workload required for each connection — The workload will also
    factor in to what system resources are needed to handle the
    additional connections.

--

--

--

Support Engineer @DataRobot

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Getting Started with Enigma: An Intro to Secret Contracts

Linux Swap Memory

Notes from 2019 Developers’ Festival in Uyo (#DevFestUyo)

Simple Media Streaming with VLC and IceCast

How to Increase Microphone Volume on OBS Studio?

Center Bottom of the Window, You can see the Audio Mixer Panel, where you can see all the Audio Setting, from Desktop Audio, Mic/Aux Audio Volume, Audio Capture Device

Sample Flutter mobile app to consume your WSO2 Cloud APIs with PKCE

Querying APIs — Techniques

Data Mesh and Architecture

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Tamil Selvan K

Tamil Selvan K

Support Engineer @DataRobot

More from Medium

Python: How to Convert Database(Tables) to XLSX Workbooks

How to Ace the Azure DP-100-Azure Data Scientist Associate Exam

#MapReduce... What !?!?