Automic Workload Automation

Expand all | Collapse all

Dynamically pass a value to the hive query

  • 1.  Dynamically pass a value to the hive query

    Posted 09-07-2016 05:39 PM
    I am very new to UC4. I am using UC4 for scheduling the job for my BIG Data ingestion project where I am using Hive. My requirement is dynamically I need to find the next available partition date of 1 hive table and then pass that date to a hive query. I have done the below steps.

    I have used  shell job and inside process tab I have written

    partitions=$(hive -e "SHOW PARTITIONS default.idmap_snapshot")
    do
    for partition in $partitions
    do
    partition_date=`echo "$partition" | cut -c4-11`
    if [ $partition_date -gt 20160718 ]; then
    echo $partition_date
    break
    fi  
    done

    hive -e "/dw/etl/home/dev/hql/dw_apt/dw_apt.dw_merged_profile_w_dtl_adelphic_us.ins.hql $partition_date"

    Please let me know whether my approach is correct or not? Right now I am facing some access issue for which I cannot debug the job. Any help would be highly appreciated.

    Thanks,
    Mrinmoy 


  • 2.  Dynamically pass a value to the hive query

    Posted 09-07-2016 05:55 PM
    1 modification..The call of hive query from job shell would be like below.

    hive -hiveconf partition_date=$partition_date
    -f "/dw/etl/home/dev/hql/dw_apt/dw_apt.dw_merged_profile_w_dtl_adelphic_us.ins.hql"