I am very new to UC4. I am using UC4 for scheduling the job for my BIG Data ingestion project where I am using Hive. My requirement is dynamically I need to find the next available partition date of 1 hive table and then pass that date to a hive query. I have done the below steps.
I have used shell job and inside process tab I have written
partitions=$(hive -e "SHOW PARTITIONS default.idmap_snapshot")
do
for partition in $partitions
do
partition_date=`echo "$partition" | cut -c4-11`
if [ $partition_date -gt 20160718 ]; then
echo $partition_date
break
fi
done
hive -e "/dw/etl/home/dev/hql/dw_apt/dw_apt.dw_merged_profile_w_dtl_adelphic_us.ins.hql $partition_date"
Please let me know whether my approach is correct or not? Right now I am facing some access issue for which I cannot debug the job. Any help would be highly appreciated.
Thanks,
Mrinmoy