-
KxWarehouseData Tutorial – data not flowing from LR to IHDB and HDB
Hello again,
I have been tasked to try the KxWarehouseData tutorial on KX 4.6.1P3 so we can embrace the use of services classes. However, unfortunately, I am facing a very annoying issue.
My monFeed service is publishing data to the kxw_tp, which in turn sends it to the RDB, but the data is not flowing to the IHDB and, I suspect, to the Log Recovery service.
I decided to investigate and find an error on the tutorial:
The log replay page lists logFileList as:ENV=DELTADATA_HOME=/KxWarehouseData/tplog/mon_tp_%DATE% ENV=DELTADATA_HOME=/KxWarehouseData/tplog/mon_tp_*%DATE%_H[0-9][0-9][0-9][0-9].[0-9][0-9].[0-9][0-9].[0-9][0-9][0-9][0-9][0-9][0-9]
However, the correct values should be:
ENV=DELTADATA_HOME=/KxWarehouseData/tplog/kxw_tp_*%DATE% ENV=DELTADATA_HOME=/KxWarehouseData/tplog/kxw_tp_*%DATE%_H_[0-9][0-9][0-9][0-9].[0-9][0-9].[0-9][0-9].[0-9][0-9][0-9][0-9][0-9][0-9]
mon_tp needs to be kxw_tp since the latter is the name of the tp service.
The first entry needs an * so it matches the port of the tp service that is captured on the filename, and the second entry needs an _ after the H.These fixes made the Log Recovery read the TP log files, but no data was put into the LR and the IHDB. No data has flowed to the HDB either.
Since the intraday directory matches, I am wondering if there is some other information missing in the tutorial?
Cheers,
Adriano -
4 Replies
-
Hi Adriano – can you please share a screenshot of the process configuration of the TP and LR?
On a scheduled basis the LR should be checking for a new tp log file, reading the previous one and writing it to the iHDB. So I am interested to see how often the TP log rolls and how often the LR is checking for the log file.
Also the process log of the LR would help.Sam
-
Hi Sam,
I have attached pictures of the configuration as requested and put links for the logs below.
(The portal does not accept .log files as attachments )I put logs for 01/07 (when i was originally facing issues) and today.
Funny thing: i noticed today that data flowed (!?!?). So if nothing catches your eyes, i will probably consider my initial issue as a synchronization error on my vm setup.Bonus:
Log File List on LR
ENV=HOME=/TP/uiCore/uiCore_tp_*%DATE%
ENV=HOME=/TP/uiCore/uiCore_tp_*%DATE%_H_[0-9][0-9][0-9][0-9].[0-9][0-9].[0-9][0-9].[0-9][0-9][0-9][0-9][0-9][0-9]LR Logs
2021-07-01D05:14: https://pastebin.com/gjVvGU5s
2021-07-07D02:30: https://pastebin.com/XpBTt1JM2021-07-07D02:30: https://pastebin.com/NUFqRxCQ
-
Hi Adriano – reviewing the logs it looks like the LR process has successfully identified the intraday TP log to ingest, replayed it and written it to the iHDB location.
//File identified to replay
Found files to replay ### `s#,`kxplatform/TP/uiCore/uiCore_tp_2339_2021.07.01_H_2021.07.01.050000
//Example of an intraday partition being upserted to
<->2021.07.01D06:00:48.077 ### dev ### normal ### (252950): Upserted 1 rows for temperature to kxplatform/INTRADAY/uiCore/2021.07.01//13/uiCore_telemetry/ ###
Good spot on missing wild card required to pick up the service class. I’ll pass that back to Core.
Cheers,
Sam
-
Hey Sam,
Thanks (again) for checking the logs. I can now confirm that there was a sync error on my vm. Same error repeated today and after I reloaded my vm I could see new data on HDB.That said, I have one final question:
How do we manually EOD under this new setup? Does calling end from a TP scratchpad still works?Cheers,
Adriano
Log in to reply.