How to handle JDBC SQL Server, Sybase for long running and high memory consumption process


How to handle JDBC SQL Server, Sybase for long running and high memory consumption process



I have a batch job querying Sybase and SQL Server databases. This batch job can run for up to 1 day or more. We are running this on a small set of data and no error so far in terms of connection timeout. My questions are



How to handle this long running process? Should I configure a reconnect period so that the connection gets closed and reopened?



How to handle the resultset when it can return back to the client with 1 million records?



EDIT #1:



This sounds like a general question for jdbc but it's not because each database provider has their own options such as fetching size. It's very much up to each provider to support this or not. If Sybase does not support this, it means it will load all results into memory at once.




1 Answer
1



This is a general question not strictly related to Sybase (SAP) ASE.



If you want the tcp/ip connection not to break then use some keep alive parameters for network connections. If you want to handle network connection breaks then use some connection polling libraries.



You don't have to store the whole result set in your memory. Just read your rows and process them on the fly. If you do want to fetch all 1 million rows before before doing anything with them - then just more memory to the JVM.





it's related Adam. I saw other databases like Oracle they have streaming options. But haven't seen this with Sybase
– TrongBang
2 days ago





Please add an example query that you are using and is grabbing into memory immediately all 1M rows at once. Normal iteration through resultset should not do that.
– Adam Leszczyński
yesterday






By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Comments

Post a Comment

Popular posts from this blog

paramiko-expect timeout is happening after executing the command

paramiko-expect timeout is happening after executing the command Can someone help me why timeout is happening after executing the command .I am trying to SSH to a machine and execute a command.I want to store the result of the executed command . I have referred paramiko-timeout but i didn't get my expected result. import traceback try: import paramiko except ImportError: raise Exception("Please install paramiko , pip install paramiko") import pexpect from paramiko_expect import SSHClientInteraction def main(): HOSTNAME = "minion5.net" USERNAME = "root" PASSWORD = "help@432" PROMPT = "root@*:~$s+" try: client = paramiko.SSHClient() client.load_system_host_keys() client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) client.connect(hostname=HOSTNAME, username=USERNAME, password=PASSWORD) with SSHClientInteraction(client, timeout=20, display=True) as interact: ...
Read more

Export result set on Dbeaver to CSV

Export result set on Dbeaver to CSV Normally I use Dbeaver for windows and always export my result set like this: run my query --> select the result --> export the result set --> select export to clipboard --> done This step by step puts my result set in my clipboard and I can paste it wherever I want to work with it. The problem is that now I am using dbeaver for mac and this guide is not working. I can go on until the moment that I select my result set like in the image below: exporting data set But once I go further in the process, in the last step I get: no query Note that in "source" it was suppose to show the query that originated the result set, but instead it says just "select. As a result it does't select my result or anything (besides being "successful"). Normally my query would show up there automatically and I couldn't find any option that corrects this problem in the menus. Please help me! Thanks. Aft...
Read more

Opening a url is failing in Swift

Opening a url is failing in Swift breakpointing inside the callback and logging success shows false. I added itms-apps to my plist: LSApplicationQueriesSchemes and put this in my app delegate: func application(_ app: UIApplication, open url: URL, options: [UIApplicationOpenURLOptionsKey : Any] = [:]) -> Bool { return true } func moreTapped() { let url = NSURL(string: "itms-apps://itunes.com/developer/quantum-productions/id979315877") if #available(iOS 10.0, *) { UIApplication.shared.open(url! as URL, options: [:], completionHandler: {(success: Bool) in }) } else { UIApplication.shared.openURL(url! as URL) } } Are you testing on a real device? – rmaddy Jun 21 at 23:57 I try on a device, get "Cannot Connect to App Store". Device is online ...
Read more