Google Cloud Bigquery Library Error


Google Cloud Bigquery Library Error



I am receiving this error

Cannot set destination table in jobs with DDL statements



Cannot set destination table in jobs with DDL statements



When I try to resubmit a job from the job.build_resource() function in the google.cloud.bigquery library.


job.build_resource()


google.cloud.bigquery



It seems that the destination table is set to something like this after that function call.




'destinationTable': {'projectId': 'xxx', 'datasetId': 'xxx', 'tableId': 'xxx'},



'destinationTable': {'projectId': 'xxx', 'datasetId': 'xxx', 'tableId': 'xxx'},



Am I doing something wrong here? Thanks to anyone that can give me any guidance here.



EDIT:



The job is initially being triggered by this


query = bq.query(sql_rendered)



We store the job id and use it later to check the status.



We get the job like this


job = bq.get_job(job_id=job_id)



If it meets a condition, in this case it failed due to rate limiting. We retry the job.



We retry the job like this


di = job._build_resource()
jo = bigquery.Client(project=self.project_client).job_from_resource(di)
jo._begin()



I think that's pretty much all of the code you need, but happy to provide more if needed.





Please share more of the code that you are using. It isn't possible to tell what is happening to cause the destination table to be set just from the error message.
– Elliott Brossard
2 days ago





Is there a reason that you don't resubmit the original job? It's kind of weird to submit the one returned by bq.get_job, since that contains unrelated attributes like query statistics and so on (and in this case the destination table).
– Elliott Brossard
2 days ago


bq.get_job





The reason is that we are using Airflow. So it's passed through xcoms, so we are storing the job_id, grabbing that and then using it. Should we change that workflow?
– dillon
2 days ago






The (easy) alternative would be to strip down the job resource that you get back just to the relevant fields, but a better long-term solution may be to propagate the original job or query that you want to run. It's hard to say which is more reasonable given your current setup, though.
– Elliott Brossard
2 days ago









By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Comments

Post a Comment

Popular posts from this blog

paramiko-expect timeout is happening after executing the command

paramiko-expect timeout is happening after executing the command Can someone help me why timeout is happening after executing the command .I am trying to SSH to a machine and execute a command.I want to store the result of the executed command . I have referred paramiko-timeout but i didn't get my expected result. import traceback try: import paramiko except ImportError: raise Exception("Please install paramiko , pip install paramiko") import pexpect from paramiko_expect import SSHClientInteraction def main(): HOSTNAME = "minion5.net" USERNAME = "root" PASSWORD = "help@432" PROMPT = "root@*:~$s+" try: client = paramiko.SSHClient() client.load_system_host_keys() client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) client.connect(hostname=HOSTNAME, username=USERNAME, password=PASSWORD) with SSHClientInteraction(client, timeout=20, display=True) as interact: ...
Read more

Opening a url is failing in Swift

Opening a url is failing in Swift breakpointing inside the callback and logging success shows false. I added itms-apps to my plist: LSApplicationQueriesSchemes and put this in my app delegate: func application(_ app: UIApplication, open url: URL, options: [UIApplicationOpenURLOptionsKey : Any] = [:]) -> Bool { return true } func moreTapped() { let url = NSURL(string: "itms-apps://itunes.com/developer/quantum-productions/id979315877") if #available(iOS 10.0, *) { UIApplication.shared.open(url! as URL, options: [:], completionHandler: {(success: Bool) in }) } else { UIApplication.shared.openURL(url! as URL) } } Are you testing on a real device? – rmaddy Jun 21 at 23:57 I try on a device, get "Cannot Connect to App Store". Device is online ...
Read more

Export result set on Dbeaver to CSV

Export result set on Dbeaver to CSV Normally I use Dbeaver for windows and always export my result set like this: run my query --> select the result --> export the result set --> select export to clipboard --> done This step by step puts my result set in my clipboard and I can paste it wherever I want to work with it. The problem is that now I am using dbeaver for mac and this guide is not working. I can go on until the moment that I select my result set like in the image below: exporting data set But once I go further in the process, in the last step I get: no query Note that in "source" it was suppose to show the query that originated the result set, but instead it says just "select. As a result it does't select my result or anything (besides being "successful"). Normally my query would show up there automatically and I couldn't find any option that corrects this problem in the menus. Please help me! Thanks. Aft...
Read more