Mgiyuk

Spark SQL nested query I have the following spark sql query SELECT count(*), channel FROM channelusage a WHERE a.starttime>= windowstarttime AND a.endtime <= windowendtime GROUP BY channel I have to generate these counts for 10 windows. Currently I use a while loop to generate the windowstarttime and windowendtime. What I want to do is - I want to generate the windows in the sql query itself something like a nested sql similar to this - SELECT count(*), channel FROM channelusage a WHERE (nested query logic) GROUP BY channel so that I get a output similar to this windowstarttime | windowentime | channel | count 11:00:01 | 11:00:10 | ABC |2 11:00:11 | 11:00:20 | ABC |4 11:00:21 | 11:00:30 | NBC |10 11:00:31 | 11:00:40 | CNN |5 What is your input data set? – Kannan Kandasamy Jun 29 at 20:24 ...

Check if an IP address is in a IPNetwork with Pyspark With Pyspark, I would like to join/merge if an IP address in the dataframe A is in a IP network range or hits the same IP address in the dataframe B. The dataframe A contains IP addresses only and the other one has IP addresses or IP addresses with a CIDR. Here is an example. Dataframe A +---------------+ | ip_address| +---------------+ | 192.0.2.2| | 164.42.155.5| | 52.95.245.0| | 66.42.224.235| | ...| +---------------+ Dataframe B +---------------+ | ip_address| +---------------+ | 123.122.213.34| | 41.32.241.2| | 66.42.224.235| | 192.0.2.0/23| | ...| +---------------+ then an expected output is something like below +---------------+--------+ | ip_address| is_in_b| +---------------+--------+ | 192.0.2.2| true| -> This is in the same network range as 192.0.2.0/23 | 164.42.155.5| false| | 52.95.245.0| false| | 66.42.224.235| true| -> This is in B | ...